Document processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a document processing method and device, electronic equipment and storage medium, and relates to the technical field of data processing, in particular to the technical field of artificial intelligence, large language model, intelligent search, intelligent question answering and the like. The specific implementation scheme is as follows: a plurality of similar documents similar to a search sentence are obtained; validity scores of the plurality of similar documents are determined; for each similar document, the validity score of the similar document is used to represent the effective degree of responding to the search sentence by using the similar document; and based on the validity scores of the plurality of similar documents, valid documents are screened out from the plurality of similar documents to obtain a valid document set.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to the technical field of artificial intelligence, large language model, intelligent search, intelligent question answering, and the like. BACKGROUND

[0002] Electronic documents, as carriers of information, facilitate people's learning, work and entertainment. In addition, with the continuous development of various fields, a large amount of knowledge has been accumulated. These knowledge can also be in the form of documents for learning, research and application.

[0003] While providing convenience, the existence of a large amount of documents also brings challenges to document analysis and processing. SUMMARY

[0004] The present disclosure provides a document processing method and device, an electronic device and a storage medium.

[0005] According to an aspect of the present disclosure, a document processing method is provided, comprising:

[0006] Obtaining a plurality of similar documents similar to a retrieval sentence;

[0007] Determining an effectiveness score of the plurality of similar documents; for each similar document, the effectiveness score of the similar document is used to represent the effectiveness degree of responding to the retrieval sentence by using the similar document;

[0008] Based on the effectiveness scores of the plurality of similar documents, filtering out effective documents from the plurality of similar documents to obtain a set of effective documents.

[0009] According to an aspect of the present disclosure, a training method of a large language model is provided, comprising:

[0010] Obtaining a retrieval sample and a plurality of document samples associated with the retrieval sample;

[0011] Inputting the retrieval sample and the plurality of document samples into a to-be-trained large language model to obtain an effectiveness score of each document sample predicted by the to-be-trained large language model; for each document sample, the effectiveness score of the document sample is used to represent the effectiveness degree of responding to the retrieval sample by using the document sample;

[0012] Based on the effectiveness score of each document sample, performing binary classification on each document sample to determine whether each document sample can effectively meet the demand expressed by the retrieval sample;

[0013] Based on the binary classification result of each document sample and the classification true value, determining a loss value;

[0014] Based on the loss value, fine-tuning the to-be-trained large language model, and under the condition of meeting the convergence condition, obtaining a first large language model for predicting the effectiveness score.

[0015] According to another aspect of the present disclosure, a document processing apparatus is provided, comprising:

[0016] a first screening module configured to obtain a plurality of similar documents similar to the retrieval statement;

[0017] a prediction module configured to determine an effectiveness score of each of the plurality of similar documents, the effectiveness score of each of the plurality of similar documents indicating an effectiveness of the similar document in responding to the retrieval statement;

[0018] a second screening module configured to screen, based on the effectiveness scores of the plurality of similar documents, effective documents from the plurality of similar documents, to obtain a set of effective documents.

[0019] According to another aspect of the present disclosure, a large language model training apparatus is provided, comprising:

[0020] an obtaining module configured to obtain a retrieval sample and a plurality of document samples associated with the retrieval sample;

[0021] an estimation module configured to input the retrieval sample and the plurality of document samples into a large language model to be trained, to obtain an effectiveness score of each of the plurality of document samples predicted by the large language model to be trained, the effectiveness score of each of the plurality of document samples indicating an effectiveness of the document sample in responding to the retrieval sample;

[0022] a classification module configured to perform binary classification on each of the plurality of document samples based on the effectiveness score of each of the plurality of document samples, to determine whether each of the plurality of document samples can effectively meet a requirement expressed by the retrieval sample;

[0023] a determination module configured to determine a loss value based on a binary classification result of each of the plurality of document samples and a classification true value;

[0024] an adjustment module configured to fine-tune the large language model to be trained based on the loss value, and obtain a first large language model for predicting the effectiveness score when a convergence condition is met.

[0025] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein

[0028] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the methods in the embodiments of the present disclosure.

[0029] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any of the methods according to embodiments of the present disclosure.

[0030] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements any of the methods according to embodiments of the present disclosure.

[0031] It should be understood that the description of the background of the present disclosure is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0033] Figure 1 is a flowchart of a document processing method according to an embodiment of the present disclosure;

[0034] Figure 2 is a flowchart of estimating effectiveness scores in a plurality of similar documents using a first large language model according to an embodiment of the present disclosure;

[0035] Figure 3 is a flowchart of processing text feature information according to an embodiment of the present disclosure;

[0036] Figure 4 is a flowchart of a training method of a large language model according to an embodiment of the present disclosure;

[0037] Figure 5 is a flowchart of training a large language model to predict effectiveness scores according to an embodiment of the present disclosure;

[0038] Figure 6 is a structural diagram of a document processing apparatus according to an embodiment of the present disclosure;

[0039] Figure 7 is a structural diagram of a training apparatus of a large language model according to an embodiment of the present disclosure;

[0040] Figure 8 is a block diagram of an electronic device for implementing a document processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0042] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0043] Faced with massive amounts of documents, intelligent recommendation and analysis systems can filter out some documents to meet user needs. For example, in the case of a question, an intelligent question-and-answer application can automatically analyze at least one document to obtain the answer to the question and respond to it. However, the quality of the document affects the accuracy of the answer. Especially in knowledge retrieval applications, effective strategies are needed to filter out valid documents to respond to users. In view of this, this disclosure proposes a document processing method. This method can be applied not only to document-based intelligent question-and-answer applications but also to document-based recommendation systems.

[0044] like Figure 1 The diagram shown is a flowchart of a document processing method provided in this embodiment of the disclosure, including the following:

[0045] S101, retrieve multiple similar documents that are similar to the search query.

[0046] You can search for literature, news reports, blog posts, research reports, papers, etc. related to the search query in academic databases, library resources, or on the Internet.

[0047] By finding documents similar to the search query, it is possible to effectively filter out content related to the query. However, this related content is only considered from the perspective of similarity and cannot clearly and directly describe whether it can effectively answer the search query. Therefore, this disclosure proposes a validity assessment method to finely filter multiple similar documents.

[0048] In S102, the validity scores of multiple similar documents are determined; for each similar document, the validity score is used to indicate the effectiveness of using similar documents to respond to the search statement.

[0049] It can also be understood that the similar document meets the degree of satisfaction of the demand expressed by the search statement.

[0050] In the embodiments of the present disclosure, the effectiveness score can directly measure whether a document can effectively meet the demand of the search statement. Taking a question statement as an example, the effectiveness score can quantify to what extent the document can effectively answer the question statement, and how effective the answer is.

[0051] For example, if the search statement is about the application of artificial intelligence technology in imaging analysis, a document with a high effectiveness score may contain detailed case studies, the latest research results, authoritative opinions and analysis processes. While a document with a low effectiveness score may only contain an introduction to some basic concepts or outdated information.

[0052] In S103, a plurality of effective documents are selected from the plurality of similar documents based on the effectiveness scores of the plurality of similar documents, to obtain an effective document set.

[0053] In implementation, similar documents with effectiveness scores greater than a preset threshold can be selected from the plurality of similar documents to obtain the effective document set. Alternatively, the top n similar documents can be selected to obtain the effective document set. Alternatively, similar documents ranked in the top n and having effectiveness scores greater than the preset threshold can be selected to obtain the effective document set.

[0054] The preset threshold is a preset score limit for distinguishing which documents are effective for the search statement and which documents are ineffective. In order to accurately select effective documents, the preset threshold can be set based on experience or training. It can be understood that the higher the effectiveness score, the more likely the document is considered to effectively answer the search statement.

[0055] In the embodiments of the present disclosure, by finding similar documents to the search statement, documents related to the content of the search statement can be obtained, thereby achieving coarse screening of massive documents. Further, it is proposed to further screen the coarse screened documents from the effectiveness of answering the search statement, thereby selecting effective documents that can effectively meet the demand of the search statement, to obtain an effective document set. Therefore, the scheme provided by the embodiments of the present disclosure can further improve the quality of the screened documents, and ultimately improve the efficiency of retrieving effective documents from massive documents.

[0056] In some possible embodiments, the effectiveness score can be calculated based on a series of criteria, including but not limited to the content quality, structural integrity, information accuracy and timeliness, and theme relevance to the search statement of the document. Therefore, in implementation, a neural network model can be used to analyze the information in these dimensions to obtain the effectiveness score of the similar document.

[0057] In some other possible embodiments, the first pre-trained large language model can also be used to estimate the validity scores of the plurality of similar documents.

[0058] wherein the large language model (LLM, Large Language Models) refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model can handle various natural language tasks such as text classification, question answering, dialogue, etc.

[0059] The large language model has strong knowledge background and logical reasoning ability, and the first large language model in the embodiments of the present disclosure is pre-trained to meet the demand of evaluating the validity of the document. Through the first large language model, documents that are more relevant to the search statement and are more likely to contain accurate answers can be screened out, thereby improving the quality of document retrieval and being able to accurately answer the search statement.

[0060] In addition, when the set of valid documents retrieved is applied to the field of intelligent question answering, the set of valid documents screened can provide a small amount of information to reduce the resource consumption of document analysis processing, and the documents in the set of valid documents are of high quality, which can also quickly and accurately provide answers, enhancing the user experience.

[0061] In the embodiments of the present disclosure, the search statement and each similar document can be spliced together to obtain splicing information corresponding to each similar document, and each splicing information can be input into the first large language model, so that the first large language model can evaluate the validity score of each similar document respectively and independently.

[0062] In some other embodiments, in order to further efficiently and accurately evaluate the validity score of the document, the first pre-trained large language model is used to estimate the validity score of the plurality of similar documents, which can also be implemented as shown in the following content: Figure 2

[0063] S201, splice the search statement and the plurality of similar documents together to obtain splicing information.

[0064] The plurality of similar documents can be spliced in order, for example, spliced according to the length of the similar documents. The similar documents can also be spliced according to the similarity between the similar documents and the search statement. The similar documents with high similarity can be sorted in the front, and the similar documents with low similarity can be sorted in the back, so that the first large language model can learn the features and knowledge of the similar documents with high similarity more preferentially, so as to improve the accuracy of estimating the validity score.

[0065] ​In other embodiments, the documents can also be sorted according to the authority of the documents. That is, the utility of each similar document can be initially determined, and the similar documents with high utility are sorted in the front. The utility of each similar document can be analyzed based on the aforementioned multiple dimensions (such as similarity, content quality, structural integrity, information accuracy, timeliness, and topic relevance), and then the effectiveness score is predicted by the first large language model.

[0066] S202, input the spliced information into the first large language model to obtain the feature information of each similar document output in sequence by the intermediate layer of the first large language model in the encoding stage.

[0067] The first large language model can be an open source LLM, such as ChatGLM (Generative Language Model, a large-scale language model based on pre-training). The first large language model extracts features based on an attention mechanism in the encoding stage, and the feature information of the intermediate layer in the embodiment of the present disclosure can better predict the effectiveness score.

[0068] S203, input the feature information of each similar document into the full connection layer to obtain the effectiveness score of each similar document.

[0069] In the embodiment of the present disclosure, by splicing the retrieval statement and the multiple similar documents together, a complete information set can be formed, and the processing efficiency of each similar document can be improved. In addition, the spliced information integrates the content of multiple similar documents, which can utilize the powerful semantic understanding and processing capability of the trained first large language model to extract key feature information from each similar document and learn the potential association between similar documents, so as to improve the evaluation effect of the effectiveness score. In addition, by predicting the effectiveness score through the full connection layer, the effectiveness of the similar documents can be quantitatively evaluated simply and quickly, and the document processing efficiency is improved.

[0070] In implementation, in order to improve the evaluation accuracy of the effectiveness, a document end symbol is arranged after each similar document in the spliced information; the document end symbol is used as a token for the first large language model to pass the prior knowledge of the similar document sorted in the front to the similar document sorted in the back.

[0071] As shown in Figure 3 An example of document spliced information is: <retrieval statement>, <document 1>, <end symbol 1>, <document 2>, <end symbol 2>, … <document n>, <end symbol n>. In the document spliced information, a document end symbol is arranged after each similar document; the document end symbol is used as a token for the first large language model to pass the prior knowledge down to the last similar document.

[0072] It can be understood that in the process of the document end character being processed as a token by the first large language model, the learning of the previous token and the prediction result of the validity score of each document are re-encoded and passed to the subsequent document. Therefore, the knowledge of each similar document is accumulated and passed back, which enables the first large language model to make predictions by combining rich prior knowledge when estimating the validity score of each similar document, thereby improving the accuracy of the validity score.

[0073] In implementation, the output features of any intermediate layer of the encoding stage of the first large language model can be selected to estimate the validity score of the similar document.

[0074] In order to improve the estimation accuracy of the validity score, in the embodiment of the present disclosure, when the encoding stage includes M encoding units, the intermediate layer is the Nth encoding unit, where M and N are positive integers, and N is within a preset range from M / 2 in the sequence number of all encoding units. That is, when the encoding stage includes M encoding units, if M is even, the M / 2th encoding unit or the encoding unit near the M / 2th encoding unit is preferentially selected as the intermediate layer of the output feature information, and if M is odd, the (M+1) / 2th encoding unit or the (M-1) / 2th encoding unit or the encoding unit near the two encoding units is preferentially selected to output the feature information of the similar document.

[0075] As shown in Figure 3 The assembled information is input into the first large language model, and when the first large language model encodes in the encoding stage, an embedding vector is obtained as feature information at the intermediate layer in the middle of the encoding stage, which is used to input the full connection layer to obtain the validity score.

[0076] Using the feature information output by the intermediate layer to estimate the validity score can make good use of the rich basic features and appropriate high-level features retained by the intermediate layer, and can better capture the feature patterns in the data of the similar document, thereby improving the estimation accuracy of the validity score.

[0077] The foregoing describes that the validity score of the similar document is estimated by using the first large language model, so that the effective document for the retrieval statement can be screened out. The invalid document is filtered out. This is conducive to analyzing the effective document set by using the second large language model to answer the retrieval statement.

[0078] It can be understood that when analyzing multiple documents, the document content is generally long. If all similar documents are input to the second large language model for analysis, the length of all documents will significantly affect the response speed of the second large language model and consume the processing resources of the second large language model. In addition, there may be documents or content in the multiple similar documents that cannot effectively answer the search statement, i.e., content irrelevant to the search statement. The existence of invalid documents not only interferes with the logical reasoning result of the second large language model, but also wastes the computing power resources of the second large language model and reduces the analysis performance of the second large language model.

[0079] Therefore, by using the scheme provided by the embodiments of the present disclosure, the similarity is used for rough screening first, and then the effective degree is used for fine screening, which can filter out truly useful documents from a large number of documents. On the one hand, invalid documents can be filtered out as much as possible to improve the performance of the second large language model, and on the other hand, the accuracy of the second large language model in answering the search statement can be improved.

[0080] In the process of processing multiple documents, as the length of the processed data increases, the large model will pay more attention to the content input at the beginning, and there may be a problem of low attention to intermediate information. Therefore, in the embodiments of the present disclosure, for the set of valid documents, each valid document in the set of valid documents is sorted in descending order of the validity score to obtain a document sequence. The prompt information is constructed based on the document sequence and the search statement. The prompt information is input into the second large language model to obtain the answer to the search statement.

[0081] For example, the validity score of the valid document 1 is higher than the validity score of the valid document 2, and the valid document 1 is preferentially input to the second large language model for analysis and understanding.

[0082] Therefore, by sorting each valid document according to the high and low of the validity score, the second large language model can preferentially and fully comb the documents with better effective degree, so that the second large language model can more effectively focus on the documents with high validity score and mine useful information to answer the search statement.

[0083] Based on the foregoing content, by finding documents similar to the search statement, the content related to the search statement can be effectively rough-screened. Although these documents have high similarity to the search statement, part of the content may not be actually helpful or helpful to answering the search statement. In order to ensure the processing efficiency of the document, a text similarity matching model can be used to select a specified number of documents similar to the search statement from the candidate document set to obtain multiple similar documents.

[0084] For example, m documents are screened out from the candidate document set as multiple similar document samples by using a text similarity matching model. The text similarity matching model can be a small-scale model, such as a BERT (Bidirectional Encoder Representation from Transformers, a pre-trained language representation model) model, which has lower requirements for computing power and resources and can quickly obtain similar documents.

[0085] In the embodiments of the present disclosure, the text similarity matching model can quickly identify a number of documents most relevant to the search statement based on less computing power, thereby providing an efficient information retrieval means to improve the efficiency of rough screening.

[0086] To sum up, in the embodiments of the present disclosure, a small-scale text similarity matching model is used to roughly screen multiple similar documents related to the search statement from a large number of documents. Then, a first large language model is used to evaluate the similar documents to obtain an effectiveness score of each similar document for the search statement. Further, the effective documents for the search statement are precisely screened from the roughly screened documents. This is conducive to reducing the length of the prompt information of the second large language model and improving the performance of the second large language model in answering the search statement. In addition, in order to fully focus on the content in the high effectiveness document, the effective documents are sorted in the order of the effectiveness score in the embodiments of the present disclosure, so that useful information can be preferentially mined to answer the search statement.

[0087] For the foregoing first large language model, a fine-tuning can be performed based on an open-source LLM in the embodiments of the present disclosure. Therefore, the present disclosure provides a training method of a large language model, as shown in Figure 4 The method can include the following steps:

[0088] S401, obtaining a search sample and multiple document samples associated with the search sample.

[0089] The search sample is a search statement sample; the document sample is a sample for responding to the search statement. The document sample can also be multiple samples similar to the search sample obtained based on a text matching model. The document sample can include documents that are not similar to the search sample as negative samples for training and learning.

[0090] S402, inputting the search sample and the multiple document samples into a to-be-trained large language model to obtain an effectiveness score of each document sample predicted by the to-be-trained large language model; for each document sample, the effectiveness score of the document sample is used to represent the effectiveness degree of responding to the search sample by using the document sample.

[0091] That is, the effectiveness score is used to measure whether the document sample is effective for the search statement, and to quantify the effectiveness. Thus, the effective documents can be further distinguished based on the effectiveness score.

[0092] S403, based on the effectiveness score of each document sample, the document sample is classified into two categories to determine whether the document sample can effectively meet the demand expressed by the search sample.

[0093] That is, the document sample is classified as effective or ineffective based on the effectiveness score.

[0094] S404, based on the binary classification result of each document sample and the classification true value, the loss value is determined.

[0095] For example, after being classified into two categories, document A is correctly classified as being able to effectively answer the search sample, while document B is incorrectly classified as being unable to effectively answer the search sample. In this case, the loss value can be calculated by comparing the predicted result (binary classification result) and the Groundtruth (classification true value). Cross-entropy loss function can be used to measure the loss between the predicted binary classification result and the classification true value.

[0096] In S405, based on the loss value, the to-be-trained large language model is fine-tuned, and when the convergence condition is met, the first large language model for predicting the effectiveness score is obtained.

[0097] In the embodiments of the present disclosure, the parameters of most modules of the to-be-trained large language model can be frozen, and only a small number of neural network layer parameters are adjusted. For example, only the parameters of the intermediate layer and the neural network layer before it that output the feature information in the encoding stage are adjusted, and the other parameters are frozen.

[0098] In the embodiments of the present disclosure, by inputting the search sample and the plurality of document samples into the to-be-trained large language model, during the training process, by comparing the difference between the binary classification result of the model and the actual classification true value, data labeling can be facilitated. Furthermore, by fine-tuning the to-be-trained large language model with the effectiveness score of each document sample, the to-be-trained large language model can accurately predict the effectiveness of the document sample to the search sample.

[0099] In the embodiments of the present disclosure, similar to the foregoing, the search sample and each document sample can be spliced together to obtain the splicing information corresponding to each document sample, thereby obtaining a plurality of splicing information. So that the to-be-trained large language model can process the splicing information of each document sample respectively to predict the effectiveness score of each document sample.

[0100] In another embodiment, in order to improve efficiency and the accuracy of predicting the effectiveness score, the search sample and the plurality of document samples are input into the to-be-trained large language model to obtain the effectiveness score of each document sample, and the specific implementation steps are as follows:Figure 5 As shown:

[0101] S501, the retrieved sample and multiple document samples are concatenated together to obtain the assembly information to be trained.

[0102] S502, input the assembly information to be trained into the large language model to be trained, and obtain the feature information of each document sample output sequentially by the intermediate layer of the large language model to be trained in the encoding stage.

[0103] S503, input the feature information of each document sample into the fully connected layer to obtain the validity score of each document sample.

[0104] In this embodiment, by concatenating the retrieval sample and multiple document samples, a complete information set can be formed, improving the processing efficiency of each document sample. Furthermore, the concatenated information integrates the content of multiple document samples, leveraging the powerful semantic understanding and processing capabilities of the large language model to be trained to extract key feature information from each document sample and mine potential relationships between learning documents, thereby improving the evaluation effect of the validity score. In addition, by predicting the validity score through a fully connected layer, the validity of document samples can be easily and quickly quantified, improving the learning efficiency of the large language model to be trained.

[0105] Similar to using the first-largest language model, the training phase also relies on similar operational procedures and techniques. This ensures a seamless transition from training to application, enabling the first-largest language model to efficiently handle various tasks. As described earlier, in the assembled information to be trained, a document end-of-file marker is added after each document sample; this marker serves as a token for the language model to process, transferring prior knowledge of earlier-ordered document samples to later-ordered document samples.

[0106] like Figure 3 As shown, an example of document assembly information is: <search statement>, <document 1>, <end symbol 1>, <document 2>, <end symbol 2>, ..., <document n>, <end symbol n>. In this document assembly information, a document end symbol is set after each document sample; the document end symbol serves as a token for the large language model to be trained, thus passing on prior knowledge until the last document sample.

[0107] In the embodiment of the present disclosure, the document end marker is used as a mark, and the learning of the previous token and the prediction result of the validity score of each document are re-encoded and passed to the subsequent document in the process of the document end marker being processed as a token by the large language model to be trained. Therefore, the knowledge of each document sample is accumulated and passed backward, which enables the large language model to be trained to make predictions by combining rich prior knowledge when estimating the validity scores of similar documents, so as to improve the efficiency of learning to evaluate the validity scores.

[0108] Similarly, in the case where the encoding stage includes M encoding units, the intermediate layer is the Nth encoding unit. Wherein, M and N are positive integers, and in the sequence numbers of all encoding units, N and M / 2 are within a preset range.

[0109] As shown in Figure 3 , the assembled information is input into the first large language model, and when the first large language model encodes in the encoding stage, an embedding vector is obtained as feature information at the intermediate layer in the middle of the encoding stage, which is used to input the fully connected layer to obtain the validity score.

[0110] In the embodiment of the present disclosure, the validity score is estimated by the feature information output by the intermediate layer, which can well utilize the rich basic features and effective high-level features retained by the intermediate layer, and can better capture the feature patterns in the document data, thereby improving the estimation accuracy of the validity score.

[0111] Continuing with Figure 3 as an example, the assembled information to be trained is input into the large language model to be trained, and the feature information of each document sample output by the intermediate layer of the large language model to be trained in the encoding stage is obtained (i.e., the embedding vector shown in Figure 3 ). Then, in order to enable the large language model to be trained to understand and learn more complex patterns and relationships, the intermediate features output by the intermediate layer are input into the fully connected layer to convert the intermediate features into more easily processed features, so as to obtain the validity score of each document sample. Finally, the validity score is used to predict whether the document sample is effective for answering the retrieval sample.

[0112] In implementation, only the parameters of the neural network and the fully connected layer of the intermediate layer and the n layers around the intermediate layer which need to output embedding features can be adjusted, and the parameters of other neural network layers in the large language model to be trained are frozen and unchanged.

[0113] In the embodiment of the present disclosure, the loss value is calculated by the loss function and the large language model to be trained is fine-tuned. For the classification of a single document, the loss function that can be used is shown in formula (1):

[0114]

[0115] In formula (1), Loss represents a loss value; neg represents an invalid sample; s i represents an effectiveness score of an invalid sample; s p represents an effectiveness score of a valid sample; and r represents a temperature coefficient.

[0116] In the fine-tuning stage, it is expected that s p is as large as possible, and s i is as small as possible. In formula (1), the final training can only obtain a relative value between the valid sample score and the invalid sample score. In a possible case, the positive sample score and the negative sample score are both negative, or the positive sample score is much larger than the negative sample score. In this way, it is not conducive to screening valid samples.

[0117] Further, in the embodiments of the present disclosure, the loss function for fine-tuning the large language model to be trained includes an optimization parameter term for constraining the effectiveness score of the valid sample to be greater than a preset threshold and constraining the effectiveness score of the invalid sample to be less than the preset threshold.

[0118] Through the preset threshold, not only the valid sample and the invalid sample can be distinguished, but also the valid sample can be conveniently screened with the help of the preset threshold, and the efficiency of the fine screening is improved.

[0119] In the embodiments of the present disclosure, when it is expected that the preset threshold is 0, the expression of the loss function for fine-tuning the large language model is as shown in formula (2):

[0120]

[0121] In formula (3), Loss represents a loss value; pos represents a valid sample; neg represents an invalid sample; s i represents an effectiveness score, and -s i is the optimization parameter term.

[0122] In the embodiments of the present disclosure, by adjusting the loss function, a suitable preset threshold is found, so that the first large language model obtained by training can obtain a reasonable effectiveness score S, so as to screen out valid documents.

[0123] In other embodiments, in order to further improve the recall rate of the valid sample, the expression of the loss function for fine-tuning the large language model is in the form of cumulative sum as shown in formula (3):

[0124]

[0125] In formula (4), Loss represents a loss value; Pos represents a valid sample; neg represents an invalid sample; s irepresents the effectiveness score; m is a constant; t is a temperature coefficient; -(s i -m) is the optimization parameter term, m is greater than 0.

[0126] In the embodiments of the present disclosure, the m parameter and the temperature coefficient are added to make the effectiveness score more distinguishable. The m can force the effectiveness score of the effective sample to be greater than or equal to m, and make the effectiveness score of the ineffective sample less than 0. Thus, the effective sample and the ineffective sample can be more effectively distinguished. If m is 0, some positive samples are allowed to be less than 0, but setting m forces the positive sample to be greater than m to effectively distinguish, thereby improving the recall rate of the effective sample and improving the quality of the answer.

[0127] In summary, the document processing method provided in the embodiments of the present disclosure is as follows:

[0128] (1) Training data preparation

[0129] Collect queries (search samples), for example, about 10w pieces of data are collected. Use a text similarity matching model, for example, BERT, to select m most relevant similar documents in the search library as a similar document set, and obtain the classification true value of whether each document in the similar document set is valid through labeling.

[0130] (2) Large language model training of document effectiveness

[0131] By selecting an open source large language model, the query and the similar document set are spliced as the input of the large language model to be trained. The specific splicing method is: query, <document 1>, <end symbol 1>, <document 2>, <end symbol 2>, … <document n>, <end symbol n>. Wherein, the <end symbol> exists as an end identifier of a document and is added after each document. For each <end symbol>, the large language model will get an embedding in the middle layer when encoding, and the obtained embedding is respectively connected to a fully connected layer to output a score (i.e. effectiveness score) for predicting whether the document is valid for answering the query.

[0132] (3) Multi-document question answering based on the large language model

[0133] For the query (search statement), use the similarity matching model to select m most relevant documents, splice the query and the m documents, and then predict the effectiveness score of each document. Only take the document with an effectiveness score greater than a preset threshold as the valid document. Then, sort according to the order of the effectiveness score from high to low, splice the query together, and input into the second large language model for analysis and reasoning to obtain the final answer to the search statement.

[0134] Based on the above document processing method, multiple documents can be merged, useful documents can be retrieved from a large amount of information more quickly for a second large language model to analyze and answer, so as to improve the accuracy and efficiency of the answer.

[0135] Based on the same technical concept, the disclosure also provides a document processing device 600, as shown in the accompanying drawings, comprising: Figure 6

[0136] The first screening module 601 is configured to obtain multiple similar documents similar to the retrieval statement;

[0137] The prediction module 602 is configured to determine the effectiveness score of the multiple similar documents; for each similar document, the effectiveness score of the similar document indicates the effectiveness of using the similar document to respond to the retrieval statement;

[0138] The second screening module 603 is configured to screen the effective documents from the multiple similar documents based on the effectiveness scores of the multiple similar documents, and obtain a set of effective documents.

[0139] In some embodiments, the prediction module is specifically configured to:

[0140] The first large language model is used to estimate the effectiveness score of the multiple similar documents.

[0141] In some embodiments, the prediction module comprises:

[0142] The first splicing unit is configured to splice the retrieval statement and the multiple similar documents together to obtain spliced information;

[0143] The first feature acquisition unit is configured to input the spliced information into the first large language model to obtain the feature information of each similar document output in sequence by the intermediate layer of the first large language model in the encoding stage;

[0144] The prediction unit is configured to input the feature information of each similar document into a fully connected layer respectively to obtain the effectiveness score of each similar document.

[0145] In some embodiments, in the spliced information, a document end symbol is arranged after each similar document; the document end symbol is used as a token for the first large language model to process, so as to pass the prior knowledge of the similar document arranged in front to the similar document arranged behind.

[0146] In some embodiments, when the encoding stage comprises M encoding units, the intermediate layer is the Nth encoding unit, wherein M and N are positive integers, and the difference between N and M / 2 is within a preset range in the serial numbers of all encoding units.

[0147] ​In some embodiments, further comprising:

[0148] a response module configured to analyze the set of valid documents using the second large language model to respond to the search statement.

[0149] In some embodiments, the response module comprises:

[0150] a sorting unit configured to sort the valid documents in the set of valid documents in descending order of the validity scores to obtain a document sequence;

[0151] a prompt construction unit configured to construct prompt information based on the document sequence and the search statement;

[0152] a response unit configured to input the prompt information into the second large language model to obtain an answer to the search statement.

[0153] In some embodiments, the first screening module is specifically configured to screen a specified number of documents similar to the search statement from the set of candidate documents using a text similarity matching model to obtain a plurality of similar documents.

[0154] Based on the same technical concept, the embodiments of the present disclosure further provide a training device 700 of a large language model, as shown in Figure 7 The training device 700 comprises:

[0155] an acquisition module 701 configured to acquire a search sample and a plurality of document samples associated with the search sample;

[0156] an estimation module 702 configured to input the search sample and the plurality of document samples into a large language model to be trained to obtain a validity score of each document sample predicted by the large language model to be trained; for each document sample, the validity score of the document sample is used to represent an effective degree of responding to the search sample using the document sample;

[0157] a classification module 703 configured to perform binary classification on each document sample based on the validity score of the document sample to determine whether the document sample can effectively meet the demand expressed by the search sample;

[0158] a determination module 704 configured to determine a loss value based on the binary classification result of each document sample and a classification true value;

[0159] an adjustment module 705 configured to fine-tune the large language model to be trained based on the loss value, and obtain a first large language model for predicting the validity score in the case of meeting a convergence condition.

[0160] In some embodiments, the estimation module comprises:

[0161] The second splicing unit is configured to splice the search sample and the plurality of document samples together to obtain training assembly information;

[0162] The second feature acquisition unit is configured to input the training assembly information into the large language model to be trained to obtain feature information of each document sample output in sequence by an intermediate layer of the large language model to be trained in an encoding stage.

[0163] The estimation unit is configured to input the feature information of each document sample into a full connection layer respectively to obtain an effectiveness score of each document sample.

[0164] In some embodiments, the document end symbol is provided after each document sample in the training assembly information; and the document end symbol is taken as a token for the large language model to be trained to pass prior knowledge of a document sample arranged in front to a document sample arranged behind.

[0165] In some embodiments, when the encoding stage includes M encoding units, the intermediate layer is an Nth encoding unit, where M and N are positive integers, and N and M / 2 are within a preset range in the sequence numbers of all the encoding units.

[0166] In some embodiments, the loss function for fine-tuning the large language model to be trained includes an optimization parameter term for constraining the effectiveness score of the valid sample to be greater than a preset threshold and constraining the effectiveness score of the invalid sample to be less than the preset threshold.

[0167] In some embodiments, the expression of the loss function is as follows:

[0168]

[0169] wherein Loss represents a loss value; pos represents a valid sample; neg represents an invalid sample; s i represents an effectiveness score, wherein -s i is an optimization parameter term.

[0170] In some embodiments, the expression of the loss function for fine-tuning the large language model is as follows:

[0171]

[0172] wherein Loss represents a loss value; pos represents a valid sample; neg represents an invalid sample; s i represents an effectiveness score; m is a constant; r is a temperature coefficient; and -(s i -m) is an optimization parameter term.

[0173] The specific functions and examples of the modules and sub-modules of the apparatuses in the embodiments of the present disclosure are described in the related description of the corresponding steps in the method embodiments, which will not be described here.

[0174] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0175] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0176] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0177] As shown in Figure 8 The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0178] Various components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc., an output unit 807, such as various types of displays, a speaker, etc., a storage unit 808, such as a magnetic disk, an optical disk, etc., and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0179] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 801 performs various methods and processes described above, such as the document processing method. For example, in some embodiments, the document processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the document processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the document processing method by any other suitable means, such as by means of firmware.

[0180] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0181] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0182] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0184] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0186] It should be understood that the various forms of flow shown above can be re-ordered, steps added or removed, etc. For example, the steps recited in the present disclosure can be performed in parallel, in series, in a different order, etc., as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0187] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A document processing method based on a large language model, comprising: obtaining a plurality of similar documents similar to a search statement; determining validity scores of the plurality of similar documents; for each similar document, the validity score of the similar document indicating a degree of effectiveness of using the similar document to respond to the search statement; based on the validity scores of the plurality of similar documents, filtering valid documents from the plurality of similar documents to obtain a valid document set; wherein the plurality of similar documents are concatenated together and input to a first large language model to obtain the validity score of each similar document, and a loss function of the first large language model obtained by training comprises: wherein, represents a loss value; represents a valid sample; represents an invalid sample; represents a validity score; is a constant; is a temperature coefficient; is an optimization parameter term, m is greater than 0; the optimization parameter term is used to constrain the validity score of the valid sample to be greater than a preset threshold, and constrain the validity score of the invalid sample to be less than the preset threshold; a document end symbol is set after each similar document; the document end symbol is provided as a token for the first large language model to process, so that the first large language model re-encodes the learning of the token ranked in front and the prediction result of the validity score of each document, and passes them to the similar document ranked behind.

2. The method of claim 1, wherein, The determination of the validity scores of the plurality of similar documents comprises: using a pre-trained first large language model to estimate the validity scores of the plurality of similar documents.

3. The method of claim 2, wherein, The use of the pre-trained first large language model to estimate the validity scores of the plurality of similar documents comprises: concatenating the search statement and the plurality of similar documents together to obtain spliced information; inputting the spliced information into the first large language model to obtain feature information of each similar document output in sequence by an intermediate layer of the first large language model in an encoding stage; inputting the feature information of each similar document into a fully connected layer respectively to obtain the validity score of each similar document.

4. The method of claim 3, wherein, In the case that the encoding stage comprises M encoding units, the intermediate layer is the Nth encoding unit, wherein M and N are positive integers, and the difference between N and M / 2 is within a preset range in the sequence numbers of all encoding units.

5. The method of claim 1, further comprising: using a second large language model to analyze the valid document set to answer the search statement.

6. The method of claim 5, wherein, The use of the second large language model to analyze the valid document set to answer the search statement comprises: sorting each valid document in the valid document set in descending order of validity score to obtain a document sequence; constructing prompt information based on the document sequence and the search statement; inputting the prompt information into the second large language model to obtain an answer to the search statement.

7. The method of any one of claims 1-6, wherein, The obtaining of the plurality of similar documents similar to the search statement comprises: using a text similarity matching model to filter a specified number of documents similar to the search statement from a candidate document set to obtain the plurality of similar documents.

8. A training method of a large language model, comprising: obtaining a search sample and a plurality of document samples associated with the search sample; concatenating the search sample and the plurality of document samples together and inputting them into a large language model to be trained to obtain validity scores of the plurality of document samples predicted by the large language model to be trained; The effectiveness score of each document sample is used to represent an effective degree of responding to the search sample by using the document sample; The effectiveness score of each document sample is used to represent an effective degree of responding to the search sample by using the document sample; The loss value is determined based on the binary classification result of each document sample and the classification true value; The loss value is determined based on the loss value of the trained large language model, and the first large language model for predicting the effectiveness score is obtained under the condition that the convergence condition is met. The loss function for fine-tuning the trained large language model includes: wherein, represents a loss value; represents a valid sample; represents an invalid sample; represents a validity score; is a constant; is a temperature coefficient; is an optimization parameter term, m is greater than 0; the optimization parameter term is used to constrain the validity score of the valid sample to be greater than a preset threshold value, and constrain the validity score of the invalid sample to be less than the preset threshold value; The document end symbol is set after each document sample; the document end symbol is used as a token for the trained large language model to process, so that the trained large language model re-encodes the learning of the token in the front of the sequence and the prediction result of the effectiveness score of each document sample, and delivers it to the document sample in the back of the sequence.

9. The method of claim 8, wherein, The search sample and the plurality of document samples are input into the trained large language model to obtain the effectiveness score of each document sample predicted by the trained large language model, including: The search sample and the plurality of document samples are spliced together to obtain the training splicing information; The feature information of each document sample is input into the full connection layer respectively to obtain the effectiveness score of each document sample. In the case that the encoding stage includes M encoding units, the intermediate layer is the Nth encoding unit, where M and N are positive integers, and N and M / 2 have a difference within a preset range in the sequence number of all encoding units.

10. The method of claim 9, wherein, 11. A document processing device based on a large language model, comprising: A first screening module is used to obtain a plurality of similar documents similar to a search statement; A prediction module is used to determine the effectiveness score of the plurality of similar documents; For each similar document, the effectiveness score of the similar document is used to represent the effective degree of responding to the search statement by using the similar document; A second screening module is used to screen effective documents from the plurality of similar documents based on the effectiveness score of the plurality of similar documents to obtain an effective document set; wherein the plurality of similar documents are spliced together and input into a first large language model to obtain the effectiveness score of each similar document, and the loss function of the first large language model is obtained by training, including: The document end symbol is set after each similar document; the document end symbol is used as a token for the first large language model to process, so that the first large language model re-encodes the learning of the token in the front of the sequence and the prediction result of the effectiveness score of each document, and delivers it to the similar document in the back of the sequence. wherein, represents a loss value; represents a valid sample; represents an invalid sample; represents a validity score; is a constant; is a temperature coefficient; is an optimization parameter term, m is greater than 0; the optimization parameter term is used to constrain the validity score of the valid sample to be greater than a preset threshold value, and constrain the validity score of the invalid sample to be less than the preset threshold value; The prediction module is specifically used to:

12. The apparatus of claim 11, wherein, Estimate the effectiveness score of the plurality of similar documents by using a pre-trained first large language model. The prediction module includes:

13. The apparatus of claim 12, wherein, ​ The first splicing unit is configured to splice the search statement and the plurality of similar documents together to obtain spliced information. The first feature acquisition unit is configured to input the spliced information into the first large language model to obtain feature information of each similar document output in sequence by an intermediate layer of the first large language model in an encoding stage. The prediction unit is configured to input the feature information of each similar document into a full connection layer respectively to obtain an effectiveness score of each similar document.

14. The apparatus of claim 13, wherein, In a case where the encoding stage includes M encoding units, the intermediate layer is an Nth encoding unit, where M and N are positive integers, and in the sequence numbers of all the encoding units, N is within a preset range from M / 2.

15. The apparatus of claim 11, further comprising: The answering module is configured to analyze the set of valid documents using a second large language model to answer the search statement.

16. The apparatus of claim 15, wherein, The answering module includes: The sorting unit is configured to sort each valid document in the set of valid documents in a descending order of the effectiveness score to obtain a document sequence. The prompt construction unit is configured to construct prompt information based on the document sequence and the search statement. The answering unit is configured to input the prompt information into the second large language model to obtain an answer to the search statement.

17. The apparatus of any of claims 11-16, wherein, The first screening module is specifically configured to screen a specified number of documents similar to the search statement from the candidate document set using a text similarity matching model to obtain the plurality of similar documents.

18. A training apparatus of a large language model, comprising: The acquisition module is configured to acquire a search sample and a plurality of document samples associated with the search sample. The estimation module is configured to input the search sample and the plurality of document samples spliced together into a to-be-trained large language model to obtain an effectiveness score of each document sample predicted by the to-be-trained large language model. For each document sample, the effectiveness score of the document sample is used to represent an effectiveness degree of responding to the search sample using the document sample. The classification module is configured to perform binary classification on each document sample based on the effectiveness score of the document sample to determine whether the document sample can effectively meet a demand expressed by the search sample. The determination module is configured to determine a loss value based on a binary classification result of each document sample and a classification true value. The adjustment module is configured to fine-tune the to-be-trained large language model based on the loss value, and obtain a first large language model for predicting the effectiveness score under a condition that a convergence condition is met. The loss function for fine-tuning the to-be-trained large language model includes: wherein, represents a loss value; represents a valid sample; represents an invalid sample; represents a validity score; is a constant; is a temperature coefficient; is an optimization parameter term, m is greater than 0; the optimization parameter term is used to constrain the validity score of the valid sample to be greater than a preset threshold value, and constrain the validity score of the invalid sample to be less than the preset threshold value; Each document sample is followed by a document end symbol; the document end symbol is used as a token for the to-be-trained large language model to re-encode learning of tokens in sequence and a prediction result of the effectiveness score of each document sample and pass the re-encoded result to a document sample in sequence.

19. The apparatus of claim 18, wherein, The estimation module includes: The second splicing unit is configured to splice the search sample and the plurality of document samples together to obtain to-be-trained spliced information. The second feature acquisition unit is configured to input the to-be-trained assembly information into the to-be-trained large language model to obtain feature information of each document sample output in sequence by an intermediate layer of the to-be-trained large language model in an encoding stage. The estimation unit is configured to input the feature information of each document sample into a full connection layer respectively to obtain an effectiveness score of each document sample.

20. The apparatus of claim 19, wherein, In the case that the encoding stage includes M encoding units, the intermediate layer is an Nth encoding unit, where M and N are positive integers, and in the serial numbers of all the encoding units, N and M / 2 are within a preset range. 21.An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication;wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-10. 23.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-10.

Images