Document processing method and apparatus, and electronic device and storage medium

Abstract

The present disclosure relates to the technical field of data processing, and in particular, to the technical fields of artificial intelligence, large language models, intelligent search, intelligent question answering, etc. Provided in the present disclosure are a document processing method and apparatus, and an electronic device and a storage medium. The specific implementation comprises: acquiring a plurality of similar documents similar to a retrieval statement; determining validity scores for the plurality of similar documents, wherein for each similar document, the validity score for the similar document is used for representing the degree of validity of using the similar document to respond to the retrieval statement; and on the basis of the validity scores for the plurality of similar documents, performing screening on the plurality of similar documents to obtain valid documents, so as to obtain a set of valid documents.

Description

Document processing method and device, electronic device, and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202410733042.X, filed on June 6, 2024, and entitled "Document processing method, device, electronic device, and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of data processing, and in particular to the technical field of artificial intelligence, large language model, intelligent search, intelligent question answering, and the like. BACKGROUND

[0003] Electronic documents, as carriers of information, facilitate people's learning, work, and entertainment. In addition, with the continuous development of various fields, a large amount of knowledge has been accumulated. This knowledge can also be in the form of documents for learning, research, and application.

[0004] While providing convenience, electronic documents also bring challenges to document analysis and processing due to the existence of a large amount of documents.

[0005] SUMMARY

[0006] The present disclosure provides a document processing method, device, electronic device, and storage medium.

[0007] According to an aspect of the present disclosure, a document processing method is provided, comprising:

[0008] obtaining a plurality of similar documents similar to a retrieval sentence;

[0009] determining an effectiveness score of the plurality of similar documents; for each similar document, the effectiveness score of the similar document is used to represent the effectiveness degree of responding to the retrieval sentence using the similar document;

[0010] based on the effectiveness scores of the plurality of similar documents, filtering out effective documents from the plurality of similar documents to obtain a set of effective documents.

[0011] According to an aspect of the present disclosure, a training method of a large language model is provided, comprising:

[0012] obtaining a retrieval sample and a plurality of document samples associated with the retrieval sample;

[0013] inputting the retrieval sample and the plurality of document samples into a to-be-trained large language model to obtain an effectiveness score of each document sample predicted by the to-be-trained large language model; for each document sample, the effectiveness score of the document sample is used to represent the effectiveness degree of responding to the retrieval sample using the document sample;

[0014] classify each document sample based on the effectiveness score of each document sample, to determine whether each document sample can effectively meet the demand expressed by the search sample;

[0015] determine a loss value based on the classification result of each document sample and the classification true value;

[0016] fine-tune the large language model to be trained based on the loss value, and obtain a first large language model for predicting the effectiveness score under the condition that a convergence condition is met.

[0017] According to another aspect of the present disclosure, a document processing apparatus is provided, comprising:

[0018] a first screening module configured to obtain a plurality of similar documents similar to the search statement;

[0019] a prediction module configured to determine an effectiveness score of each similar document in the plurality of similar documents, the effectiveness score of each similar document indicating an effectiveness of responding to the search statement using the similar document;

[0020] a second screening module configured to screen an effective document from the plurality of similar documents based on the effectiveness score of each similar document in the plurality of similar documents, to obtain a set of effective documents.

[0021] According to another aspect of the present disclosure, a large language model training apparatus is provided, comprising:

[0022] an acquisition module configured to acquire a search sample and a plurality of document samples associated with the search sample;

[0023] an estimation module configured to input the search sample and the plurality of document samples into a large language model to be trained, to obtain an effectiveness score of each document sample predicted by the large language model to be trained, the effectiveness score of each document sample indicating an effectiveness of responding to the search sample using the document sample;

[0024] a classification module configured to classify each document sample based on the effectiveness score of each document sample, to determine whether each document sample can effectively meet the demand expressed by the search sample;

[0025] a determination module configured to determine a loss value based on the classification result of each document sample and a classification true value;

[0026] an adjustment module configured to fine-tune the large language model to be trained based on the loss value, and obtain a first large language model for predicting the effectiveness score under the condition that a convergence condition is met.

[0027] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0028] at least one processor; and

[0029] a memory in communication with the at least one processor; wherein

[0030] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the methods in the embodiments of the present disclosure.

[0031] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform any of the methods in the embodiments of the present disclosure.

[0032] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements any of the methods in the embodiments of the present disclosure.

[0033] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure.

[0035] FIG. 1 is a flow diagram of a document processing method according to an embodiment of the present disclosure.

[0036] FIG. 2 is a flow diagram of estimating an effectiveness score in a plurality of similar documents using a first large language model according to an embodiment of the present disclosure.

[0037] FIG. 3 is a flow diagram of processing text feature information according to an embodiment of the present disclosure.

[0038] FIG. 4 is a flow diagram of a training method of a large language model according to an embodiment of the present disclosure.

[0039] FIG. 5 is a training flow diagram of a large language model predicting an effectiveness score according to an embodiment of the present disclosure.

[0040] FIG. 6 is a structural diagram of a document processing apparatus according to an embodiment of the present disclosure.

[0041] FIG. 7 is a structural diagram of a training apparatus of a large language model according to an embodiment of the present disclosure.

[0042] FIG. 8 is a block diagram of an electronic device for implementing a document processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are cited by way of example only. The drawings and description are intended to convey a thorough understanding of the present disclosure, and merely in explanation. Consequently, various changes and modifications can be made to the embodiments described and illustrated herein, without departing from the scope of the present disclosure. Also, for the purpose of clarity and a concise description, descriptions of well-known functions and constructions can be omitted from the following detailed description.

[0044] The terms "first", "second", and the like in the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a particular order or sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0045] In the face of a large number of documents, intelligent recommendation and analysis systems can filter some documents to meet user needs. For example, in the case of a question statement, an intelligent question and answer application can automatically analyze at least one document to obtain an answer to the question statement and answer the question statement. However, the quality of the document affects the accuracy of the answer content. Especially in the application demand of knowledge retrieval, effective strategies are needed to filter out effective documents to respond to users. In view of this, a document processing method is proposed in the embodiments of the present disclosure. This method can not only be applied to intelligent question and answer applications based on documents, but also be applied to recommendation systems based on documents.

[0046] As shown in FIG. 1, a flowchart of a document processing method provided in the embodiments of the present disclosure is shown, including the following content:

[0047] S101, a plurality of similar documents similar to a retrieval statement are obtained.

[0048] The literature, news reports, blog articles, research reports, papers, etc. related to the retrieval statement can be searched on academic databases, library resources or the Internet.

[0049] By finding documents similar to the retrieval statement, the content related to the retrieval statement can be effectively screened out. However, these related contents are only considered from the perspective of similarity, and cannot directly describe whether they can effectively answer the retrieval statement. Therefore, the embodiments of the present disclosure propose an effectiveness evaluation method to fine-screen the plurality of similar documents.

[0050] In S102, the effectiveness scores of the plurality of similar documents are determined; for each similar document, the effectiveness score of the similar document is used to represent the effectiveness of the similar document in responding to the retrieval statement.

[0051] It can also be understood that the similar document meets the degree of satisfaction of the requirement expressed by the search statement.

[0052] In the embodiments of the present disclosure, the effectiveness score can directly measure whether a document can effectively meet the requirement of the search statement. Taking a question statement as an example, the effectiveness score can quantify to what extent the document can effectively answer the question statement, and how effective the answer is.

[0053] For example, if the search statement is about the application of artificial intelligence technology in imaging analysis, a document with a high effectiveness score may contain detailed case studies, the latest research results, authoritative opinions and analysis processes. While a document with a low effectiveness score may only contain an introduction to some basic concepts or outdated information.

[0054] In S103, a plurality of effective documents are selected from the plurality of similar documents based on the effectiveness scores of the plurality of similar documents, to obtain an effective document set.

[0055] In implementation, similar documents with effectiveness scores greater than a preset threshold can be selected from the plurality of similar documents to obtain the effective document set. Alternatively, the top n similar documents can be selected to obtain the effective document set. Alternatively, similar documents ranked in the top n and having effectiveness scores greater than the preset threshold can be selected to obtain the effective document set.

[0056] The preset threshold is a preset score limit for distinguishing which documents are effective for the search statement and which documents are ineffective. In order to accurately select effective documents, the preset threshold can be set based on experience or training. It can be understood that the higher the effectiveness score, the more likely the document is considered to effectively answer the search statement.

[0057] In the embodiments of the present disclosure, by finding similar documents to the search statement, documents related to the content of the search statement can be obtained, thereby achieving coarse screening of massive documents. Further, it is proposed to further screen the coarse screened documents from the effectiveness of answering the search statement, thereby selecting effective documents that can effectively meet the requirements of the search statement, to obtain an effective document set. Therefore, the scheme provided by the embodiments of the present disclosure can further improve the quality of the screened documents, and ultimately improve the efficiency of retrieving effective documents from massive documents.

[0058] In some possible embodiments, the effectiveness score can be calculated based on a series of criteria, including but not limited to the content quality, structural integrity, information accuracy and timeliness, and theme relevance to the search statement of the document. Therefore, in implementation, a neural network model can be used to analyze the information in these dimensions to obtain the effectiveness score of the similar document.

[0059] In some other possible embodiments, the first pre-trained large language model can also be used to estimate the validity scores of the plurality of similar documents.

[0060] The large language model (LLM) refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model can process various natural language tasks such as text classification, question answering, dialogue, etc.

[0061] The large language model has strong knowledge background and logical reasoning ability. The first large language model in the embodiments of the present disclosure is pre-trained to meet the demand of evaluating the validity of the document. Through the first large language model, documents that are more relevant to the search statement and are more likely to contain accurate answers can be screened out, thereby improving the quality of document retrieval and being able to accurately answer the search statement.

[0062] In addition, when the set of valid documents retrieved is applied to the field of intelligent question answering, the set of valid documents screened can provide less information to reduce the resource consumption of document analysis processing, and the documents in the set of valid documents are of high quality, which can also quickly and accurately provide answers, enhancing the user experience.

[0063] In the embodiments of the present disclosure, the search statement and each similar document can be spliced together to obtain splicing information corresponding to each similar document, and each splicing information can be input into the first large language model, so that the first large language model can evaluate the validity score of each similar document respectively and independently.

[0064] In some other embodiments, in order to further efficiently and accurately evaluate the validity score of the document, the first pre-trained large language model is used to estimate the validity score of the plurality of similar documents, which can also be implemented as shown in FIG. 2:

[0065] S201, splice the search statement and the plurality of similar documents together to obtain splicing information.

[0066] The plurality of similar documents can be spliced in order, for example, spliced according to the length of the similar documents. The similar documents can also be spliced according to the similarity between the similar documents and the search statement. The similar documents with high similarity can be sorted in front, and the similar documents with low similarity can be sorted in back, so that the first large language model can learn the features and knowledge of the similar documents with high similarity more preferentially, so as to improve the accuracy of estimating the validity score.

[0067] In other embodiments, the documents can also be sorted according to the authority of the documents. That is, the utility of each similar document can be initially determined, and the similar documents with high utility are sorted in the front. The utility of each similar document can be analyzed based on the aforementioned multiple dimensions (such as similarity, content quality, structural integrity, information accuracy, timeliness, and topic relevance), and then the effectiveness score is predicted by the first large language model.

[0068] S202, input the assembled information into the first large language model to obtain the feature information of each similar document output in sequence by the intermediate layer of the first large language model in the encoding stage.

[0069] The first large language model can be an open source LLM, such as ChatGLM (Generative Language Model, a large-scale language model based on pre-training). The first large language model extracts features based on an attention mechanism in the encoding stage, and the feature information of the intermediate layer in the embodiment of the present disclosure can better predict the effectiveness score.

[0070] S203, input the feature information of each similar document into the full connection layer to obtain the effectiveness score of each similar document.

[0071] In the embodiment of the present disclosure, by splicing the retrieval statement and the multiple similar documents together, a complete information set can be formed, and the processing efficiency of each similar document can be improved. In addition, the spliced information integrates the content of multiple similar documents, which can utilize the powerful semantic understanding and processing capability of the trained first large language model to extract key feature information from each similar document and learn the potential association between similar documents, so as to improve the evaluation effect of the effectiveness score. In addition, by predicting the effectiveness score through the full connection layer, the effectiveness of the similar documents can be quantitatively evaluated simply and quickly, and the document processing efficiency can be improved.

[0072] In implementation, in order to improve the evaluation accuracy of the effectiveness, a document end symbol is arranged after each similar document in the assembled information; the document end symbol is used as a token for the first large language model to process, so as to pass the prior knowledge of the similar document sorted in the front to the similar document sorted in the back.

[0073] As shown in FIG. 3, an example of the document assembled information is: <retrieval statement>, <document 1>, <end symbol 1>, <document 2>, <end symbol 2>, … <document n>, <end symbol n>. In the document assembled information, a document end symbol is arranged after each similar document; the document end symbol is used as a token for the first large language model to analyze, so as to pass the prior knowledge down to the last similar document.

[0074] It can be understood that in the process of the document end character being processed as a token by the first large language model, the learning of the previous token and the prediction result of the validity score of each document are re-encoded and passed to the subsequent document. Therefore, the knowledge of each similar document is accumulated and passed back, which enables the first large language model to make predictions by combining rich prior knowledge when estimating the validity score of each similar document, thereby improving the accuracy of the validity score.

[0075] In implementation, the output features of any intermediate layer of the encoding stage of the first large language model can be selected to estimate the validity score of the similar document.

[0076] To improve the estimation accuracy of the validity score, in the embodiment of the present disclosure, when the encoding stage includes M encoding units, the intermediate layer is the Nth encoding unit, where M and N are positive integers, and N is within a preset range from M / 2 in the sequence numbers of all encoding units. That is, when the encoding stage includes M encoding units, if M is even, the M / 2th encoding unit or the encoding unit near the M / 2th encoding unit is preferentially selected as the intermediate layer of the output feature information, and if M is odd, the (M+1) / 2th encoding unit or the (M-1) / 2th encoding unit or the encoding unit near the two encoding units is preferentially selected to output the feature information of the similar document.

[0077] As shown in FIG. 3, the assembly information is input into the first large language model, and when the first large language model encodes in the encoding stage, an embedding vector is obtained as feature information at the intermediate layer in the middle of the encoding stage, which is used to input the fully connected layer to obtain the validity score.

[0078] Using the feature information output by the intermediate layer to estimate the validity score can make good use of the rich basic features and appropriate high-level features retained by the intermediate layer, and can better capture the feature patterns in the data of the similar document, thereby improving the estimation accuracy of the validity score.

[0079] The foregoing describes that the validity score of the similar document is estimated by using the first large language model, so that the effective document for the retrieval statement can be screened out. The invalid document is filtered out. This is conducive to analyzing the effective document set by using the second large language model to answer the retrieval statement.

[0080] It can be understood that when analyzing multiple documents, the document content is generally long. If all similar documents are input to the second large language model for analysis, the length of all documents will significantly affect the response speed of the second large language model and consume the processing resources of the second large language model. In addition, there may be documents or content in the multiple similar documents that cannot effectively answer the search statement, that is, content irrelevant to the search statement. The existence of invalid documents not only interferes with the logical reasoning result of the second large language model, but also wastes the computing power resources of the second large language model and reduces the analysis performance of the second large language model.

[0081] Therefore, by using the scheme provided by the embodiments of the present disclosure, the similarity is used for rough screening first, and then the effective degree is used for fine screening, which can filter out truly useful documents from a large number of documents. On the one hand, invalid documents can be filtered out as much as possible to improve the performance of the second large language model, and on the other hand, the accuracy of the second large language model in answering the search statement can be improved.

[0082] In the process of processing multiple documents, as the length of the processed data increases, the large model will pay more attention to the content input at the beginning, and there may be a problem of low attention to intermediate information. Therefore, in the embodiments of the present disclosure, for the set of valid documents, each valid document in the set of valid documents is sorted in descending order of the validity score to obtain a document sequence. The prompt information is constructed based on the document sequence and the search statement. The prompt information is input into the second large language model to obtain the answer to the search statement.

[0083] For example, the validity score of the valid document 1 is higher than the validity score of the valid document 2, and the valid document 1 is preferentially input to the second large language model for analysis and understanding.

[0084] Therefore, by sorting each valid document according to the high and low of the validity score, the second large language model can preferentially and fully comb the documents with good effective degree, so that the second large language model can more effectively focus on the documents with high validity score and mine useful information to answer the search statement.

[0085] Based on the foregoing content, by finding documents similar to the search statement, the content related to the search statement can be effectively rough-screened. Although these documents have high similarity with the search statement, part of the content may not be actually helpful or helpful to answering the search statement. In order to ensure the processing efficiency of the document, a text similarity matching model can be used to select a specified number of documents similar to the search statement from the candidate document set to obtain multiple similar documents.

[0086] For example, m documents are screened out from the candidate document set as multiple similar document samples by using a text similarity matching model. The text similarity matching model can be a small-scale model, such as a BERT (Bidirectional Encoder Representation from Transformers, a pre-trained language representation model) model, which has lower requirements for computing power and resources and can quickly obtain similar documents.

[0087] In the embodiments of the present disclosure, the text similarity matching model can quickly identify a number of documents most relevant to the search statement based on less computing power, thereby providing an efficient information retrieval means to improve the efficiency of rough screening.

[0088] To sum up, in the embodiments of the present disclosure, a small-scale text similarity matching model can be used to roughly screen multiple similar documents related to the search statement from a large number of documents. Then, a first large language model is used to evaluate the similar documents to obtain an effectiveness score of each similar document for the search statement. Further, the effective documents for the search statement are precisely screened from the roughly screened documents. This is conducive to reducing the length of the prompt information of the second large language model and improving the performance of the second large language model in answering the search statement. In addition, in order to fully focus on the content in the high effectiveness document, the effective documents are sorted in the order of the effectiveness score in the embodiments of the present disclosure, so that useful information can be preferentially mined to answer the search statement.

[0089] For the foregoing first large language model, a fine-tuning can be performed based on an open-source LLM in the embodiments of the present disclosure. Therefore, the present disclosure provides a training method of a large language model, as shown in FIG. 4, which can include the following steps:

[0090] S401, obtaining a search sample and multiple document samples associated with the search sample.

[0091] The search sample is the search statement sample; the document sample is the sample for responding to the search statement. The document sample can also be multiple samples similar to the search sample obtained based on the text matching model. The document sample can include documents that are not similar to the search sample as negative samples for training and learning.

[0092] S402, inputting the search sample and the multiple document samples into a to-be-trained large language model to obtain an effectiveness score of each document sample predicted by the to-be-trained large language model; for each document sample, the effectiveness score of the document sample is used to represent the effectiveness degree of responding to the search sample by using the document sample.

[0093] That is, the effectiveness score is used to measure whether the document sample is effective for the search statement, and to quantify the effectiveness. Thus, the effective documents can be further distinguished based on the effectiveness score.

[0094] S403, based on the effectiveness score of each document sample, the document sample is classified into two categories to determine whether the document sample can effectively meet the demand expressed by the search sample.

[0095] That is, based on the effectiveness score, the document sample is classified as valid or invalid.

[0096] S404, based on the binary classification result of each document sample and the classification true value, the loss value is determined.

[0097] For example, after being classified, document A is correctly classified as being able to effectively answer the search sample, while document B is incorrectly classified as being unable to effectively answer the search sample. In this case, the loss value can be calculated by comparing the predicted result (binary classification result) and the Groundtruth (classification true value). Cross-entropy loss function can be used to measure the loss between the predicted binary classification result and the classification true value.

[0098] In S405, based on the loss value, the large language model to be trained is fine-tuned, and when the convergence condition is met, the first large language model for predicting the effectiveness score is obtained.

[0099] In the embodiments of the present disclosure, the parameters of most modules of the large language model to be trained can be frozen, and only a small number of neural network layer parameters are adjusted. For example, only the parameters of the intermediate layer and the neural network layer before it that output the feature information in the encoding stage are adjusted, and the other parameters are frozen.

[0100] In the embodiments of the present disclosure, by inputting the search sample and the plurality of document samples into the large language model to be trained, during the training process, by comparing the difference between the binary classification result of the model and the actual classification true value, data labeling can be facilitated. Furthermore, by fine-tuning the effectiveness score of each document sample, the large language model to be trained can accurately predict the effectiveness of the document sample to the search sample.

[0101] In the embodiments of the present disclosure, similar to the foregoing, the search sample and each document sample can be spliced together to obtain the splicing information corresponding to each document sample, thereby obtaining a plurality of splicing information. So that the large language model to be trained can process the splicing information of each document sample respectively to predict the effectiveness score of each document sample.

[0102] In another embodiment, in order to improve the efficiency and the accuracy of the effectiveness score, the retrieval sample and the plurality of document samples are input into the large language model to be trained to obtain the effectiveness score of each document sample, and the specific implementation steps are shown in FIG. 5:

[0103] S501, the retrieval sample and the plurality of document samples are spliced together to obtain the to-be-trained spliced information.

[0104] S502, the to-be-trained spliced information is input into the large language model to be trained to obtain the feature information of each document sample output in sequence by the intermediate layer of the large language model to be trained in the encoding stage.

[0105] S503, the feature information of each document sample is input into the full connection layer respectively to obtain the effectiveness score of each document sample.

[0106] In the embodiment of the present disclosure, by splicing the retrieval sample and the plurality of document samples together, a complete information set can be formed, and the processing efficiency of each document sample can be improved. In addition, the spliced information integrates the content of the plurality of document samples, which can utilize the powerful semantic understanding and processing capability of the large language model to be trained to extract the key feature information in each document sample and mine the potential association between documents, so as to improve the evaluation effect of the effectiveness score. In addition, by predicting the effectiveness score through the full connection layer, the effectiveness of the document sample can be quantitatively evaluated simply and quickly, so as to improve the learning efficiency of the large language model to be trained.

[0107] Similarly, the training stage also relies on similar operation processes and technical means. This ensures seamless connection from training to use, so that the first large language model can efficiently process various tasks. As described above, in the to-be-trained spliced information, a document end symbol is arranged after each document sample; the document end symbol is used as a token for the large language model to be trained to pass the prior knowledge of the document sample sorted first to the document sample sorted later.

[0108] As shown in FIG. 3, an example of the document spliced information is: <retrieval sentence>, <document 1>, <end symbol 1>, <document 2>, <end symbol 2>, … <document n>, <end symbol n>. In the to-be-trained document spliced information, a document end symbol is arranged after each document sample; the document end symbol is used as a token for the large language model to be trained to pass the prior knowledge down until the last document sample.

[0109] In the embodiment of the present disclosure, the document end marker is used as a mark, and the learning of the previous token and the prediction result of the validity score of each document are re-encoded and passed to the subsequent document in the process of the document end marker being processed as a token by the large language model to be trained. Therefore, the knowledge of each document sample is accumulated and passed back, which enables the large language model to be trained to make predictions by combining rich prior knowledge when estimating the validity scores of similar documents, thereby improving the efficiency of learning to evaluate the validity scores.

[0110] Similarly, in the case where the encoding stage includes M encoding units, the intermediate layer is the Nth encoding unit. Wherein M and N are positive integers, and in the sequence numbers of all encoding units, N and M / 2 are within a preset range.

[0111] As shown in FIG. 3, the assembled information is input into the first large language model, and when the first large language model encodes in the encoding stage, an embedding vector is obtained as feature information at the intermediate layer in the middle of the encoding stage, which is used to input the fully connected layer to obtain the validity score.

[0112] In the embodiment of the present disclosure, the validity score is estimated by the feature information output by the intermediate layer, which can well utilize the rich basic features and effective high-level features retained by the intermediate layer, and can better capture the feature patterns in the document data, thereby improving the estimation accuracy of the validity score.

[0113] Continuing with FIG. 3 as an example, the assembled information to be trained is input into the large language model to be trained, and the feature information (i.e., the embedding vector shown in FIG. 3) of each document sample output by the intermediate layer of the large language model to be trained in the encoding stage is obtained. Then, in order to enable the large language model to be trained to understand and learn more complex patterns and relationships, the intermediate features output by the intermediate layer are input into the fully connected layer to convert the intermediate features into more easily processed features, so as to obtain the validity score of each document sample. Finally, the validity score is used to predict whether the document sample is effective for answering the retrieval sample.

[0114] In implementation, only the parameters of the neural network and the fully connected layer of the intermediate layer and the n layers around it which need to output embedding features can be adjusted, and the parameters of other neural network layers in the large language model to be trained are frozen and unchanged.

[0115] In the embodiment of the present disclosure, the loss value is calculated by the loss function and the large language model to be trained is fine-tuned. For the classification of a single document, the loss function that can be used is shown in formula (1):

[0116] In formula (1), Loss represents the loss value; neg represents the invalid sample; s represents the valid sample.i s represents the effectiveness score of the invalid sample; s p s represents the effectiveness score of the valid sample; r represents the temperature coefficient.

[0117] s is expected to be as large as possible p s is expected to be as large as possible i s is expected to be as small as possible. In formula (1), the final training can only obtain a relative value between the valid sample score and the invalid sample score. In one possible case, the positive sample score and the negative sample score are both negative, or the positive sample score is much larger than the negative sample score. In this way, it is not conducive to screening valid samples.

[0118] Further, in the embodiments of the present disclosure, the loss function for fine-tuning the large language model to be trained includes an optimization parameter term for constraining the effectiveness score of the valid sample to be greater than a preset threshold and constraining the effectiveness score of the invalid sample to be less than the preset threshold.

[0119] Through the preset threshold, not only can the valid sample and the invalid sample be distinguished, but also the valid sample can be conveniently screened out with the help of the preset threshold, and the efficiency of the fine screening is improved.

[0120] In the embodiments of the present disclosure, when the expected preset threshold is 0, the expression of the loss function for fine-tuning the large language model is as shown in formula (2):

[0121] In formula (2), Loss represents the loss value; pos represents the valid sample; neg represents the invalid sample; s i s represents the effectiveness score, and -s i is the optimization parameter term.

[0122] In the embodiments of the present disclosure, by adjusting the loss function to find a suitable preset threshold, the first large language model obtained by training can obtain a reasonable effectiveness score S, so as to screen out valid documents.

[0123] In other embodiments, in order to further improve the recall rate of the valid sample, the expression of the loss function for fine-tuning the large language model is in the form of cumulative sum as shown in formula (3):

[0124] In formula (3), Loss represents the loss value; pos represents the valid sample; neg represents the invalid sample; s i s represents the effectiveness score; m is a constant; t is a temperature coefficient; and -(s i -m) is the optimization parameter term, and m is greater than 0.

[0125] In the embodiments of the present disclosure, an m parameter and a temperature coefficient are added to make the effectiveness score more distinguishable. The m can force the effectiveness score of the effective sample to be greater than or equal to m, and make the effectiveness score of the ineffective sample less than 0. Thus, the effective sample and the ineffective sample can be more effectively distinguished. If m is 0, some positive samples are allowed to be less than 0, but setting m forces the positive sample to be greater than m to effectively distinguish, thereby improving the recall rate of the effective sample and improving the quality of the answer.

[0126] In summary, the document processing method provided in the embodiments of the present disclosure is as follows:

[0127] (1) Training data preparation

[0128] Collect queries (search samples), for example, about 10w pieces of data are collected. Use a text similarity matching model, for example, BERT, to select m most relevant similar documents in the search library as a similar document set for each query, and obtain the classification true value of whether each document in the similar document set is effective by labeling.

[0129] (2) Large language model training of document effectiveness

[0130] By selecting an open source large language model, the query and the similar document set are spliced as the input of the large language model to be trained. The specific splicing method is: query, <document 1>, <end symbol 1>, <document 2>, <end symbol 2>, … <document n>, <end symbol n>. Wherein, the <end symbol> is an end identifier of a document, which is added after each document. For each <end symbol>, the large language model will get an embedding in the middle layer when encoding, and the obtained embedding is respectively connected to a fully connected layer to output a score (i.e. effectiveness score) for predicting whether the document is effective for answering the query.

[0131] (3) Multi-document question answering based on large language model

[0132] For the query (search statement), use the similarity matching model to select m most relevant documents, splice the query and the m documents, and then predict the effectiveness score of each document. Only take the document with an effectiveness score greater than a preset threshold as an effective document. Then, sort according to the order of effectiveness score from high to low, splice with the query and input into the second large language model for analysis and reasoning to obtain the final answer to the search statement.

[0133] Based on the above document processing method, multiple documents can be merged, useful documents can be retrieved from a large amount of information more quickly for a second large language model to analyze and answer, so as to improve the accuracy and efficiency of the answer.

[0134] Based on the same technical concept, the disclosure also provides a document processing device 600, as shown in FIG. 6, comprising:

[0135] The first screening module 601 is configured to obtain a plurality of similar documents similar to the retrieval statement;

[0136] The prediction module 602 is configured to determine the effectiveness score of the plurality of similar documents; for each similar document, the effectiveness score of the similar document indicates the effectiveness of using the similar document to respond to the retrieval statement;

[0137] The second screening module 603 is configured to filter out effective documents from the plurality of similar documents based on the effectiveness scores of the plurality of similar documents, and obtain a set of effective documents.

[0138] In some embodiments, the prediction module is specifically configured to:

[0139] The first large language model is used to estimate the effectiveness score of the plurality of similar documents.

[0140] In some embodiments, the prediction module comprises:

[0141] The first splicing unit is configured to splice the retrieval statement and the plurality of similar documents together to obtain spliced information;

[0142] The first feature acquisition unit is configured to input the spliced information into the first large language model to obtain feature information of each similar document output in sequence by an intermediate layer of the first large language model in an encoding stage;

[0143] The prediction unit is configured to input the feature information of each similar document into a fully connected layer respectively to obtain the effectiveness score of each similar document.

[0144] In some embodiments, in the spliced information, a document end symbol is arranged after each similar document; the document end symbol is used as a token for the first large language model to process, so as to pass the prior knowledge of the similar document arranged in front to the similar document arranged behind.

[0145] In some embodiments, when the encoding stage comprises M encoding units, the intermediate layer is the Nth encoding unit, where M and N are positive integers, and the difference between N and M / 2 is within a preset range in the serial numbers of all encoding units.

[0146] In some embodiments, the document processing device further comprises:

[0147] The answering module is configured to analyze the set of valid documents by using the second large language model to answer the search statement.

[0148] In some embodiments, the answering module comprises:

[0149] The sorting unit is configured to sort the valid documents in the set of valid documents in descending order of the validity scores to obtain a document sequence.

[0150] The prompt construction unit is configured to construct prompt information based on the document sequence and the search statement.

[0151] The answering unit is configured to input the prompt information into the second large language model to obtain an answer to the search statement.

[0152] In some embodiments, the first screening module is specifically configured to screen a specified number of documents similar to the search statement from the set of candidate documents by using a text similarity matching model to obtain a plurality of similar documents.

[0153] Based on the same technical concept, the embodiments of the present disclosure further provide a training device 700 of a large language model, as shown in FIG. 7, comprising:

[0154] The acquisition module 701 is configured to acquire a search sample and a plurality of document samples associated with the search sample.

[0155] The estimation module 702 is configured to input the search sample and the plurality of document samples into a large language model to be trained to obtain a validity score of each document sample predicted by the large language model to be trained, wherein the validity score of the document sample is used to represent an effective degree of responding to the search sample by using the document sample.

[0156] The classification module 703 is configured to perform binary classification on each document sample based on the validity score of the document sample to determine whether the document sample can effectively meet the demand expressed by the search sample.

[0157] The determination module 704 is configured to determine a loss value based on the binary classification result of each document sample and a classification true value.

[0158] The adjustment module 705 is configured to fine-tune the large language model to be trained based on the loss value, and obtain a first large language model for predicting the validity score under the condition that a convergence condition is met.

[0159] In some embodiments, the estimation module comprises:

[0160] The second splicing unit is configured to splice the search sample and the plurality of document samples together to obtain training spliced information.

[0161] The second feature acquisition unit is used to input the assembly information to be trained into the large language model to be trained, and obtain the feature information of each document sample output sequentially by the intermediate layer of the large language model to be trained in the encoding stage.

[0162] The estimation unit is used to input the feature information of each document sample into the fully connected layer to obtain the validity score of each document sample.

[0163] In some embodiments, in the assembly information to be trained, a document end marker is set after each document sample; the document end marker is used as a token for the large language model to be trained to process, so as to pass the prior knowledge of the document samples that are ordered first to the document samples that are ordered later.

[0164] In some embodiments, where the encoding stage includes M encoding units, the intermediate layer is the Nth encoding unit, where M and N are both positive integers, and the difference between N and M / 2 in the sequence number of all encoding units is within a preset range.

[0165] In some embodiments, the loss function for fine-tuning the large language model to be trained includes optimization parameter terms for constraining the validity scores of valid samples to be greater than a preset threshold and constraining the validity scores of invalid samples to be less than a preset threshold.

[0166] In some embodiments, the expression for the loss function is:

[0167] Where Loss represents the loss value; pos represents the valid samples; neg represents the invalid samples; s i This represents the validity score, where -s i To optimize the parameter terms.

[0168] In some embodiments, the expression for the loss function of fine-tuning the large language model is:

[0169] Where Loss represents the loss value; pos represents the valid samples; neg represents the invalid samples; s i Indicates the effectiveness score; m is a constant; r is the temperature coefficient; -(s) i -m) is an optimization parameter.

[0170] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0171] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0172] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0173] FIG. 8 shows a schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0174] As shown in FIG. 8, the device 800 includes a computing unit 801 that can perform various suitable actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded into a random access memory (RAM) 803 from a storage unit 808. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0175] Various components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0176] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the document processing method. For example, in some embodiments, the document processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the document processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the document processing method by any other suitable means, such as by means of firmware.

[0177] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0178] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0179] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0181] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0182] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0183] It should be understood that the various forms of flow shown above can be re-ordered, steps added or removed, etc. For example, the steps recited in the present disclosure can be performed in parallel, in series, in a different order, etc., as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0184] The above detailed description does not constitute a limitation of the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A document processing method, comprising: obtaining a plurality of similar documents similar to a search statement; determining validity scores of the plurality of similar documents; for each similar document, the validity score of the similar document indicating an effectiveness degree of responding to the search statement by using the similar document; based on the validity scores of the plurality of similar documents, filtering valid documents from the plurality of similar documents to obtain a valid document set.

2. The method of claim 1, wherein, The determining of the validity scores of the plurality of similar documents comprises: using a pre-trained first large language model to estimate the validity scores of the plurality of similar documents.

3. The method of claim 2, wherein, The using of the pre-trained first large language model to estimate the validity scores of the plurality of similar documents comprises: concatenating the search statement and the plurality of similar documents to obtain spliced information; inputting the spliced information into the first large language model to obtain feature information of each similar document output in sequence by an intermediate layer of the first large language model in an encoding stage; inputting the feature information of each similar document into a fully connected layer respectively to obtain the validity score of each similar document.

4. The method of claim 3, wherein, In the spliced information, a document end symbol is arranged after each similar document; the document end symbol is used as a token for processing by the first large language model to pass prior knowledge of a similar document arranged in front to a similar document arranged behind.

5. The method of claim 3, wherein, In a case where the encoding stage comprises M encoding units, the intermediate layer is an Nth encoding unit, wherein M and N are positive integers, and in the sequence numbers of all the encoding units, N is within a preset range from M / 2.

6. The method of claim 1, further comprising: using a second large language model to analyze the valid document set to answer the search statement.

7. The method of claim 6, wherein, The using of the second large language model to analyze the valid document set to answer the search statement comprises: sorting each valid document in the valid document set in an order from high to low according to the validity score to obtain a document sequence; constructing prompt information based on the document sequence and the search statement; inputting the prompt information into the second large language model to obtain an answer to the search statement.

8. The method of any one of claims 1-7, wherein, The obtaining of the plurality of similar documents similar to the search statement comprises: using a text similarity matching model to filter a specified number of documents similar to the search statement from a candidate document set to obtain the plurality of similar documents.

9. A training method of a large language model, comprising: obtaining a search sample and a plurality of document samples associated with the search sample; inputting the search sample and the plurality of document samples into a large language model to be trained to obtain validity scores of the plurality of document samples predicted by the large language model to be trained; for each document sample, the validity score of the document sample indicating an effectiveness degree of responding to the search sample by using the document sample; based on the validity scores of the plurality of document samples, performing binary classification on each document sample to determine whether the document sample can effectively meet a demand expressed by the search sample; based on a binary classification result of each document sample and a classification true value, determining a loss value. Fine-tune the to-be-trained large language model based on the loss value, and in a case where a convergence condition is met, Obtain a first large language model for predicting the effectiveness score.

10. The method of claim 9, wherein, The to-be-trained large language model is input with the retrieval sample and the plurality of document samples to obtain the effectiveness score of each document sample predicted by the to-be-trained large language model, including: The retrieval sample and the plurality of document samples are spliced together to obtain to-be-trained assembly information; The to-be-trained assembly information is input into the to-be-trained large language model to obtain feature information of each document sample sequentially output by an intermediate layer of the to-be-trained large language model in an encoding stage; The feature information of each document sample is respectively input into a fully connected layer to obtain the effectiveness score of each document sample.

11. The method of claim 10, wherein, In the to-be-trained assembly information, a document end symbol is arranged after each document sample; the document end symbol is used as a token for processing by the to-be-trained large language model to pass prior knowledge of a document sample ranked first to a document sample ranked second.

12. The method of claim 10, wherein, In a case where the encoding stage includes M encoding units, the intermediate layer is an Nth encoding unit, where M and N are positive integers, and in the serial numbers of all the encoding units, N and M / 2 are within a preset range.

13. The method of any one of claims 9-12, wherein, The loss function for fine-tuning the to-be-trained large language model includes an optimization parameter term for constraining the effectiveness score of an effective sample to be greater than a preset threshold and constraining the effectiveness score of an ineffective sample to be less than the preset threshold.

14. The method of claim 13, wherein, An expression of the loss function is: wherein Loss represents a loss value; pos represents valid samples; neg represents invalid samples; s i represents a validity score, wherein -s i is the optimization parameter term.

15. The method of claim 13, wherein, The expression of the loss function of fine-tuning the large language model is: wherein Loss represents a loss value; pos represents valid samples; neg represents invalid samples; s i represents a validity score; m is a constant; r is a temperature coefficient; -(s i m) is the optimization parameter term.

16. A document processing apparatus, comprising: a first screening module configured to obtain a plurality of similar documents similar to a retrieval statement; a prediction module configured to determine effectiveness scores of the plurality of similar documents; for each similar document, the effectiveness score of the similar document is used to represent an effective degree of responding to the retrieval statement by using the similar document; a second screening module configured to screen effective documents from the plurality of similar documents based on the effectiveness scores of the plurality of similar documents to obtain a set of effective documents.

17. The apparatus of claim 16, wherein, The prediction module is specifically configured to: estimate the effectiveness scores of the plurality of similar documents by using a pre-trained first large language model.

18. The apparatus of claim 17, wherein, The prediction module includes: a first splicing unit configured to splice the retrieval statement and the plurality of similar documents together to obtain assembly information; a first feature acquisition unit configured to input the assembly information into the first large language model to obtain feature information of each similar document sequentially output by an intermediate layer of the first large language model in an encoding stage; a prediction unit configured to input the feature information of each similar document into a fully connected layer to obtain the effectiveness score of each similar document. In the assembly information, a document end symbol is arranged after each similar document; the document end symbol is used as a token for processing by the first large language model to pass prior knowledge of a similar document ranked first to a similar document ranked second.

19. The apparatus of claim 18, wherein, In a case where the encoding stage includes M encoding units, the intermediate layer is an Nth encoding unit, where M and N are positive integers, and in the serial numbers of all the encoding units, N and M / 2 are within a preset range.

20. The apparatus of claim 18, wherein, ​ 21. The apparatus of claim 16, further comprising: a response module configured to analyze the set of valid documents using a second large language model to respond to the search statement.

22. The apparatus of claim 21, wherein, The response module comprises: a sorting unit configured to sort each valid document in the set of valid documents in order of validity score from high to low to obtain a document sequence; a prompt construction unit configured to construct prompt information based on the document sequence and the search statement; a response unit configured to input the prompt information into the second large language model to obtain an answer to the search statement.

23. The apparatus of any of claims 16-22, wherein, The first screening module is specifically configured to screen a specified number of documents similar to the search statement from the candidate document set using a text similarity matching model to obtain the plurality of similar documents.

24. A training apparatus of a large language model, comprising: an acquisition module configured to acquire a search sample and a plurality of document samples associated with the search sample; an estimation module configured to input the search sample and the plurality of document samples into a to-be-trained large language model to obtain a validity score of each document sample predicted by the to-be-trained large language model; for each document sample, the validity score of the document sample is used to represent the effectiveness of responding to the search sample using the document sample; a classification module configured to perform binary classification on each document sample based on the validity score of the document sample to determine whether the document sample can effectively meet the demand expressed by the search sample; a determination module configured to determine a loss value based on the binary classification result of each document sample and a classification true value; an adjustment module configured to fine-tune the to-be-trained large language model based on the loss value, and obtain a first large language model for predicting a validity score when a convergence condition is met.

25. The apparatus of claim 24, wherein, The estimation module comprises: a second splicing unit configured to splice the search sample and the plurality of document samples together to obtain to-be-trained spliced information; a second feature acquisition unit configured to input the to-be-trained spliced information into the to-be-trained large language model to obtain feature information of each document sample output in sequence by an intermediate layer of the to-be-trained large language model in an encoding stage; an estimation unit configured to input the feature information of each document sample into a fully connected layer respectively to obtain the validity score of each document sample.

26. The apparatus of claim 25, wherein, In the to-be-trained spliced information, a document end symbol is arranged after each document sample; the document end symbol is used as a token for processing by the to-be-trained large language model to pass prior knowledge of a document sample arranged in front to a document sample arranged behind.

27. The apparatus of claim 25, wherein, In a case where the encoding stage comprises M encoding units, the intermediate layer is an Nth encoding unit, where M and N are positive integers, and N and M / 2 have a difference within a preset range in the sequence numbers of all the encoding units.

28. The apparatus of any one of claims 24-27, wherein, The loss function for fine-tuning the to-be-trained large language model comprises an optimization parameter term for constraining the validity score of a valid sample to be greater than a preset threshold and constraining the validity score of an invalid sample to be less than the preset threshold.

29. The apparatus of claim 28, wherein, An expression of the loss function is: wherein Loss represents a loss value; pos represents valid samples; neg represents invalid samples; s i represents a validity score, wherein -s i is the optimization parameter term.

30. The apparatus of claim 28, wherein, The expression of the loss function of fine-tuning the large language model is: wherein Loss represents a loss value; pos represents valid samples; neg represents invalid samples; s i represents a validity score; m is a constant; r is a temperature coefficient; and -(s i m) is the optimization parameter term.

31. An electronic device, comprising: at least one processor; and ​ a memory in communication with the at least one processor; wherein the memory has stored instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

32. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, the computer instructions are for causing the computer to perform the method of any one of claims 1-15.

33. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-15.

Images