Training and reasoning method and device of a reliable large language model for archives

Abstract

The present invention provides a trustworthy large language model training, reasoning method and device for archives, the training method comprising: converting multiple text-based archive data into tensor-based data to obtain a first trustworthy training set; based on multiple text-based archive data, using an effective model architecture to build a pre-trained language model; using the first trustworthy training set to perform unsupervised learning on the pre-trained language model, and performing supervised fine-tuning training on the result to obtain a second trustworthy large language model; according to user preference ranking between the same user query content and different candidate outputs and a preset template set, a second trustworthy training set is obtained; using the second trustworthy training set to perform unsupervised learning on the second trustworthy large language model, and performing supervised training on the result to obtain a trustworthy reward model; based on the trustworthy reward model, a trustworthy large language model is obtained. The present invention can adapt to archive reasoning task scenarios more accurately and professionally.

Description

Technical Field

[0001] The present invention relates to the field of language processing technology, and in particular to a reliable large language model training and reasoning method and device for archives. Background Art

[0002] Large Language Model (LLM) is a natural language processing technology based on deep learning. It can generate fluent responses similar to humans by learning a large amount of language data. This technology is widely used in machine translation, text generation, question-answering systems and other fields, and has become one of the important research directions in the field of natural language processing. Although large language models are widely used, their generation capabilities are still limited when dealing with professional fields or tasks such as archival science. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a reliable large language model training and reasoning method and device for archives, which can adapt to archive reasoning task scenarios more accurately and professionally.

[0004] To achieve the above-mentioned purpose, an embodiment of the present invention provides a method for training a credible large language model for archives, the method comprising: preprocessing different types of archive data to obtain archive data in multiple text forms; converting the multiple text-based archive data into data in tensor form to obtain a first credible training set; constructing a pre-trained language model based on the multiple text-based archive data using an effective model architecture; performing unsupervised learning on the pre-trained language model using the first credible training set to obtain a first credible large language model; performing supervised fine-tuning training on the first credible large language model using the first credible training set to obtain a second credible large language model; obtaining a second credible training set according to user preference ranking between the same user query content and different candidate outputs and a preset template set; performing unsupervised learning on the second credible large language model using the second credible training set to obtain a third credible large language model; performing supervised training on the third credible large language model using the second credible training set to obtain a credible reward model; and training the second credible large language model under reinforcement learning based on artificial feedback based on the credible reward model to obtain a credible large language model.

[0005] Preferably, converting the multiple text-form archival data into tensor-form data to obtain a first trusted training set includes: performing the following operations for each text-form archival data: dividing the text-form archival data into sub-units; adding special sub-units, including a start sub-unit and an end sub-unit; constructing a vocabulary to map each sub-unit to a unique integer ID; converting the text-form archival data into corresponding data in the form of integer ID; creating input and output, wherein the input is data in the form of integer ID excluding the end sub-unit, and the output is data in the form of integer ID excluding the start sub-unit; converting the input and the output into tensor form; and composing the input and output of each text-form archival data into the first trusted training set.

[0006] Preferably, based on the trusted reward model, training the second trusted large language model under reinforcement learning based on artificial feedback to obtain the trusted large language model includes: determining the score returned by the trusted reward model for user queries and each alternative output; based on the score returned by the trusted reward model, training the second trusted large language model under reinforcement learning based on artificial feedback to obtain the trusted large language model.

[0007] An embodiment of the present invention also provides a trusted large language model reasoning method for archives, which uses the trusted large language model for archives trained as described above, and the method includes: constructing a first trusted reasoning data set according to an input task provided by a user; decomposing the first trusted prompt data set into multiple sub-units; encoding the multiple sub-units to convert each sub-unit into a vector form; adjusting the encoded data to the input format of the trusted large language model to input the trusted large language model for reasoning; and decoding the reasoning result using a corresponding encoded decoding method to obtain the reasoning result.

[0008] Preferably, the method further comprises automatically evaluating the inference result using the BLEU indicator.

[0009] An embodiment of the present invention also provides a trusted large language model training device for archives, the device comprising: a data textualization unit, a first trusted training set construction unit, a pre-trained language model construction unit, a first trusted large language model construction unit, a second trusted large language model construction unit, a second trusted training set construction unit, a third trusted large language model construction unit, a trusted reward model construction unit and a reinforcement learning unit, wherein the data textualization unit is used to pre-process different types of archive data to obtain multiple archive data in text form; the first trusted training set construction unit is used to convert the multiple archive data in text form into tensor form data to obtain a first trusted training set; the pre-trained language model construction unit is used to construct a pre-trained language model based on the multiple archive data in text form using an effective model architecture; the first trusted large language model construction unit is used to use the first trusted training set to train the pre-trained language model The first credible large language model is used for unsupervised learning on the second credible large language model using the first credible training set to obtain a first credible large language model; the second credible large language model construction unit is used for using the first credible training set to perform supervised fine-tuning training on the first credible large language model to obtain a second credible large language model; the second credible training set construction unit is used for obtaining a second credible training set according to user preference ranking between the same user query content and different candidate outputs and a preset template set; the third credible large language model construction unit is used for using the second credible training set to perform unsupervised learning on the second credible large language model to obtain a third credible large language model; the credible reward model construction unit is used for using the second credible training set to perform supervised training on the third credible large language model to obtain a credible reward model; the reinforcement learning unit is used for training the second credible large language model under reinforcement learning based on artificial feedback based on the credible reward model to obtain a credible large language model.

[0010] Preferably, the first trusted training set construction unit is used to: perform the following operations for each text-form archival data: divide the text-form archival data into sub-units; add special sub-units, including a start sub-unit and an end sub-unit; construct a vocabulary to map each sub-unit to a unique integer ID; convert the text-form archival data into data in the form of corresponding integer IDs; create inputs and outputs, wherein the input is data in the form of integer IDs excluding the end sub-unit, and the output is data in the form of integer IDs excluding the start sub-unit; convert the inputs and outputs into tensor forms; and compose the inputs and outputs of each text-form archival data into the first trusted training set.

[0011] Preferably, the reinforcement learning unit is used to: determine the score returned by the trusted reward model for the user query and each alternative output; based on the score returned by the trusted reward model, train the second trusted large language model under reinforcement learning based on artificial feedback to obtain the trusted large language model.

[0012] An embodiment of the present invention also provides a trusted large language model reasoning device for archives, which uses the trusted large language model for archives described above, and includes: a first trusted reasoning data set construction unit, a sub-unit decomposition unit, a sub-unit encoding unit, an reasoning unit and a decoding unit, wherein the first trusted reasoning data set construction unit is used to construct a first trusted reasoning data set according to an input task provided by a user; the sub-unit decomposition unit is used to decompose the first trusted prompt data set into multiple sub-units; the sub-unit encoding unit is used to encode the multiple sub-units to convert each sub-unit into a vector form; the reasoning unit is used to adjust the encoded data to the input format of the trusted large language model to input the trusted large language model for reasoning; the decoding unit is used to decode the reasoning result using the corresponding encoding decoding method to obtain the reasoning result.

[0013] Preferably, the device also includes an automatic evaluation unit, which is used to automatically evaluate the inference result using the BLEU indicator.

[0014] Through the above technical scheme, the use of a trusted large language model training, reasoning method and device for archives provided by the embodiment of the present invention will greatly improve the work efficiency of archives or archives departments, help the dataization and intelligence of archives, and form a one-stop archive resource training and reasoning service based on the characteristics of archive resources, and adapt to archive reasoning task scenarios more accurately and professionally.

[0015] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0017] Figure 1 It is a flow chart of a reliable large language model training method for archives provided by an embodiment of the present invention;

[0018] Figure 2 is a flow chart of converting data in tensor form provided by an embodiment of the present invention;

[0019] Figure 3It is a flow chart of a reliable large language model reasoning method for archives provided by an embodiment of the present invention;

[0020] Figure 4 It is a structural schematic diagram of a reliable large language model training device for archives provided by an embodiment of the present invention;

[0021] Figure 5 It is a schematic diagram of the structure of a reliable large language model reasoning device for archives provided by an embodiment of the present invention;

[0022] Figure 6 It is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0023] Description of Reference Numerals

[0024] 1-Data textualization unit, 2-First trusted training set construction unit, 3-Pre-trained language model construction unit, 4-First trusted large language model construction unit, 5-Second trusted large language model construction unit, 6-Second trusted training set construction unit, 7-Third trusted large language model construction unit, 8-Trusted reward model construction unit, 9-Reinforcement learning unit, 10-First trusted reasoning data set construction unit, 11-Subunit decomposition unit, 12-Subunit encoding unit, 13-Reasoning unit, 14-Decoding unit, 15-Automatic evaluation unit, 16-Trusted reference display unit DETAILED DESCRIPTION

[0025] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0026] Figure 1 FIG. 1 is a flow chart of a reliable large language model training method for archives provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0027] Step S101, preprocessing different types of archive data to obtain multiple archive data in text form;

[0028] Specifically, digital archive resources are mostly managed by archives and other departments. They go through preprocessing (selection, inventory, registration, sorting, cleaning, and repair), digital processing / conversion (converting archive information recorded on traditional carriers into information resources represented in digital form through analog-to-digital conversion technology and equipment), information processing (checking, compression, edge removal, decontamination, and noise removal), information organization (creating catalogs and indexes), information storage, information services, and information maintenance. The digitization process follows a strict management mechanism, and the data itself is credible. Digitally stored archive resources come in a variety of forms, including images, audio, video, text, and scans. Data in different forms need to undergo corresponding preprocessing operations before they can participate in the training of large language models.

[0029] The sample data used in this embodiment include images, audio, video, text, scanned copies. The number and length of different types of sample data are not limited and can be determined according to actual needs, test scale, etc. For example, for scenes with high image requirements, the number of image samples can be relatively large. The non-text type data in the sample data is converted into text type data using different methods, and then subsequent processing is performed. Among them, the image data set first defines the image preprocessing function for size adjustment, data enhancement, image decoding and other processing, and uses convolutional neural networks (such as ResNet, VGG, etc.) to complete feature extraction; video data includes a series of continuous image frames, so it is necessary to define an image preprocessing function for image frame recognition, image size adjustment, image decoding and other processing, and use convolutional neural networks (such as ResNet, VGG, etc.) to complete feature extraction; audio data exists in the form of digital signals, and first it is necessary to define an audio preprocessing function for audio format conversion, audio length standardization, noise removal and other processing, and then use audio feature representation methods (such as Mel-frequency cepstral coefficients (Mel-frequency cepstral coefficients, MFCCs) and Mel spectrum graph (Mel spectrogram) to process the audio data, and then use pooling strategies or convolutional neural networks (such as CNN, etc.) to extract features; the scanned data includes image format and PDF format. The preprocessing method of the image format is the same as the above. The PDF format needs to use the existing PDF library (such as Python's PyPDF2 library, Pdfplumber, etc.) to read the PDF file, convert each page of the PDF file into image format, and then perform image decoding, brightness adjustment, OCR recognition, proofreading and verification, feature extraction and other operations.

[0030] Step S102, converting the plurality of text-based archive data into tensor-based data to obtain a first credible training set;

[0031] Specifically, different types of data are converted into text data after the above strict preprocessing operations. Next, all text data needs to be further tokenized, vocabulary built, special tokens added, integer IDs converted, input and output created, and data vectorized. The following steps are included, such as Figure 2 As shown: For each text file data, the following operations are performed:

[0032] Step S201, dividing the archive data in text form into sub-units;

[0033] Specifically, Tokenization refers to dividing text data into a series of smaller sub-units, which are usually called "tokens". Tokens can be words, subwords, characters, or other smaller text units. They are the basic units for models to understand and process text. The data after Tokenization is divided into multiple tokens, and each token has its corresponding index.

[0034] Step S202, adding special subunits, including a start subunit and an end subunit;

[0035] Specifically, some special tokens, such as "start-of-sentence" and "end-of-sentence", need to be added to the training data to mark the beginning and end of a sentence.

[0036] Step S203, constructing a vocabulary table to map each subunit to a unique integer ID;

[0037] Specifically, each Token establishes its corresponding index. A vocabulary is constructed based on the Tokenization result. The vocabulary size is usually a hyperparameter, which needs to be determined based on the scale of the training data and the complexity of the task. Each token is then mapped to a unique integer ID.

[0038] Step S204, converting the archive data in text form into data in the form of corresponding integer ID;

[0039] Specifically, the integer ID of each token is arranged in the order of arrangement of each token, so that data in the form of integer ID can be obtained.

[0040] Step S205, creating an input and an output, wherein the input is data in the form of an integer ID excluding an end subunit, and the output is data in the form of an integer ID excluding a start subunit;

[0041] Specifically, in order to perform supervised training, it is necessary to create input and output pairs based on text data. For example, the text sequence is shifted by one bit to obtain the input sequence, which is used to predict the output of the next token.

[0042] Step S206, converting the input and the output into a tensor form;

[0043] Specifically, all tokens represented by integer IDs are converted into word embedding vectors to form tensors that can be input into the model (a tensor is a multidimensional array, which can be a 0-dimensional scalar, a 1-dimensional vector, a 2-dimensional matrix, or a higher-dimensional array).

[0044] Step S207, the input and output of each text-based archive data are combined into the first credible training set.

[0045] Specifically, for ease of understanding, an example is provided in the embodiment of the present invention. It is understandable that the example is only provided for ease of understanding, but does not limit the present invention.

[0046] Suppose there are the following two sentences as training data:

[0047] A: The National Archives Administration has a wealth of information.

[0048] B: Is the admission score for the archives major high?

[0049] The first step is to tokenize the data in this article:

[0050] A:["country","archives bureau","information","very","rich","."]

[0051] B:["File","Major","Admission","Score","High","Is it","?"]

[0052] The second step is to add special tokens "start" and "end":

[0053] A:["start","State","Archives Bureau","Information","Very","Rich",".","end"]

[0054] B:["start","File","Major","Admission","Score","High","Is it",".","end"]

[0055] The third step is to build a vocabulary that maps each token to a unique integer ID:

[0056] {"country":0,"archives":1,"information":2,"very":3,"rich":4,"."":5,"archives":6,"major":7,"admission":8,"score":9,"high":10,"?"":11,"start":12,"end":13}

[0057] The fourth step is to convert each token into the corresponding integer ID data:

[0058] A:[12,0,1,2,3,4,5,13]

[0059] B:[12,6,7,8,9,10,11,5,13]

[0060] Step 5: Create input and output:

[0061] A input: [12,0,1,2,3,4,5], output: [0,1,2,3,4,5,13]

[0062] B input: [12,6,7,8,9,10,11,5], output: [6,7,8,9,10,11,5,13]

[0063] Step 6: Data vectorization:

[0064] Assuming the word embedding dimension is 4, the vocabulary is as follows (this is just an example, the actual word embedding may be a more complex vector)

[0065] Vocabulary size: 14, word embedding dimension: 4

[0066] [0.1,0.2,0.3,0.4], #corresponding to integer ID 0

[0067] [0.5,0.6,0.7,0.8], #corresponding to integer ID 1

[0068] [0.9,0.1,0.2,0.3], #corresponding to integer ID 2

[0069] [0.2,0.3,0.4,0.5], #corresponding to integer ID 3

[0070] [0.6,0.7,0.8,0.9], #corresponding to integer ID 4

[0071] [0.5,0.4,0.3,0.2], #corresponding to integer ID 5

[0072] [0.8,0.9,0.1,0.2], #corresponding to integer ID 6

[0073] [0.3,0.4,0.5,0.6], #corresponding to integer ID 7

[0074] [0.7,0.8,0.9,0.1], #corresponding to integer ID 8

[0075] [0.4,0.5,0.6,0.7], #corresponding to integer ID 9

[0076] [0.1,0.2,0.3,0.4], #corresponding integer ID 10

[0077] [0.5,0.6,0.7,0.8], #corresponding to integer ID 11

[0078] [0.9,0.1,0.2,0.3], #corresponding to integer ID 12

[0079] [0.2,0.3,0.4,0.5], #corresponding to integer ID 13

[0080] A input tensor representation:

[0081] [[0.9,0.1,0.2,0.3], #corresponding to integer ID 12

[0082] [0.1,0.2,0.3,0.4], #corresponding to integer ID 0

[0083] [0.5,0.6,0.7,0.8], #corresponding to integer ID 1

[0084] [0.9,0.1,0.2,0.3], #corresponding to integer ID 2

[0085] [0.2,0.3,0.4,0.5], #corresponding to integer ID 3

[0086] [0.6,0.7,0.8,0.9], #corresponding to integer ID 4

[0087] [0.5,0.4,0.3,0.2] #corresponding to integer ID 5]

[0088] The tensor output by A is represented as:

[0089] [[0.1,0.2,0.3,0.4], #corresponding to integer ID 0

[0090] [0.5,0.6,0.7,0.8], #corresponding to integer ID 1

[0091] [0.9,0.1,0.2,0.3], #corresponding to integer ID 2

[0092] [0.2,0.3,0.4,0.5], #corresponding to integer ID 3

[0093] [0.6,0.7,0.8,0.9], #corresponding to integer ID 4

[0094] [0.5,0.4,0.3,0.2], #corresponding to integer ID 5

[0095] [0.8,0.9,0.1,0.2] #corresponding to integer ID 13]

[0096] The tensor representation of B input is:

[0097] [[0.9,0.1,0.2,0.3], #corresponding to integer ID 12

[0098] [0.8,0.9,0.1,0.2], #corresponding to integer ID 6

[0099] [0.3,0.4,0.5,0.6], #corresponding to integer ID 7

[0100] [0.7,0.8,0.9,0.1], #corresponding to integer ID 8

[0101] [0.4,0.5,0.6,0.7], #corresponding to integer ID 9

[0102] [0.1,0.2,0.3,0.4], #corresponding integer ID 10

[0103] [0.5,0.6,0.7,0.8] #corresponding integer ID 11]

[0104] The tensor output of B is:

[0105] [[0.8,0.9,0.1,0.2], #corresponding to integer ID 6

[0106] [0.3,0.4,0.5,0.6], #corresponding to integer ID 7

[0107] [0.7,0.8,0.9,0.1], #corresponding to integer ID 8

[0108] [0.4,0.5,0.6,0.7], #corresponding to integer ID 9

[0109] [0.1,0.2,0.3,0.4], #corresponding integer ID 10

[0110] [0.5,0.6,0.7,0.8], #corresponding to integer ID 11

[0111] [0.9,0.1,0.2,0.3], #corresponding to integer ID 5

[0112] [0.2,0.3,0.4,0.5] #corresponding to integer ID 13]

[0113] In this way, by mapping the integer ID to the corresponding word embedding vector, the tensor representation of the embedding vector is obtained. Here, each row represents the embedding vector of a token, and each integer ID in A and B corresponds to a token in the vocabulary. After the above rigorous preprocessing operations, the tensor form data obtained can be used for pre-training of the large language model constructed later, that is, this step constructs the first trusted training set.

[0114] Step S103, constructing a pre-trained language model using an effective model architecture based on the multiple text-based archival data;

[0115] Specifically, through the large language modeling task, based on the multiple text-based archival data, the Transformer architecture is used to build a pre-trained language model. The Transformer uses a method called "self-attention mechanism" to calculate the representation of the output sequence by weighted aggregation of each element in the input sequence. Of course, other commonly used effective model architectures, such as RNN, CNN, etc., can also be selected according to actual needs.

[0116] Step S104, performing unsupervised learning on the pre-trained language model using the first credible training set to obtain a first credible large language model;

[0117] Specifically, it is necessary to define pre-training tasks such as language modeling, masked language modeling, next sentence prediction, vocabulary replacement, etc. to guide the model to learn the characteristics of the training set, update the parameters of the model through the back propagation algorithm, and use indicators such as perplexity, recall rate, and accuracy to evaluate the performance of the pre-trained language model. Based on the first trusted training set, the constructed pre-trained language model is subjected to unsupervised learning to learn the language knowledge and text structure of the first trusted training set, and the pre-trained model, namely the first trusted large language model, is completed.

[0118] Step S105, using the first credible training set to perform supervised fine-tuning training on the first credible large language model to obtain a second credible large language model;

[0119] Specifically, since the first credible training set is credible, the first credible training set is used as a fine-tuning training set to perform supervised fine-tuning training (Supervised Fine-Tuning, SFT) on the pre-trained first credible large language model, and a second credible large language model can be obtained. SFT technology is a supervised learning method for natural language processing (NLP), and a general large language model is obtained by training with the first credible training sample. With the help of SFT technology, the huge time required to train the model from scratch is avoided. It only uses a smaller scale (the smaller scale here is relative to the large scale in constructing the first credible training set) first credible training set to fine-tune the first credible large language model that has been pre-trained using the large-scale unlabeled first credible training set, and then the second credible large language model can be obtained.

[0120] Step S106, obtaining a second credible training set according to the user preference ranking between the same user query content and different candidate outputs and a preset template set;

[0121] Specifically, the second credible training set is constructed by user preference ranking and preset template set between the same user query content and different candidate outputs. For each user's input task, a sample pair of the user input task and each candidate output result is constructed, and the sample pairs are sorted according to the user preference ranking of the candidate output in each sample pair. Then, the preset template set (the preset template set contains multiple preset templates) that records the input task is combined to jointly construct the second credible training set. The second credible training set constructed in this way can allow the large language model to learn which results are more in line with the actual needs of users.

[0122] The training samples described above for constructing the second trusted training set can be extracted by non-artificial objects based on the implicit correspondence in the data sample set (for example, search logs in archives or archives institutions that reflect search terms and search results in user archive retrieval operations, archive-related questions or answers posted by users in archives communities, etc.), or they can be manually labeled and generated by archives professionals based on their own subjective understanding. Both methods can achieve efficient construction of sample pairs.

[0123] When the data sample set is a search log, the user's preference ranking can be expressed as the cumulative number of clicks on each candidate output in the candidate output list returned for the same user query text recorded in the search log within a period of time, that is, the more cumulative clicks, the more the candidate output meets the user's actual needs. The length of the time period can be set by yourself, such as one week or one month. If the data sample set comes from an archive community, the user's preference ranking can also be expressed as other parameters that can reflect the user's preference, such as the number of likes, the number of answers, the number of comments, etc., which will not be listed here one by one.

[0124] Step S107, performing unsupervised learning on the second credible large language model using the second credible training set to obtain a third credible large language model;

[0125] Specifically, the second credible large language model is trained based on the second credible training set. Compared with the first credible training set, the second credible training set can be understood as adding some training corpus. It is necessary to perform unsupervised learning on the second credible large language model again. The trained model is the third credible large language model.

[0126] Step S108, using the second credible training set to perform supervised training on the third credible large language model to obtain a credible reward model;

[0127] Specifically, the third trusted large language model after pre-training is supervisedly trained using the second trusted training set to obtain a trusted reward model. The Trust Reward Model (TRM) is a model that learns how to give different rewards (generally returned in the form of high and low scores) to different alternative outputs under the same input by learning the user preference ranking between different sample pairs under each training sample in the second trusted training set, so that the model learns in the direction of obtaining higher rewards, thereby being able to output more trusted results that meet the actual needs of users.

[0128] Step S109: Based on the credible reward model, the second credible large language model is trained by reinforcement learning based on artificial feedback to obtain a credible large language model.

[0129] Specifically, first determine the score returned by the trusted reward model for the user query and each alternative output. Then, based on the score returned by the trusted reward model, the second trusted large language model is trained under reinforcement learning based on human feedback to obtain a trusted large language model. The reinforcement learning method based on human feedback (RL HF) follows the following steps: initialize a new large language model S based on the parameters of the second trusted large language model; based on the new trusted prompt (prompt word, generally the user's natural language input), let S generate a reply for each trusted prompt, and then input the reply to the trusted reward model (TRM); TRM will calculate a score for each reply as a scalar reward, and its score indicates the quality of the reply; using the RLHF method, based on the total reward score obtained by model S, continuously update its strategy until convergence, and S trained to this point is a trusted large language model that meets the requirements. This model not only has credibility, but also has the ability to output results that are more in line with user needs.

[0130] To help you understand the solution described in this step, let's use an example:

[0131] Take the user query question "Which universities in China offer archives science majors?" as an example. There is only a demand for querying the situation of universities offering archives science majors, and the credible texts that respond to this demand have different contents, such as "There are 32 universities in China offering archives science majors," "The universities offering archives science majors are Renmin University of China, Nanjing University, Wuhan University and other 32 universities," "There are 32 undergraduate universities offering archives science majors and 23 graduate universities offering archives science majors," and other credible contents of different degrees. Aggregating the number of clicks on different credible results during this period, it is assumed that the cumulative number of clicks in Table 1 is obtained.

[0132] Table 1

[0133]

[0134]

[0135] Then the sample pairs constructed based on the above table are:

[0136] Rel(q,s1)>Rel(q,s2); Rel(q,s1)>Rel(q,s3); Rel(q,s2)>Rel(q,s3), Rel is a function used to express the user preference between the two.

[0137] Automatically generating the second trusted training set by selecting the data recorded in the retrieval log is a completely real reflection of user needs and a completely real reflection of the distribution of user preferences for different trusted results, thereby making the constructed second trusted training set more in line with the actual needs of users.

[0138] Based on the above sample pairs, we can find that the sample pair (q, s1) has a higher credibility and users give it a higher number of clicks, so a higher score or weight can be assigned in the credible reward model. In the sample pair (q, s3), the credibility is lower than other sample pairs, so a lower score or weight is assigned in the credible reward model. Similarly, for other problems, different scores or weights are assigned to different sample pairs in the credible reward model, so that the model is improved in the direction of high credibility and provides users with more credible results. The calculation formula is Rel(s, a, s') = γΦ(s') - Φ(s), where Rel(s, a, s') is the formed credible reward function, Φ(s') is the modified credible reward function, and Φ(s) is the original credible reward function.

[0139] Figure 3FIG. 1 is a flow chart of a reliable large language model reasoning method for archives provided by an embodiment of the present invention. Figure 3 As shown, the method uses the archive-oriented trusted large language model trained as described above, and the method includes:

[0140] Step S301, constructing a first credible reasoning data set according to an input task provided by a user;

[0141] Specifically, the user provides an input task to the trusted large language model, and the trusted large language model can optimize the context prompt of the input task in a variety of ways, such as extracting keywords from the input task text and matching them with the trusted data set used in the trusted large language model training process, or directly converting the input task text into a feature vector and matching it with the trusted data set in the model training. A better method can be selected in combination with practical applications. This embodiment is described using the former as an example, and the input task is extracted using a keyword extraction algorithm (such as TF_IDF, YAKE, etc.). The extracted keywords are used to search in the archive database constructed by the trusted data set in the model training, and the trusted information with the highest correlation with the above keywords is queried to obtain the trusted information and the preset prompt template. The matched prompt template will be submitted to the trusted large language model as context, and the matched prompt template will construct the first trusted reasoning data set.

[0142] For example, if the user's input task is "What interesting archival stories are there in Beijing's Nanluoguxiang?", the keywords extracted by the YAKE (Yet Another Keyword Extractor, YAKE; this algorithm uses the statistical features of a single document to extract keywords) method are "Beijing", "Nanluoguxiang", "interesting", and "archival stories". The model uses the above keywords to search in the archival database constructed by the trusted data set in training, and obtains the archival materials with the highest correlation with the above three keywords. These archival materials come from the archival database and are credible. The above archival materials are matched with the preset prompt template, and the matched prompt template is "a list of interesting archival stories in Beijing's Nanluoguxiang" and submitted to the trusted large language model as context. The prompt templates such as "a list of interesting archival stories in Beijing's Nanluoguxiang" construct the first credible prompt dataset.

[0143] Step S302, decomposing the first credible prompt data set into a plurality of sub-units;

[0144] Specifically, the first trusted prompt data set is decomposed into a series of discrete units or sub-elements, that is, a series of tokens.

[0145] Step S303, encoding the multiple sub-units to convert each sub-unit into a vector form;

[0146] Specifically, the decomposed series of tokens are encoded, each token is converted into a vector representation, and each token is mapped to a continuous vector space using word embedding or sub-word embedding technology. Word embedding is actually a type of technology in which a single word is represented as a real vector in a predefined vector space, and each word is mapped to a vector. In the case of Chinese, a token is usually a Chinese character; in the case of English, a token usually refers to a sub-word, not a word, and its corresponding sub-word granularity is a form between word granularity and character granularity. While retaining the meaning of the language, sub-words can effectively reduce the size of the vocabulary (i.e., the number of words in the vocabulary) and can also effectively solve the problem of out-of-vocabulary words. For example, the three English words "working", "worked", and "works" can get 4 sub-words: "work", "ed", "ing", and "s". In this way, the meaning of the word itself and the tense can be separated to obtain sub-words with smaller granularity.

[0147] Step S304, adjusting the encoded data into an input format of the credible large language model to input the credible large language model for reasoning;

[0148] Specifically, the encoded data is organized into an input format for trusted large language model reasoning. If the encoded data exceeds the length allowed by the trusted large language model, it needs to be broken down into smaller token units, that is, the above-mentioned word segmentation and encoding steps are repeated.

[0149] Step S305: decode the inference result using a decoding method corresponding to the encoding to obtain an inference result.

[0150] Specifically, the output of the trusted large language model is a probability distribution or vector representation, which needs to be combined with the input task to perform output decoding and generate the corresponding type of reasoning results. Finally, the generated reasoning results are removed from special symbols and formatted.

[0151] Preferably, the method further comprises automatically evaluating the inference result using the BLEU indicator. After the above post-processing, the inference result is obtained, and the inference result is automatically evaluated using the BLEU (Bilingual Evaluation Understudy) indicator, and the quality of the result is measured by calculating the n-gram overlap between the inference result and the reference answer. The higher the BLEU score obtained, the closer the inference result is to the reference answer, and the more credible the result is.

[0152] Figure 4 FIG. 1 is a schematic diagram of a structure of a reliable large language model training device for archives provided by an embodiment of the present invention. Figure 4 As shown, the device includes: a data text unit 1, a first trusted training set construction unit 2, a pre-trained language model construction unit 3, a first trusted large language model construction unit 4, a second trusted large language model construction unit 5, a second trusted training set construction unit 6, a third trusted large language model construction unit 7, a trusted reward model construction unit 8 and a reinforcement learning unit 9, wherein the data text unit 1 is used to pre-process different types of archival data to obtain multiple archival data in text form; the first trusted training set construction unit 2 is used to convert the multiple archival data in text form into tensor form data to obtain a first trusted training set; the pre-trained language model construction unit 3 is used to construct a pre-trained language model based on the multiple archival data in text form using an effective model architecture; the first trusted large language model construction unit 4 is used to use the first trusted training set to perform unsupervised training on the pre-trained language model Learning to obtain a first credible large language model; the second credible large language model construction unit 5 is used to use the first credible training set to perform supervised fine-tuning training on the first credible large language model to obtain a second credible large language model; the second credible training set construction unit 6 is used to obtain a second credible training set according to user preference ranking between the same user query content and different candidate outputs and a preset template set; the third credible large language model construction unit 7 is used to use the second credible training set to perform unsupervised learning on the second credible large language model to obtain a third credible large language model; the credible reward model construction unit 8 is used to use the second credible training set to perform supervised training on the third credible large language model to obtain a credible reward model; the reinforcement learning unit 9 is used to, based on the credible reward model, train the second credible large language model under reinforcement learning based on artificial feedback to obtain a credible large language model.

[0153] Preferably, the first trusted training set construction unit 2 is used to: perform the following operations for each text-form archival data: divide the text-form archival data into sub-units; add special sub-units, including a start sub-unit and an end sub-unit; construct a vocabulary to map each sub-unit to a unique integer ID; convert the text-form archival data into data in the form of corresponding integer IDs; create inputs and outputs, wherein the input is data in the form of integer IDs excluding the end sub-unit, and the output is data in the form of integer IDs excluding the start sub-unit; convert the inputs and outputs into tensor forms; and compose the inputs and outputs of each text-form archival data into the first trusted training set.

[0154] Preferably, the reinforcement learning unit 9 is used to: determine the score returned by the trusted reward model for the user query and each alternative output; based on the score returned by the trusted reward model, train the second trusted large language model under reinforcement learning based on artificial feedback to obtain the trusted large language model.

[0155] The embodiment of the archive-oriented credible large language model training device described above is similar to the embodiment of the archive-oriented credible large language model training method described above, and will not be repeated here.

[0156] Figure 5 FIG. 1 is a schematic diagram of a reliable large language model reasoning device for archives provided by an embodiment of the present invention. Figure 5 As shown, the device uses the above-mentioned archive-oriented trusted large language model, and the device includes: a first trusted reasoning data set construction unit 10, a sub-unit decomposition unit 11, a sub-unit encoding unit 12, an inference unit 13 and a decoding unit 14, wherein the first trusted reasoning data set construction unit 10 is used to construct a first trusted reasoning data set according to an input task provided by a user; the sub-unit decomposition unit 11 is used to decompose the first trusted prompt data set into a plurality of sub-units; the sub-unit encoding unit 12 is used to encode the plurality of sub-units to convert each sub-unit into a vector form; the inference unit 13 is used to adjust the encoded data to the input format of the trusted large language model to input the trusted large language model for inference; the decoding unit 14 is used to decode the inference result using a decoding method corresponding to the encoding to obtain the inference result.

[0157] Preferably, the device further comprises an automatic evaluation unit 15 for automatically evaluating the inference result using the BLEU indicator.

[0158] The device also includes a credible reference display unit 16, which is used to remove special symbols, format and other operations based on the generated reasoning result to obtain a final reasoning result. When outputting the final reasoning result, the reference link or the source of the citation is displayed together to obtain a credible reasoning result.

[0159] The embodiment of the archive-oriented trusted large language model reasoning device described above is similar to the embodiment of the archive-oriented trusted large language model reasoning method described above, and will not be repeated here.

[0160] An embodiment of the present invention also provides an electronic device, comprising at least one high-performance GPU (graphics processing unit) and at least one memory communicating with the GPU, wherein the memory stores a computer program for the processor to execute any of the above-mentioned archive-oriented trusted large language model training and reasoning methods.

[0161] Figure 6 is a schematic block diagram of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of computers, such as desktop computers, notebooks, servers, and other suitable computers. The electronic device may also represent mobile devices, such as smart phones, iPads, and other similar devices. The components, the relationships between the components, and the functions are only examples and do not limit the implementation described in the present invention. Figure 6 It includes a computing unit, a memory, a bus, an input / output interface unit, an input unit, an output unit, a communication unit, and a storage unit. The computing unit is a variety of general or special processing components with processing and computing capabilities, including image processing units GPU, tensor processing units TPU, central processing units CPU, etc. The computing unit executes the above methods, such as the archive-oriented trusted large language model training method and reasoning method. The input unit includes a keyboard, a mouse, etc., the output unit includes various displays, speakers, etc., the storage unit includes a disk, an optical disk, etc., and the communication unit includes a network card, a wireless router, etc. The computing unit and the memory are connected to each other through a bus, and the input / output interface is also connected to the bus.

[0162] An embodiment of the present invention provides a storage medium storing a computer program, which, when executed, implements any of the above-mentioned archive-oriented trusted large language model training and reasoning methods.

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0168] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0169] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0171] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A reliable large language model training method for archives, characterized in that: The method comprises: Preprocess different types of archival data to obtain multiple textual archival data; Converting the plurality of text-based archival data into tensor-based data to obtain a first credible training set; Based on the multiple text-based archival data, a pre-trained language model is constructed using an effective model architecture; Using the first credible training set to perform unsupervised learning on the pre-trained language model to obtain a first credible large language model; Using the first credible training set to perform supervised fine-tuning training on the first credible large language model to obtain a second credible large language model; A second credible training set is obtained according to user preference ranking between the same user query content and different candidate outputs and a preset template set; Using the second credible training set to perform unsupervised learning on the second credible large language model to obtain a third credible large language model; Using the second credible training set to perform supervised training on the third credible large language model to obtain a credible reward model; Based on the credible reward model, the second credible large language model is trained under reinforcement learning based on artificial feedback to obtain a credible large language model.

2. The archive-oriented credible large language model training method according to claim 1, characterized in that: The converting the plurality of text-based archival data into tensor-based data to obtain a first credible training set includes: For each text file, perform the following operations: Segmenting archival data in text form into subunits; Add special subunits, including start subunit and end subunit; Build a vocabulary that maps each subunit to a unique integer ID; Convert the text-based archive data into corresponding integer ID-based data; Creating an input and an output, wherein the input is data in the form of an integer ID excluding an end subunit, and the output is data in the form of an integer ID excluding a start subunit; Convert the input and the output into tensor form; The input and output of each text-form archive data constitute the first credible training set.

3. The archive-oriented credible large language model training method according to claim 1, characterized in that: The step of training the second credible large language model based on the credible reward model under reinforcement learning based on artificial feedback to obtain a credible large language model includes: Determining the score returned by the trusted reward model for the user query and each candidate output; Based on the score returned by the trusted reward model, the second trusted large language model is trained under reinforcement learning based on artificial feedback to obtain the trusted large language model.

4. A reliable large language model reasoning method for archives, characterized in that: The method uses a credible large archive-oriented language model trained by any one of claims 1 to 3, and the method comprises: Constructing a first credible reasoning dataset according to an input task provided by a user; Decomposing the first trusted reasoning data set into a plurality of sub-units; encoding the plurality of sub-units to convert each sub-unit into a vector form; Adjusting the encoded data into an input format of the credible large language model to input the credible large language model for reasoning; The inference result is decoded using the decoding method corresponding to the encoding to obtain the inference result.

5. The archive-oriented credible large language model reasoning method according to claim 4, characterized in that: The method further includes: The inference results are automatically evaluated using the BLEU indicator.

6. A reliable large language model training device for archives, characterized in that: The device comprises: A data textualization unit, a first credible training set construction unit, a pre-trained language model construction unit, a first credible large language model construction unit, a second credible large language model construction unit, a second credible training set construction unit, a third credible large language model construction unit, a credible reward model construction unit, and a reinforcement learning unit, wherein: The data texting unit is used to pre-process different types of archive data to obtain multiple archive data in text form; The first credible training set construction unit is used to convert the plurality of text-based archival data into tensor-based data to obtain a first credible training set; The pre-trained language model construction unit is used to construct a pre-trained language model based on the multiple text-based archival data using an effective model architecture; The first credible large language model construction unit is used to perform unsupervised learning on the pre-trained language model using the first credible training set to obtain a first credible large language model; The second credible large language model construction unit is used to perform supervised fine-tuning training on the first credible large language model using the first credible training set to obtain a second credible large language model; The second trusted training set construction unit is used to obtain a second trusted training set according to the user preference ranking between the same user query content and different candidate outputs and a preset template set; The third credible large language model construction unit is used to perform unsupervised learning on the second credible large language model using the second credible training set to obtain a third credible large language model; The trusted reward model construction unit is used to perform supervised training on the third trusted large language model using the second trusted training set to obtain a trusted reward model; The reinforcement learning unit is used to obtain a credible large language model by training the second credible large language model under reinforcement learning based on artificial feedback based on the credible reward model.

7. The archive-oriented credible large language model training device according to claim 6, characterized in that: The first credible training set construction unit is used for: For each text file, perform the following operations: Segmenting archival data in text form into subunits; Add special subunits, including start subunit and end subunit; Build a vocabulary that maps each subunit to a unique integer ID; Convert the text-based archive data into corresponding integer ID-based data; Creating an input and an output, wherein the input is data in the form of an integer ID excluding an end subunit, and the output is data in the form of an integer ID excluding a start subunit; Convert the input and the output into tensor form; The input and output of each text-form archive data constitute the first credible training set.

8. The archive-oriented credible large language model training device according to claim 6, characterized in that: The reinforcement learning unit is used to: Determining the score returned by the trusted reward model for the user query and each candidate output; Based on the score returned by the trusted reward model, the second trusted large language model is trained under reinforcement learning based on artificial feedback to obtain the trusted large language model.

9. A reliable large language model reasoning device for archives, characterized in that: The device uses a credible large language model for archives trained by any one of claims 6 to 8, and the device comprises: A first trusted reasoning data set construction unit, a subunit decomposition unit, a subunit encoding unit, a reasoning unit, and a decoding unit, wherein: The first trusted reasoning data set construction unit is used to construct a first trusted reasoning data set according to an input task provided by a user; The sub-unit decomposition unit is used to decompose the first trusted reasoning data set into a plurality of sub-units; The sub-unit encoding unit is used to encode the multiple sub-units to convert each sub-unit into a vector form; The inference unit is used to adjust the encoded data into the input format of the credible large language model so as to input the credible large language model for inference; The decoding unit is used to decode the inference result using a decoding method corresponding to the encoding to obtain the inference result.

10. The archive-oriented credible large language model reasoning device according to claim 9, characterized in that: The device also includes: The automatic evaluation unit is used to automatically evaluate the inference result using the BLEU indicator.

Images