An adaptive document selection summary generation method based on transfer learning

Abstract

The present invention relates to an adaptive document selection summary generation method based on transfer learning, and belongs to the field of artificial intelligence. In order to solve the problem of artificial intelligence-assisted archive research, the present invention uses a document selection method based on word distribution and a document selection method based on summary trend to filter a small amount of training documents of source documents and archives, screens out training data to train a first word vector generator, and obtains a basic summary generation model; the source documents and test documents are screened by an adaptive document selection method, and according to the characteristics of the test documents, a source document with strong correlation is selected to train a second word vector generator, optimizes the basic summary generation model, and constructs a customized summary generation model to more accurately meet the needs of specific archive categories. The present invention can realize automatic generation of summaries, promote archive intelligence, remove negative migration of archives, increase summary relevance, realize summary customization, and improve summary accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to an adaptive document selection summary generation method based on transfer learning. Background Art

[0002] Intelligence is an important and cutting-edge issue in the deepening application of archives. Traditional archive systems are mostly carried out manually in the field of archive compilation and research, which is often inefficient and difficult to adapt to the needs of large-scale data processing and rapid retrieval. Intelligent archive systems are gradually becoming a revolutionary force in the field of archive management. Intelligent technology can greatly improve the efficiency and accuracy of archive management through automated data processing, intelligent classification and precise retrieval. It can not only reduce the burden of manual labor, but also provide more intelligent analysis and prediction functions, thereby helping organizations to better utilize and manage archive resources.

[0003] Aiming at the automatic compilation and research needs of multi-source, heterogeneous, massive and complex archived electronic files, this paper studies the integration of artificial intelligence, large models and other technologies into the field of electronic archives, and realizes the goal of automatic compilation and research after massive heterogeneous electronic data enters the archive management system through the automatic generation of summaries, and innovates the work mode of artificial intelligence-assisted compilation and research in archives. The summary generation methods mainly include extractive summarization and generative summarization. The extractive summarization method selects sentences as summaries from a given document. The extractive summarization generates summaries by directly selecting key sentences from the original document. The advantage of this method is that it is relatively simple to implement, but the generated summaries lack fluency and coherence. The generative summaries generate new sentences to form summaries by analyzing the relationship pattern between the input text and the output summary in the training data, which has greater flexibility and expression ability. However, this method requires that the training data should be consistent with the domain of the target document, and it is difficult to construct large-scale labeled data in the archive field. Transfer learning can effectively solve the problem of insufficient training data and use the training data of the source domain to improve the model performance. By extracting relevant instances from the source domain, transfer learning can improve the quality of summary generation. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is how to provide an adaptive document selection summary generation method based on transfer learning to solve the problem of artificial intelligence-assisted archive compilation and research.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention proposes an adaptive document selection summary generation method based on transfer learning, which includes:

[0008] The source documents are selected by using the document selection method based on word distribution and the document selection method based on summary trend. and archives Filter a small number of training documents, select a batch of high-quality training data, use it to train the first word vector generator, and then obtain the basic summary generation model;

[0009] The source documents are selected by adaptive document selection method and test documentation Further screening is carried out to select source documents with strong relevance to the test documents based on their characteristics to generate customized training data, which are used to train the second word vector generator, optimize the basic summary generation model, and further build a customized summary generation model to more accurately meet the needs of specific archive categories.

[0010] (III) Beneficial effects

[0011] The present invention proposes an adaptive document selection summary generation method based on transfer learning. The present invention proposes an adaptive document selection summary generation method, which is helpful to improve the document summary model in the archive field. The method selects training data according to the word distribution and summary trend of the archive. The present invention adaptively selects archives to train a customized model through a word vector generation network, and establishes a customized summary model for each type of archive. The present invention improves the relevance of archives by using an adaptive document selection summary generation method, avoids the generation of negative transfer, and improves the performance of summary generation.

[0012] 1) Automatically generate summaries and promote intelligent archives

[0013] Transfer learning technology is used to avoid using a large number of archives to train the summary generation model. The training data in the source domain and a small amount of archive data are input into the word vector generation network to train the automatic summary generation model. The extractive and generative methods are combined to effectively copy the information in the archives, while retaining the ability of generative innovation. By setting up a coverage mechanism to track the summary generation process, the repetition rate of elements in the summary is reduced, the accuracy of archive summary generation is improved, and the intelligent generation of archive summary is realized.

[0014] 2) Remove negative transfer of archives and increase the relevance of summaries

[0015] The archives and source documents are highly correlated, and the features of irrelevant documents in the source domain will interfere with the model training process, resulting in negative transfer, causing the generated summary to deviate from the actual needs and context of the archives. The document selection method based on word distribution and the document selection method based on summary trend proposed in the present invention can improve the similarity between the source documents and archives, ensure that the model trained in the transfer learning process is relevant to the archives, reduce the negative transfer phenomenon, increase the relevance of the summary, and achieve high-quality document summary generation.

[0016] 3) Customize summaries and improve their accuracy

[0017] There are specific test documents in the archives that contain secondary topics, which may provide important information when summarizing archives covering the same topic. The above filtering process may ignore some source documents that are critical to a specific test document. The adaptive document selection method establishes a customized summary model to select source documents on a specific topic by analyzing the word distribution of the test document, ensuring that valuable source documents on a specific topic in the test document are fully utilized when building the summary model. The adaptive selection mechanism not only avoids the loss of important information due to the limitations of calculating word distribution probabilities, but also accurately captures the core content of the test document, thereby improving the accuracy of the summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the overall flow chart of the present invention;

[0019] Figure 2 Generate a network graph for the word vector of the present invention;

[0020] Figure 3 A diagram of a document selection method based on word distribution according to the present invention;

[0021] Figure 4 A diagram of a document selection method based on summary trends of the present invention;

[0022] Figure 5 A diagram of the adaptive document selection method of the present invention;

[0023] Figure 6 Generate a model interface design diagram for the abstract of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0025] The purpose of the present invention is to propose a document selection method based on transfer learning to realize the automatic generation design of archive summaries, which has the characteristics of small amount of training data, removal of negative transfer, and customized summary. Source documents are selected according to the similarity of the vocabulary probability distribution between each source document and the archive document set. At the same time, similar source documents are selected for each test archive, so as to construct a customized summary model and improve the quality of archive summary generation.

[0026] 1. Overall process

[0027] Figure 1 The overall flow chart of the present invention is as follows. The goal of the present invention is to use transfer learning to build an automatic archive summary generation model. Figure 1 As shown, three types of archive documents are provided, including a large number of source documents and the corresponding abstracts of the source documents, from the archives A small number of training documents extracted from the archive and their corresponding summaries, and test documents from the archive .

[0028] First, the source documents are selected using the word distribution-based document selection method and the summary trend-based document selection method. and archives A small number of training documents are filtered out, and a batch of high-quality training data is selected from them, which is used to train the first word vector generator, and then the basic summary generation model is obtained.

[0029] At the same time, the source documents are selected by adaptive document selection method. and test documentation Further screening is carried out to select source documents with strong relevance to the test documents based on their characteristics to generate customized training data, which are used to train the second word vector generator, optimize the basic summary generation model, and further build a customized summary generation model to more accurately meet the needs of specific archive categories.

[0030] 2. Word vector generator network structure

[0031] The network diagram of the word vector generator used by the first word vector generator and the second word vector generator is as follows Figure 2 As shown, enter the source document ,in Indicates The document is processed by the encoder and decoder to obtain the Output of documents , the output for all source documents is The word embedding generator has replication capability and includes an attention mechanism, which can represent the source document The copy mechanism can extract the parts related to the archive field from the source document at will for training the model. The word vector generator has the ability to copy words from the source document and generate new words through the generator to ensure the quality and accuracy of the generated summary. The process is tracked in real time through the coverage mechanism to reduce the repetitiveness of words in the summary, further improving the coherence and information density of the summary.

[0032] 2.1 Encoder structure design

[0033] The encoder uses a conventional bidirectional The network structure is used to capture the long-range dependencies and location information of the source documents. Input archive data set ,in Indicates documents, Unique hot vector representing the word size dimension. Each encoder receives a word The embedding of is taken as input, and then bidirectionally Output encoding status after network processing . Encoding status It has position features, long-distance dependencies and context features between words.

[0034] 2.2 Decoder structure design

[0035] The decoder uses a single layer The network structure receives the encoding state generated by the encoder In single step time Internal calculation decoding status , by encoding the state and decoding status Calculating attention weights .

[0036] Since there may be duplication of words in the source document, the words generated in the expanded vocabulary may be repeated with the words in the source document. The present invention designs a method for detecting over-leakage, which obtains the over-leakage vector by accumulating the attention weights of the previous time step and incorporates it into the calculation process of the attention weight to avoid the duplication of words in the same position, so as to enhance the model's ability to recognize word duplication. The calculation formula is as follows, where represents the attention weight of the previous time step, , And the weight matrix , and The parameters of the trained model are:

[0037] (1)

[0038] Then calculate the context feature vector, Represents single step time The context feature vector is obtained by adding the attention weight and the encoding state of the word The product accumulation and summation of represents the part of information most relevant to the generated vocabulary in the current decoding step. The specific calculation formula is as follows:

[0039] (2)

[0040] The word probability is calculated by Used to predict the words to be generated: the decoder's decoding state and context vector Perform a splicing operation and perform a linear transformation on the spliced ​​vectors using the parameter matrix , and the bias term , , apply the calculated scoring vector Function, get the probability of vocabulary , predict the next most likely word by probability. The calculation formula is as follows, where , and , are the parameters obtained from model training:

[0041] (3)

[0042] because Only the distribution of words in the generated vocabulary is processed. The present invention introduces a probability variable , used to determine whether the generated words are repeated in the source document. Combined with the source document data , splicing decoding status and context vector ,pass Function calculation to obtain probability variables The calculation formula is as follows, where , and are the parameters obtained through training:

[0043] (4)

[0044] Finally, we get the probability of expanding vocabulary , the calculation formula is as follows:

[0045] (5)

[0046] in is the probability of being in the generated vocabulary when the word is not in the generated vocabulary , when the word is not in the source document . Probability variables The source of the weighed vocabulary is the source document or the generated vocabulary. If the generated words are repeated in the source document, it means that the probability that the vocabulary comes from the source document is relatively high.

[0047] The decoder parses , according to the expanded vocabulary probability Generate the corresponding output sequence , and optimize the model through cross entropy loss so that the model's predicted output is as close to the true target output as possible. During the training process, the loss function helps adjust the model parameters to improve the prediction accuracy.

[0048] 3. Document selection method based on word distribution

[0049] Document selection methods based on word distribution are as follows Figure 3 As shown, first calculate the archive and source documents Word probability distribution of all words in the training data and , and then through Divergence measures the difference between the probability distributions of two words, and chooses the one with the highest similarity documents as training data for the summary generation model.

[0050] file The word probability distribution of each word in is calculated using the following equation, where the numerator represents the frequency of the target word and the denominator represents the frequency of all words:

[0051] (6)

[0052] in Expressing words In the archive document collection The frequency of occurrence in Indicates the number of words; represents the smoothing parameter.

[0053] Next, for the source document Calculate the word probability distribution of each word in each document, denoted as , and compare it with the probability distribution of the archive For comparison, use Divergence quantifies the difference between probability distributions.

[0054] Given a word probability distribution Sum word probability distribution , calculated by the following formula Divergence:

[0055] (7)

[0056] , The word probability distribution is Sum word probability distribution The i-th probability of , as the difference between the probability distribution of the archive and the source document increases, The greater the divergence, the greater the and When equal, The divergence is 0. By evaluating the relevance of the source document to the archive, select indivual The source documents with the lowest divergence are used as training data.

[0057] 4. Document selection method based on summary trend

[0058] Archival summaries have different styles, structures, or emphases than the summaries in the source documents. For example, documents in some fields may be more technical, while documents in other fields may focus more on overviews and background. When the summary distributions of source documents are significantly different, since the summary generation model needs to learn the conversion pattern from archive training documents to summaries, differences in summary length and summary compression rate will result in negative transfer, causing the model to perform lower than expected in archives.

[0059] Document selection methods based on summary trends are as follows Figure 4 As shown in the figure, the archive is characterized by two features: summary length and summary compression rate. , generate a trend feature histogram, obtain the corresponding probability distribution through word distribution calculation, and select training data with similar probability in the source document.

[0060] The specific steps are as follows:

[0061] First, a trend feature histogram is created for the archives to show the distribution of archives in different feature ranges. Each block in the histogram is represented by Indicates that, assuming the archive trend feature is , calculate the features The probability of falling within the feature range is denoted by .

[0062] Next, based on probability Select a batch of training data with similar features in the source documents , All documents in the Features in .

[0063] Finally, we get the same characteristics training documents, in all Within the scope, the feature distribution of the source document is as consistent as possible with the feature distribution of the archive.

[0064] By comparing the feature distribution, we ensure that the selected source documents can effectively represent the feature distribution of the archives and accurately capture the key information of the archives, thereby mitigating the impact of negative transfer.

[0065] 5. Adaptive document selection method

[0066] The document selection method based on word distribution and the document selection method based on summary trend may lose the secondary topic information in the document. The present invention proposes an adaptive document selection method, such as Figure 5 shown.

[0067] First use the archive The first word vector generator is trained by the training documents in and the source documents selected by the document selection method based on word distribution and the document selection method based on summary trend. The obtained model is called the basic summary generation model.

[0068] Next, enter the test document , get customized source documents through word distribution-based document selection method , according to Divergence choose The source document with the highest similarity is used as the customized source document ,in represents the test document calculated using the same method as equation (6) The word distribution.

[0069] Customized archive files and archives The second word vector generator is input and training continues on the basis of the basic summary generation model. The optimized model obtained through training is called the customized summary generation model.

[0070] Custom summary generation models can leverage the Lexical features in the test document It includes vocabulary features beyond the basic summary generation model, allowing the model to generate more accurate summaries for specific test documents.

[0071] 6. Abstract automatic generation model interface design

[0072] At the software design level, this method ultimately provides a summary automatic generation model, which provides two basic interfaces: a document interaction interface and a summary output interface, which interact with the archives comprehensive management system. Among them, the document interaction interface is an encapsulation of the adaptive document selection method. The documents input from the archives comprehensive management system include archives, source documents, and test documents. The unique identifiers of the filtered archives and source documents are used as input. If the source document filtering is completed, the feedback result is true. If the source document filtering fails, the feedback result is false. The summary output interface is an encapsulation of the customized summary generation model. The unique identifier is used as input to generate a summary result. If the summary generation is completed, true is output to the archives comprehensive management system. If the summary generation fails, false is output to the archives comprehensive management system. Specifically, Figure 6shown.

[0073] The present invention proposes an adaptive document selection summary generation method, which is helpful to improve the document summary model in the archive field. The method selects training data according to the word distribution and summary trend of the archive. The present invention adaptively selects archives to train a customized model through a word vector generation network, and establishes a customized summary model for each type of archive. The present invention improves the relevance of archives by using the adaptive document selection summary generation method, avoids the generation of negative transfer, and improves the performance of summary generation.

[0074] 1) Automatically generate summaries and promote intelligent archives

[0075] Transfer learning technology is used to avoid using a large number of archives to train the summary generation model. The training data in the source domain and a small amount of archive data are input into the word vector generation network to train the automatic summary generation model. The extractive and generative methods are combined to effectively copy the information in the archives, while retaining the ability of generative innovation. By setting up a coverage mechanism to track the summary generation process, the repetition rate of elements in the summary is reduced, the accuracy of archive summary generation is improved, and the intelligent generation of archive summary is realized.

[0076] 2) Remove negative transfer of archives and increase the relevance of summaries

[0077] The archives and source documents are highly correlated, and the features of irrelevant documents in the source domain will interfere with the model training process, resulting in negative transfer, causing the generated summary to deviate from the actual needs and context of the archives. The document selection method based on word distribution and the document selection method based on summary trend proposed in the present invention can improve the similarity between the source documents and archives, ensure that the model trained in the transfer learning process is relevant to the archives, reduce the negative transfer phenomenon, increase the relevance of the summary, and achieve high-quality document summary generation.

[0078] 3) Customize summaries and improve their accuracy

[0079] There are specific test documents in the archives that contain secondary topics, which may provide important information when summarizing archives covering the same topic. The above filtering process may ignore some source documents that are critical to a specific test document. The adaptive document selection method establishes a customized summary model to select source documents on a specific topic by analyzing the word distribution of the test document, ensuring that valuable source documents on a specific topic in the test document are fully utilized when building the summary model. The adaptive selection mechanism not only avoids the loss of important information due to the limitations of calculating word distribution probabilities, but also accurately captures the core content of the test document, thereby improving the accuracy of the summary.

[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating an adaptive document selection summary based on transfer learning, characterized in that: The method includes: The source documents are selected by using the document selection method based on word distribution and the document selection method based on summary trend. and archives Filter a small number of training documents, select a batch of high-quality training data, use it to train the first word vector generator, and then obtain the basic summary generation model; The source documents are selected by adaptive document selection method and test documentation Further screening is performed to select source documents with strong relevance to the test documents based on their characteristics to generate customized training data, which is used to train the second word vector generator, optimize the basic summary generation model, and further build a customized summary generation model to more accurately meet the needs of specific archive categories; in, The first word vector generator and the second word vector generator use the same word vector generator network; The document selection method based on summary trend includes: characterizing the archive by using two features: summary length and summary compression rate. , generate a trend feature histogram, obtain the corresponding probability distribution through word distribution calculation, and select training data with similar probability in the source document; The document selection method based on summary trend specifically includes: First, a trend feature histogram is made for the archives to show the distribution of archives in different feature ranges; each block in the histogram is represented by Indicates that, assuming the archive trend feature is , calculate the features The probability of falling within the feature range is denoted by ; Next, based on probability Select a batch of training data with similar features in the source documents , All documents in the Features in; Finally, we get the same characteristics training documents, in all Within the scope, the characteristic distribution of the source document is as consistent as possible with that of the archive; The adaptive document selection method comprises: Enter the test document , get customized source documents through word distribution-based document selection method , according to Divergence choose The source document with the highest similarity is used as the customized source document ; For source documents The word probability distribution of all words in the training data, For test documents Word probability distribution of all words in the training data; Customized archive files and archives The second word vector generator is input and training continues on the basis of the basic summary generation model. The optimized model obtained through training is called the customized summary generation model.

2. The adaptive document selection summary generation method based on transfer learning as claimed in claim 1, characterized in that: The word vector generator network used by the first word vector generator and the second word vector generator includes: input source document ,in Indicates The document is processed by the encoder and decoder to obtain the Output of documents , the output for all source documents is .

3. The adaptive document selection summary generation method based on transfer learning as claimed in claim 2, characterized in that: The encoder uses a conventional bidirectional Network structure, used to capture long-range dependencies and location information of source documents; input archive data set ,in Indicates documents, Unique hot vector representing the word size dimension. Each encoder receives a word The embedding of is taken as input, and then bidirectionally Output encoding status after network processing ; Encoding status It has position features, long-distance dependencies and context features between words.

4. The adaptive document selection summary generation method based on transfer learning as claimed in claim 3, characterized in that: The decoder uses a single layer The network structure receives the encoding state generated by the encoder In single step time Internal calculation decoding status , by encoding the state and decoding status Calculating attention weights ; The leak vector is obtained by accumulating the attention weights of the previous time step and incorporating it into the calculation process of the attention weight to avoid the repetition of words in the same position, so as to enhance the model's recognition ability of word repetition; the calculation formula is as follows, where represents the attention weight of the previous time step, , And the weight matrix , and The parameters of the trained model are: (1) Then calculate the context feature vector, Represents single step time Contextual features of internal fixed size; Context The feature vector is obtained by adding the attention weights and the encoding state of the word The product accumulation and summation of is obtained, which represents the part of information most relevant to the generated vocabulary in the current decoding step; the specific calculation formula is as follows: (2) The word probability is obtained by calculating Used to predict the words to be generated: the decoder's decoding state and context vector Perform a splicing operation and perform a linear transformation on the spliced ​​vectors using the parameter matrix , and the bias term , , apply the calculated scoring vector Function, get the probability of vocabulary , predict the next most likely word by probability; the calculation formula is as follows, where , and , are the parameters obtained from model training: (3) Introducing a probability variable , used to determine whether the generated words are repeated in the source document; combined with the source document data , splicing decoding status and context vector ,pass Function calculation to obtain probability variables ; The calculation formula is as follows, where , and are the parameters obtained through training: (4) Finally, we get the probability of expanding vocabulary , the calculation formula is as follows: (5) in is the probability of being in the generated vocabulary when the word is not in the generated vocabulary , when the word is not in the source document ; Probability variables The source of the weighed vocabulary is the source document or the generated vocabulary. If the generated words are repeated in the source document, it means that the probability that the vocabulary comes from the source document is relatively high. The decoder parses , according to the expanded vocabulary probability Generate the corresponding output sequence , and optimize the model through cross entropy loss so that the predicted output of the model is as close to the actual target output as possible.

5. The method for generating an adaptive document selection summary based on transfer learning according to any one of claims 1 to 4, characterized in that: The document selection method based on word distribution includes: First, calculate the archive and source documents Word probability distribution of all words in the training data and , and then through Divergence measures the difference between the probability distributions of two words, and chooses the one with the highest similarity documents as training data for the summary generation model; file The word probability distribution of each word in is calculated by using the following formula, where the numerator represents the frequency of the target word and the denominator represents the frequency of all words: (6) in Expressing words In the archive document collection The frequency of occurrence in Indicates the number of words; represents the smoothing parameter; Next, for the source document Calculate the word probability distribution of each word in each document, denoted as , and compare it with the probability distribution of the archive For comparison, use Divergence quantifies the differences between probability distributions; Given a word probability distribution Sum word probability distribution , calculated by the following formula Divergence: (7) , The word probability distribution is Sum word probability distribution The i-th probability of , as the difference between the probability distribution of the archive and the source document increases, The greater the divergence, the and When equal, The divergence is 0; by evaluating the relevance of the source document to the archive, select indivual The source documents with the lowest divergence are used as training data.

6. The adaptive document selection summary generation method based on transfer learning as claimed in claim 5, characterized in that: This method finally provides a summary automatic generation model, which provides two basic interfaces: document interaction interface and summary output interface, to interact with the archives comprehensive management system.

7. The method for generating an adaptive document selection summary based on transfer learning according to claim 6, characterized in that: The document interaction interface is an encapsulation of the adaptive document selection method. The documents input from the archives comprehensive management system include archives, source documents and test documents. The unique identifiers of the filtered archives and source documents are used as input. If the source document filtering is completed, the feedback result is true. If the source document filtering fails, the feedback result is false. The summary output interface is an encapsulation of the customized summary generation model. The summary result is generated with the unique identifier as input. If the summary generation is completed, true is output to the archives comprehensive management system. If the summary generation fails, false is output to the archives comprehensive management system.

Images