Automatic construction method of structured document title

Abstract

The invention provides an automatic construction method of a structured document title, which can automatically generate a title for a structured document with a long context. The method comprises the following steps of: 1, generating an abstract of a structured document by utilizing structural information and content characteristics of the document through an abstract generation module based on a hierarchical attention network; 2, generating keywords from the abstract by using a text generation model based on a bidirectional encoder and an autoregression decoder, and preferably selecting the keywords through a keyword screening mechanism; thirdly, interaction between the screened keywords and the abstract is analyzed through a cross attention mechanism, and feature representation fusing keyword features and document content is generated; and 4, automatically constructing a structured document title according to the feature representation. The whole process is automatically completed, and the title related to the structured document content can be obtained.

Description

Technical Field

[0001] The invention relates to the field of natural language processing and is an automatic construction method for structured document titles. Background Art

[0002] With the rapid development of information technology and data analysis, more and more researchers and technicians have begun to pay attention to the application of natural language processing technology in the writing of structured documents. The title of a structured document (such as an elevator design document, etc.) is not only an important part of the document, but also the key to conveying the core content of the document and attracting readers' attention. In the process of writing a structured document, an accurate and concise title can greatly improve the readability and practical value of the document. Traditional title generation methods usually require authors to manually create titles based on the core content and work results of the document, which is not only time-consuming but also inefficient. In addition, manually generated titles may not fully cover the key points of the document due to the limitations of personal experience and knowledge background.

[0003] Currently disclosed title generation methods, such as the patent document with publication number CN115496061A and the name of "A method for constructing a neural network title generation model", mainly propose a neural network structure that combines a multi-layer attention mechanism and a rich information word vector layer to enhance the coherence and simplicity of title generation, but do not solve the problem of effectively processing complex structures and professional terms in structured documents. In the patent document with publication number CN106383817A and the name of "a method for generating paper titles using distributed semantic information", the TextRank algorithm is mainly used to extract keywords and generate titles based on a recurrent neural network based on LSTM, but it only relies on keywords and does not make full use of the abstract content, and the TextRank algorithm may not be able to accurately extract highly representative keywords. In the patent document with publication number CN117787260A and the name of "a method for generating text titles based on adaptive position encoding and knowledge reasoning", a method for improving the quality of title generation through adaptive position encoding and knowledge reasoning is provided, but it does not effectively handle the problem of generating titles for structured documents with long sequences or complex structures.

[0004] In summary, although existing title generation methods have made some progress in different aspects, they still have limitations, especially when dealing with structured documents with complex structures or long lengths. In addition, these methods often rely too much on keywords and fail to make full use of the abstract content, or only use the abstract and ignore the role of keywords, and the accuracy of keyword extraction is often insufficient. Some current methods rely only on the abstract content, resulting in the possibility that title generation may be affected by redundant information in the abstract; others rely only on keywords. Although keywords can indicate the main topic of the document, they usually cannot cover all the core content, nor can they fully represent the amount of information that the title needs to convey.

[0005] Therefore, an ideal title generation method should be able to effectively combine abstract content and keywords, take advantage of both, reduce redundant information in the abstract, and strengthen the representativeness of keywords to generate more accurate and attractive titles. However, there is currently no method for generating titles based on the fusion of abstracts and keywords. The method for fusion of abstracts and keywords proposed in the present invention provides an innovative solution to this gap. Summary of the invention

[0006] The present invention relates to an automatic construction method for structured document titles, which aims to automatically generate accurate and clear titles for structured documents with complex structures and long contexts. The method comprehensively utilizes technologies such as summary generation, keyword generation and screening, interactive feature representation construction, and title generation, solves the shortcomings of the prior art in processing complex structure document title generation, and improves the quality and relevance of title generation.

[0007] The method of the present invention comprises the following steps:

[0008] Step 1: Generate a summary of a structured document: Train a summary generation model that can generate a summary of a structured document. This includes the following sub-steps: Step 1.1: For document D i For hierarchical encoding, first each word w in the document i Convert to word vector e i , use a bidirectional recurrent neural network to encode the word level, sentence level and paragraph level respectively, and obtain the importance weights of words, sentences and paragraphs. The specific implementation steps are as follows:

[0009] First, each word w in the structured document i Convert to word vector e i Secondly, a bidirectional recurrent neural network is used to encode the word vector sequence to generate a sentence vector s j Then, the bidirectional recurrent neural network is used to encode the sentence vector sequence to obtain the paragraph vector p k Next, the importance weights of each element at each level are calculated at the word level and sentence level. The specific calculation method is as follows:

[0010] Importance weight of word vector:

[0011]

[0012] Among them, e i is the word vector of the i-th word, s j is the sentence vector of the jth sentence;

[0013] Importance weights of sentence vectors:

[0014]

[0015] Among them, s j is the sentence vector of the jth sentence, p k is the paragraph vector of the kth paragraph.

[0016] Each word vector e i Multiply it by its importance weight α i And sum to get the updated sentence vector s′ j :

[0017]

[0018] The sentence vector s′ j Multiply it by its importance weight β j And sum them up to get the updated paragraph vector p′ k :

[0019]

[0020] Step 1.2: Build a neural network model for summary generation. This model uses the importance weights of each element at each level obtained in step 1.1 to perform weighted summation on the outputs of the word-level, sentence-level, and paragraph-level encoders to obtain the context vector c. t , and according to the context vector c t and the hidden state s t Use sequence-to-sequence decoder to generate document D i The summary of which is the hidden state s t The output y from the previous step t-1 and the current hidden state s t-1 Generation; Step 1.3, use the cross entropy loss function to train the summary generation model through the error back propagation algorithm, optimize the model parameters, and make the generated summary As close as possible to the true summary X, so as to achieve the structured document D i Automatic generation of summaries.

[0021] Step 2, generating keywords for structured documents: first train a text generation model, and then use the model to generate keyword sequences from the summary X in the keyword generation module. The following steps are included:

[0022] Step 2.1, train the keyword generation model in the keyword generation module: use a method based on bidirectional encoding and autoregressive decoding as the basic model for keyword generation. During the training process, the input is the summary text X generated in step 1, and the output is the target keyword sequence Y. First, the summary text X is converted into an embedding vector, which is input into the encoder after being processed by the embedding layer and position encoding to generate a feature representation. The decoder gradually generates the target keywords based on the output of the encoder and the generated keyword sequence. The model training uses a cross-entropy loss function, including encoder loss and decoder loss. The model parameters are optimized through the back-propagation algorithm so that the generated keyword sequence matches the real keywords as much as possible; Step 2.2, generate a preliminary keyword sequence: input the summary X generated in step 1 into the trained keyword generation model, and use the decoder part of the model to generate a preliminary keyword sequence {g1, g2,..., g n}. This process predicts each keyword step by step until an end symbol is generated or the preset keyword quantity limit is reached; Step 2.3, use the automatically generated keyword sequence to optimize and filter the keyword set manually input by the user to obtain the final filtered keyword set. The specific steps are as follows:

[0023] Step 2.3.1, user keyword input: The module receives a set of keywords {k1, k2, ..., k m}; Step 2.3.2, automatic keyword generation: Use the text generation model to automatically generate a keyword sequence {g1, g2, ..., g n}; Step 2.3.3, similarity calculation and weight assignment: Calculate the similarity between each automatically generated keyword and each keyword entered by the user, and use methods such as cosine similarity to obtain the weight w of each automatically generated keyword. i ; Step 2.3.4, keyword optimization: according to weight w i , select the top k keywords with the highest similarity as the input of subsequent modules.

[0024] Step 3, title generation: construct an interactive representation of the abstract and keywords, and generate the final title through the interactive representation. It includes the following steps:

[0025] Step 3.1: Convert the keyword set into a query array K and convert the summary X into a latent representation array L text , the specific steps are as follows:

[0026] Step 3.1.1: Input the keyword set obtained in step 2 into the text keyword interaction enhancement module, and use the same word embedding model as in step 2 to embed each keyword k i The vector kw mapped to the high-dimensional space i, forming a vector set {kw1,kw2,…,kw n}, and output the query array K as the matrix representation of the vector set; Step 3.1.2, split the summary X into sentences, and convert each word w i Convert to word vector v i Each word vector is combined with the position embedding p i and type embedding t i , and obtain the comprehensive embedding representation x i , to enhance the accuracy of the representation of sequence and type information; potential representation generation: through the autoencoder, the comprehensive embedding representation x i Convert to latent representation array L text , accurately capture and compress the key information in the text;

[0027] Step 3.2: Analyze the query array K and the potential representation array L text The relationship between , calculate the attention score, and generate the feature representation of the fusion keyword and summary content, that is, the output array, based on these scores. The specific steps are as follows:

[0028] Step 3.2.1, interactive feature representation construction: Analyze the query array K and potential representation array L obtained in steps 3.1.1 and 3.1.2 text The dynamic relationship between them is to calculate the query, key and value of the two. Then, the attention score Attention_energy is calculated by the query, key and value;

[0029] Step 3.2.2, feature representation construction: Finally, the attention score Attention_energy is combined with the text potential array L text The value vector V is multiplied by the value vector V to obtain the feature representation of the fusion keyword features and the core content of the summary, that is, the output array Output Array is used for subsequent title generation.

[0030] In step 3.3, the output array of step 3.2 is converted into a form suitable for the decoding task through an embedding layer. Subsequently, the embedded array is input into a decoding module based on a predefined structure. The decoder generates a title sequence step by step, and the probability of each word generated at each step is calculated by a softmax function. During the training process, the cross entropy loss function is used to optimize the model parameters so that the generated title is as close as possible to the real title. Finally, the title generation module outputs the automatically generated structured document title.

[0031] Beneficial effects of the present invention:

[0032] 1) The present invention provides an innovative method for automatically constructing structured document titles, which uses deep learning and natural language processing technology to overcome the shortcomings of the existing technology, especially when processing documents with complex structures or long lengths, and can effectively generate high-quality titles, which has important practical significance and application value;

[0033] 2) The method of the present invention improves the quality of title generation: by integrating the information of abstracts and keywords, the generated titles can more accurately reflect the core content of the document and have higher relevance and appeal; Applicable to documents with complex structures: This method targets structured documents with complex structures and long contexts, and can effectively process long sequence texts and complex structural information; High degree of automation: The entire process is highly automated, reducing manual intervention and improving the efficiency of information processing;

[0034] 3) The present invention can be widely used in the title generation of various structured documents, such as technical reports, design documents, research papers, patent documents, etc., providing a new solution for automatic text summarization and title generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the overall steps of the present invention;

[0036] Figure 2 It is the overall framework diagram of the present invention;

[0037] Figure 3 It is a relationship diagram between modules of the present invention;

[0038] Figure 4 It is the training flow chart of the keyword generation model in the keyword generation module;

[0039] Figure 5 It is the flowchart of the keyword generation module;

[0040] Figure 6 It is a schematic diagram of the principle of the text keyword interaction enhancement module;

[0041] Figure 7 It is the flowchart of the title generation module. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0043] A method for automatically constructing structured document titles, such as Figure 2 As shown, specifically including the following:

[0044] Step 1: Construct and train a summary generation model in summary generation. The summary generation model uses the structural information and content features of the structured document to automatically generate a summary of the document. The specific implementation method is as follows:

[0045] (1) Hierarchical encoding, word-level representation: Each word w in the structured document is i Convert to word vector e i , use a pre-trained word embedding model (such as Word2Vec or GloVe) to obtain word vector representation. i represents the i-th word in the document, e i is the corresponding word vector; sentence level representation: use a bidirectional recurrent neural network to represent the word vector sequence Encode and get the sentence level representation s j Among them, s j Represents the vector representation of the jth sentence, where T is the total number of words in the sentence; Paragraph-level representation: further use a bidirectional recurrent neural network to represent the sentence-level sequence Encode and get the paragraph level representation p k Among them, p k represents the vector representation of the kth paragraph, and M is the total number of sentences in the paragraph.

[0046] (2) Importance weight calculation: The importance weight of each element is calculated at the word level and sentence level. The specific calculation method is as follows:

[0047] Importance weight of word vector:

[0048]

[0049] Among them, e i is the word vector of the i-th word, s j is the sentence vector of the jth sentence;

[0050] Importance weights of sentence vectors:

[0051]

[0052] Among them, s j is the sentence vector of the jth sentence, p k is the paragraph vector of the kth paragraph.

[0053] Each word vector e i Multiply it by its importance weight α i And sum to get the updated sentence vector s′ j :

[0054]

[0055] The sentence vector s′ j Multiply it by its importance weight β j And sum them up to get the updated paragraph vector p′ k :

[0056]

[0057] (3) Summary generation: The decoder decodes the multi-level representation and attention weights generated by the encoder to generate the summary text X. The decoder uses a sequence-to-sequence model based on the attention mechanism to ensure that the generated summary is coherent and accurately reflects the core content of the document. The specific process includes: Initial state: The initial hidden state of the decoder is composed of the paragraph-level representation p generated by the encoder. k Mapping; time step recursion: at each time step t, the decoder receives the output y of the previous step t-1 and the current hidden state s t-1 , generate a new hidden state s t ; Calculate the context vector c of the current time step t :

[0058]

[0059] Where N is the number of paragraphs, and all updated paragraph vectors p′ k Perform the summation.

[0060] Output generation: Combined with hidden state s t and the context vector c t , generate the probability distribution P(y t |y <t ,X):

[0061] P(y t |y ＜t , X) = Softmax(W o [s t ;c t ]+b o )

[0062] Among them, [s t ;c t ] means to hide the state s t and the context vector c t Connect; W o is the weight matrix of the output layer; b o is the bias vector of the output layer.

[0063] According to the probability distribution P(y t |y <t ,X) Select the generated word y at the current time step tThrough the above process, the decoder generates a word at each time step until the complete summary text X is generated.

[0064] Model training: Using the cross entropy loss function, calculate the generated summary The difference between the true summary X and the true summary X is:

[0065]

[0066] Where T′ is the total number of words in the abstract; P(y t |y <t ,X) is the generated word y t probability.

[0067] Through the back-propagation algorithm, the gradient of the loss function relative to the model parameters is calculated, and the model parameters are updated using an optimization algorithm (such as Adam). Through continuous iteration, the summary generated by the model Be as close to the true summary X as possible and improve the accuracy and coherence of the summary.

[0068] Step 2: Keyword generation module: This module uses the BART model to perform the keyword generation task. The BART model consists of a bidirectional encoder and an autoregressive decoder. First, the summary text generated in step 1 is extracted to obtain the text sequence X. The input text sequence is converted into an embedding vector. These embedding vectors are obtained through a learnable embedding layer, and each word is mapped to a vector in a high-dimensional space. Then position encoding is added to the embedding vector to provide the model with information about the position of the word in the text. Figure 4 As shown, the specific steps of the process are as follows:

[0069] The input text sequence is X = [x1, x2, ..., x n ], where x i is the i-th word in the text, and each word x i It is converted into an embedding vector e through the embedding matrix E i :

[0070] e i =E[x i ]

[0071] Among them, E is the embedding layer, which is a learned parameter matrix that maps the vocabulary to a high-dimensional space. In order to enable the model to understand the position of words in the text, the positional encoding is added to the embedding vector of each word:

[0072] e′ i =e i +PositionEncoding(i)

[0073] PositionEncoding(i) is a vector associated with position i, typically computed using a combination of sine and cosine functions.

[0074] Add the positional encoding embedding vector e′ i After being processed by the encoder, that is, through multiple layers of self-attention and feed-forward networks, the embedding vector is converted into a context-rich feature representation C. The encoder is composed of multiple identical layers stacked together, each of which contains the following two sub-layers:

[0075] (1) Multi-head self-attention: The input sequence is first converted into three different vector sets: query, key, and value:

[0076]

[0077] Among them, d k is the dimension of the key vector, which is used to scale the dot product to prevent the gradient from disappearing due to excessive dot product values; Q, K, V are query, key, and value matrices, which are obtained by applying different linear transformations to the input e′:

[0078] Q=e′W Q ; K = e′W K ; V = e′W V

[0079] Among them, W Q , W K , W V is a learnable weight matrix. The self-attention layer determines how much each element in the sequence influences other elements by computing the dot product of the query and the key. The attention weights (the output of softmax) are multiplied by the value to produce a weighted representation that incorporates information from the entire sequence.

[0080] (2) Feedforward fully connected network: The output of each self-attention layer passes through a feedforward fully connected network:

[0081] FFN(x)=max(0,xW1+b1)W2+b2

[0082] Among them, W1, W2 are the weights of linear transformation; b1, b2 are bias terms; max(0,z) is the ReLU activation function.

[0083] For each sub-layer (self-attention and feed-forward network), the output is added to the input through a residual connection and layer normalization is applied:

[0084] LayerNorm(x+Sublayer(x))

[0085] This helps avoid the vanishing gradient problem in deep networks.

[0086] The decoder also consists of multiple identical layers, each of which consists of three sub-layers:

[0087] (1) Masked Multi-Head Self-Attention: Masking is implemented by setting the value of future positions to infinity in the self-attention calculation (before applying softmax), preventing future words from being looked at when generating the current word.

[0088] (2) Encoder-Decoder Attention: Here the query comes from the previous layer of the decoder, while the key and value come from the output of the encoder, which allows each position of the decoder to access the entire information of the input sequence.

[0089] (3) Feedforward fully connected network: The structure is the same as the feedforward network in the encoder.

[0090] Each sub-layer also contains a residual connection and layer normalization.

[0091] The input received by the decoder at each time step j includes the output C of the encoder and the embedding Y of the keywords generated in all previous time steps prev ={y1,y2,…,y j-1}, the decoder calculates and generates new keywords y based on these inputs j , until a sequence end condition is reached (such as a special end symbol or a maximum length limit).

[0092] Model training:

[0093] The input of the model is X = [x1, x2, ..., x n ], which is the sequence of the entire body of the paper, where each x i is a word in the text; the output is Y = [y1, y2, ..., y m ], is the target keyword sequence, where each y i is a keyword.

[0094] The loss function consists of two parts, and its calculation formula is as follows:

[0095] (1) Encoder loss:

[0096]

[0097] Among them, p(x i |x1,...,x i-1 ) represents the conditional probability of the i-th word given the previous i-1 words. This loss function encourages the encoder to learn a strong internal representation of the text to support the decoder's keyword extraction.

[0098] (2) Decoder loss:

[0099]

[0100] Among them, C is the context output of the encoder, p(y j |y1,...,y j-1 ,C) represents the conditional probability of generating the jth keyword given the previously generated keywords and the context output by the encoder.

[0101] (3)Total loss:

[0102] L total =L encoder +αL decoder

[0103] Here, α is a hyperparameter used to balance the loss contributions of the encoder and decoder.

[0104] In the reasoning process, Figure 5 As shown, the complete paper text to be analyzed is first converted into a sequence of embedded vectors, which are input into the encoder after position encoding. The encoder processes the text through a multi-layer self-attention mechanism and a feedforward network to generate a high-level representation containing full-text context information. The decoder then uses these representations, combined with the generated keyword sequence (triggered by a special start symbol at the beginning), to iteratively predict the next keyword. This process continues until a special end symbol is generated or the preset number of keywords is reached. On this basis, the present invention further introduces a keyword screening mechanism to optimize the quality of keywords input into the text keyword interaction enhancement module. The specific implementation steps are as follows:

[0105] Suppose the set of manually input keywords is {k1, k2, ..., k m}, the keyword set automatically generated by the model is {g1,g2,...,g n}. For each manually entered keyword k i (where i = 1, 2, ..., m), calculate its difference with each automatically generated keyword g j (where j = 1, 2, ..., n) between the cosine similarity cos(k i ,g j ). Then, for each manually entered keyword k i Calculate a total score S i , as shown below:

[0106]

[0107] Next, for all S i Normalize to get the final score N for each keywordi :

[0108]

[0109] Based on these scores, the top k manually entered keywords with the highest scores are selected and used as the input to the text keyword interaction enhancement module. This screening process ensures that the selected keywords are highly relevant to the article content and their importance in the model.

[0110] Step 3: Use the text-keyword interaction enhancement module to process the main text and generated keywords. In this module, the main text is used as input features, and the generated keywords are used as output query features. This module deeply analyzes the dynamic relationship between the main text and keywords through the cross-attention mechanism. In this process, the system will generate an output feature of a title rich in the core ideas and key information of the main text based on the interaction between the main text content and keywords. The specific steps are as follows:

[0111] Step 3.1: Conversion of body text to input array (Input Array): Split the body text of the structured document into sentences {s1, s2, …, s n}, where n is the total number of sentences; each sentence s i Further segmented into {w i1 ,w i2 ,…,w im}, where m is the sentence s i The total number of words in w; use a pre-trained word embedding model (such as BERT or GloVe) to embed each word w ij Convert to the corresponding word vector v ij For sentence s i Each word w in ij , calculate its position embedding p ij = PositionalEncoding(j), position embedding is generated using a variant of the sine and cosine functions to encode the relative position information of the word in the sentence. In order to distinguish different types of text (such as abstract, body, conclusion, etc.), type embedding is introduced:

[0112] t ij =TypeEmbedding(type_of(s i ))

[0113] Among them, type_of(s i ) represents sentence s i The type embedding assigns a unique vector to each text type, for example, the abstract may have one embedding, while each part of the body may have another. The final input array X is constructed by dividing the word vector v of each word by ij, position embedding p ij and type embedding t ij For sentence s i Each word w in ij , and its final embedding is expressed as:

[0114]

[0115] in Represents the concatenation operation of the vector. The input array X is the concatenated embedding vector x of all words in all sentences. ij The matrix formed.

[0116] Step 3.2: The overall process from input to output of the text keyword interaction enhancement module is as follows Figure 6 As shown in Figure 1, the input array passes through an autoencoder, which is responsible for converting the high-dimensional text input vector into a higher-level latent array. This latent array captures the key information and internal structure of the input data, providing an information-rich and compressed data representation for subsequent processing. The encoder performs cross-attention between the fixed-size latent array and the input array to obtain a compressed representation. Then, a series of self-attention operations are performed between the latent arrays to obtain L text , which is independent of the input size, so it is able to stack deeper layers using the self-attention module. Finally, the decoder performs cross attention between the latent array and the output array to produce a shape with the desired size. The calculation of cross attention crossattention is as follows:

[0117] Step 3.3: Input the main text into the keyword generation module to obtain a series of keywords {k1, k2, …, k n}, use the word embedding model to embed each keyword k i Mapped to a point in a high-dimensional space, forming a vector set {kw1, kw2, …, kw n}, the output query array K is a matrix formed by the set of vectors.

[0118] Step 3.4: Decode the latent array L of the body text through the criss-cross attention module in the decoder text , specifically, the text potential array L text As the key and value, the output query array K of the keyword is used as the query. The two interact through the cross-attention mechanism to extract key information and build highly relevant feature representations to obtain the final output array output array. The formula is as follows:

[0119] Q=W q K; K = W k ·L text; V = W v ·L text

[0120] Among them, Q, K, V represent query, key and value respectively, and W q , W k , W v is a learnable weight matrix used to transform the array into a format suitable for criss-cross attention processing.

[0121]

[0122] Among them, d k is the dimension of the key vector. This formula calculates the semantic relevance between keyword information and body text, and normalizes it through the softmax function to obtain the contribution weight of each body feature to the keyword.

[0123] OutputArray=CrossAttention(Q,K,V)=Attention_energy·V

[0124] Among them, CrossAttention represents the final interactive feature representation, which is used as the final output array Output Array, which enables the model to focus on the keyword-related parts of the text while retaining the contextual information of the keywords.

[0125] Step 3.5: If Figure 7 As shown in the figure, the output array obtained from the text-keyword interaction enhancement module in step 3.4 contains the interactive features of text and keywords, which is used as the input array X of the title generation module. The array is an n×d-dimensional matrix, where n represents the sequence length and d represents the feature dimension. The input X is first converted into a form more suitable for the decoding task through an embedding layer. Subsequently, the array is fed into a decoder based on the Transformer architecture (a deep learning architecture specifically designed for processing sequence data). Each layer in the decoder performs the following operations:

[0126] (1) Multi-head self-attention mechanism processing:

[0127] H (l) =Attention(Q (l) ,K (l) ,V (l) )

[0128] Among them, Q (l) , K (l) , V (l) They are query, key, and value matrices, which are the output H of the previous layer. (l-1)(or X for the first layer) by different weight matrices These weight matrices are the parameters in the model, which are learned through training.

[0129] (2) Feedforward fully connected network:

[0130]

[0131] Among them, W1 ( l ) and W2 ( l ) is the weight of the linear transformation, and and are the bias terms. These are also parameters learned through training.

[0132] The decoder output H( L ) is used to generate titles word by word. The probability of the words generated at each step is given by:

[0133] p(y t |y1,y2,...,y t-1 )=softmax(H (L) W out )

[0134] Among them, W out It is the mapping matrix from the output of the last layer of the decoder to the word probability. The softmax function is used to convert the output vector of the decoder into a probability distribution.

[0135] During training, the cross entropy loss function is used to optimize the model. The specific loss calculation formula is:

[0136]

[0137] in, is the tth word in the actual title, and T is the total number of words in the title. This loss function helps the model learn to generate word sequences that are as close as possible to the true title.

[0138] Through these steps, the title generation module can effectively transform the comprehensive features of text and keywords into an accurate title that describes the core topic of the article, realizing the conversion from deep features to human-readable text.

[0139] In order to further verify the effectiveness of the method of the present invention, the present invention selected the abstracts and keywords of two scientific papers to compare the actual generation effects of different methods, as shown in Table 1. The results show that the title generated by the method of the present invention has the highest similarity with the original title. The generation method that only relies on the abstract will have the problem of inaccurate capture of key information, such as the first example in Table 1; because the amount of information in the keywords is too little, the title generated by the generation method that only relies on the abstract cannot cover all the core content of the text, such as the second example in Table 1. The title generation results and similarity results of the two examples show that the method of the present invention can more accurately capture and express the core theme of the article, has a higher content coverage, and can effectively improve the quality and relevance of title generation.

[0140] Table 1 Actual generation effects of different methods

[0141]

[0142] Automatic generation of abstracts and keywords: Through the abstract generation module and keyword generation module, the core content and important keywords of the document are automatically extracted, which improves the efficiency and accuracy of information extraction. Abstract generation makes full use of the structural hierarchical information of the document, and keyword generation combines user input and automatically generated keywords to optimize the keyword selection process;

[0143] Application of cross-attention mechanism: The cross-attention mechanism is used to achieve deep interaction between keywords and summaries, which enhances the understanding of the core content of the document. By analyzing the dynamic relationship between keywords and summaries, the generated feature representation can more accurately reflect the theme and important information of the document, improving the quality and relevance of title generation.

[0144] The embodiments described in the present invention are for reference only and are not intended to limit the scope of protection of the present invention. According to the basic principles of the present invention, the embodiments of the present invention may have various changes and modifications. These changes and modifications are all under the premise of not departing from the purpose of the present invention, and should also be regarded as a part of the scope of protection of the present invention. In addition, all embodiments and modifications included in the present invention should be regarded as equivalent implementation forms under a wide range of technical ideas.

Claims

1. A method for automatically constructing a structured document title, characterized in that: Capable of automatically generating titles for structured documents with long contexts, including the following steps: Step 1, generate a summary of a structured document: train a summary generation model that can generate a structured document summary X; Step 2: Generate keywords for structured documents: First, train a text generation model, and then use the text generation model to generate keyword sequences from the summary X; Step 3, title generation: construct an interactive representation of the abstract and keywords, and generate the final title through the interactive representation.

2. The method for automatically constructing a structured document title according to claim 1, characterized in that: The specific process of step 1 is as follows: Step 1.1, for document D i Perform hierarchical coding and calculate the importance weights of elements at the word level and sentence level respectively, so as to effectively extract content elements that have an impact on summary generation; Step 1.2: Build a neural network model for summary generation. The neural network model uses the importance weights of each element at each level obtained in step 1.1 to perform weighted summation on the outputs of the word-level, sentence-level, and paragraph-level encoders to obtain the context vector c. t , and according to the context vector c t and the hidden state s t Use sequence-to-sequence decoder to generate document D i Summary of Step 1.3, use the cross entropy loss function to train the summary generation model through the error back propagation algorithm, optimize the model parameters, and make the generated summary Close to the true summary X, to achieve the structured document D i Automatic generation of summaries.

3. The method for automatically constructing a structured document title according to claim 2, characterized in that: The specific implementation steps of step 1.1 are as follows: Step 1.1.1: For each word w in the structured document i Convert to word vector e i ; Step 1.1.2: Use a bidirectional recurrent neural network to encode the word vector sequence and generate a sentence vector s j ; Step 1.1.3: Use a bidirectional recurrent neural network to encode the sentence vector sequence to obtain the paragraph vector p k ; Step 1.1.4: At the word level, sentence level, and paragraph level, extract the importance weights of the elements at each level layer by layer, and use these weights to update the corresponding sentence vector and paragraph vector.

4. The method for automatically constructing a structured document title according to claim 3, characterized in that: In the step 1.1.4, at the word level, the similarity between each word vector and its corresponding sentence vector is calculated to determine the importance of each word in the sentence; at the sentence level, the similarity between each sentence vector and its corresponding paragraph vector is further calculated to determine the relative importance of the sentence in the paragraph; The specific process of step 1.1.4 is as follows: The weight of each word is multiplied by its corresponding word vector to obtain a weighted sentence vector, and then the weight of each sentence is multiplied by the updated sentence vector to obtain a weighted paragraph vector; the weighted vector representation is used for the calculation of contextual information for subsequent summary generation.

5. The method for automatically constructing a structured document title according to claim 4, characterized in that: The specific process of step 2 is as follows: Step 2.1, training the keyword generation model in the keyword generation module: adopting a method based on bidirectional encoding and autoregressive decoding as the basic model for keyword generation; during the training process, the input is the summary text X generated in step 1, and the output is the target keyword sequence Y; Step 2.2, generate a preliminary keyword sequence: Input the summary X generated in step 1 into the trained keyword generation model, and use the decoder of the model to generate a preliminary keyword sequence {g1, g2, ..., g n }; During the generation of the preliminary keyword sequence, each keyword is predicted step by step until an end symbol is generated or a preset keyword quantity limit is reached; Step 2.3, using the automatically generated keyword sequence to optimize and filter the keyword set manually input by the user to obtain the final filtered keyword set.

6. A method for automatically constructing a structured document title according to claim 5, characterized in that: The specific process of step 2.1 is as follows: Step 2.1.1, convert the summary text X into an embedded vector, and input it into the encoder after processing through the embedding layer and position encoding to generate feature representation; Step 2.1.2, the decoder gradually generates the target keywords based on the output of the encoder and the generated keyword sequence; Step 2.1.3: The model training adopts the cross entropy loss function, including encoder loss and decoder loss, and optimizes the model parameters through the back propagation algorithm so that the generated keyword sequence matches the real keyword.

7. The method for automatically constructing a structured document title according to claim 6, characterized in that: In step 2.3, keywords are automatically generated from the abstract through the text generation model, and the manually input keywords are screened using the weight distribution mechanism; the specific implementation steps are as follows: Step 2.3.1, user keyword input: The module receives a set of keywords {k1, k2, ..., k m }; Step 2.3.2, automatic keyword generation: Use the text generation model to automatically generate a keyword sequence {g1, g2, ..., g n }; Step 2.3.3, similarity calculation and weight assignment: Calculate the similarity between each automatically generated keyword and each keyword entered by the user, and obtain the weight w of each automatically generated keyword. i ; Step 2.3.4, keyword optimization: according to weight w i , select the top k keywords with the highest similarity as the input of subsequent modules.

8. The method for automatically constructing a structured document title according to claim 7, characterized in that: The specific process of step 3 is as follows: Step 3.1: Convert the keywords generated in step 2 into vectors through the word embedding model to form a query array K. At the same time, after the summary X is divided into sentences, each word is converted into a word vector, and combined with the position and type information to obtain a comprehensive embedding representation, and then the potential representation array L is generated through the autoencoder. text ; Step 3.2: Analyze the query array K and the potential representation array L text The relationship between , calculate the attention score, and generate the feature representation of the fusion keyword and summary content according to the attention score, that is, the output array Output Array; In step 3.3, the output array obtained in step 3.2 is converted into a form suitable for the decoding task through an embedding layer; then, the embedded array is input into a decoding module based on a predefined structure, and the decoding module generates a title sequence step by step, and the probability of the words generated at each step is obtained by the corresponding calculation method; during the training process, the cross entropy loss function is used to optimize the model parameters so that the generated title is as close to the real title as possible; finally, the title generation module outputs the automatically generated structured document title.

9. The method for automatically constructing a structured document title according to claim 8, characterized in that: In step 3.1 and step 3.2, the features of the fused keyword features and the document content are generated in the text keyword interaction enhancement module; the specific implementation steps are as follows: Step 3.1.1: Input the keyword set obtained in step 2 into the text keyword interaction enhancement module, and use the word embedding model to embed each keyword k i The vector kw mapped to the high-dimensional space i , forming a vector set {kw1,kw2,…,kw n }, and output the query array K as a vector set {kw1,kw2,…,kw n }Matrix representation; Step 3.1.2, split the summary X into sentences and convert each word w i Convert to word vector v i ; Each word vector is combined with the position embedding p i and type embedding t i , and obtain the comprehensive embedding representation x i , to enhance the accuracy of the representation of sequence and type information; potential representation generation: through the autoencoder, the comprehensive embedding representation x i Convert to latent representation array L text , accurately capture and compress the key information in the text; Step 3.2.1, interactive feature representation construction: Analyze the query array K and potential representation array L obtained in steps 3.1.1 and 3.1.2 text The dynamic relationship between them is to calculate the query, key and value of the two; the attention score Attention_energy is calculated through the query, key and value; Step 3.2.2, feature representation construction: Construct the attention score Attention_energy with the text potential array L text The value vector V is multiplied by to obtain the feature representation of the fusion keyword features and the core content of the summary, that is, the output number OutputArray, which is used for subsequent title generation.

Images