An automatic construction method of structured document title

Abstract

The application provides an automatic construction method of structured document titles, which can automatically generate titles for structured documents with long context. The method comprises the following steps: first, generating an abstract of the structured document by using the structural information and content features of the document through an abstract generation module based on a hierarchical attention network; second, generating keywords from the abstract by using a text generation model based on a bidirectional encoder and an autoregressive decoder, and optimizing the keywords through a keyword screening mechanism; third, analyzing the interaction between the screened keywords and the abstract by using a cross-attention mechanism, and generating a feature representation that integrates the keyword features and the document content; and fourth, automatically constructing a title for the structured document according to the feature representation. The entire process is automated, and a title related to the content of the structured document can be obtained.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, specifically a method for automatically constructing structured document titles. Background Technology

[0002] With the rapid development of information technology and data analysis, more and more researchers and technicians are focusing on the application of natural language processing technology in structured document writing. The title of a structured document (such as an elevator design document) is not only an important component but also crucial for conveying the core content and attracting the reader's attention. In the process of writing structured documents, an accurate and concise title can greatly improve the document's readability and practical value. Traditional title generation methods usually require authors to manually create titles based on the document's core content and work results, a process that is both time-consuming and inefficient. Furthermore, manually generated titles may fail to fully cover the document's key points due to limitations in personal experience and knowledge.

[0003] Currently available title generation methods, such as patent document CN115496061A entitled "A Method for Constructing a Neural Network Title Generation Model," mainly propose a neural network structure combining multi-layer attention mechanisms and information-rich word vector layers to enhance the coherence and conciseness of title generation. However, it does not effectively address the problem of handling complex structures and technical terms in structured documents. Patent document CN106383817A entitled "A Method for Generating Paper Titles Using Distributed Semantic Information," primarily uses the TextRank algorithm to extract keywords and an LSTM-based recurrent neural network to generate titles. However, it relies solely on keywords and does not fully utilize the abstract content, and the TextRank algorithm may not accurately extract representative keywords. Patent document CN117787260A entitled "A Text Title Generation Method Based on Adaptive Position Encoding and Knowledge Reasoning," provides a method to improve title generation quality through adaptive position encoding and knowledge reasoning, but it does not effectively handle the problem of generating titles for long sequences or structurally complex structured documents.

[0004] In summary, while existing title generation methods have made some progress in various aspects, they still have limitations, especially in handling complex or lengthy structured documents. Furthermore, these methods often over-rely on keywords while failing to fully utilize the abstract content, or they rely solely on the abstract while ignoring the role of keywords, and the accuracy of keyword extraction is often insufficient. Some current methods rely solely on the abstract content, which can lead to title generation being affected by redundant information in the abstract; others rely solely on keywords, which, while indicating the main theme of the document, usually fail to cover all core content or fully represent the amount of information the title needs to convey.

[0005] Therefore, an ideal title generation method should effectively combine abstract content and keywords, leveraging the advantages of both, reducing redundant information in the abstract, and enhancing the representativeness of the keywords to generate more accurate and attractive titles. However, currently, there is no method for generating titles based on the fusion of abstract and keywords. The abstract and keyword fusion method proposed in this invention addresses this gap, providing an innovative solution. Summary of the Invention

[0006] This invention relates to a method for automatically constructing structured document titles, aiming to automatically generate accurate and clear titles for structured documents with complex structures and long contexts. This method comprehensively utilizes techniques such as summary generation, keyword generation and filtering, interactive feature representation construction, and title generation, overcoming the shortcomings of existing technologies in handling title generation for complex document structures and improving the quality and relevance of title generation.

[0007] The method of the present invention includes the following steps:

[0008] Step 1, Generate a summary of the structured document: Train a summary generation model that can generate summaries of structured documents. This includes the following sub-steps: Step 1.1, For document D i To perform hierarchical encoding, first, each word in the document... i Convert to word vector e i A bidirectional recurrent neural network is used to encode words, sentences, and paragraphs respectively, and importance weights for words, sentences, and paragraphs are obtained. The specific implementation steps are as follows:

[0009] First, each word w in the structured document i Convert to word vector e i Secondly, a bidirectional recurrent neural network is used to encode the word vector sequence to generate sentence vectors s. j Then, a bidirectional recurrent neural network is used to encode the sentence vector sequence to obtain the paragraph vector p. k Next, the importance weights of each element at both the word and sentence levels are calculated, as follows:

[0010] Importance weights of word vectors:

[0011]

[0012] Among them, e i s is the word vector of the i-th word. j It is the sentence vector of the j-th sentence;

[0013] Importance weights of sentence vectors:

[0014]

[0015] Among them, s j p is the sentence vector of the j-th sentence. k It is the paragraph vector of the k-th paragraph.

[0016] Each word vector e i Multiply by its importance weight α i Then sum them up to obtain the updated sentence vector s′. j :

[0017]

[0018] The sentence vector s′ j Multiply by its importance weight β j Then sum them up to obtain the updated paragraph vector p′. k :

[0019]

[0020] Step 1.2: Construct a neural network model for summarization. This model uses the importance weights of each element at each level obtained in Step 1.1 to perform a weighted summation of the outputs of the word-level, sentence-level, and paragraph-level encoders to calculate the context vector c. t And based on the context vector c t and hidden state s t Document D is generated using a sequence-to-sequence decoder. i The summary, in which the hidden states s t From the output y of the previous step t-1 and the current hidden state s t-1 Generation; Step 1.3, using the cross-entropy loss function, the summary generation model is trained through the error backpropagation algorithm to optimize the model parameters, so that the generated summary... To obtain a structured document D that is as close as possible to the actual summary X, thereby achieving the desired result. i Automatic generation of abstracts.

[0021] Step 2, Generate keywords for the structured document: First, train a text generation model, then use this model to generate a keyword sequence from the summary X in the keyword generation module. This includes the following steps:

[0022] Step 2.1, Training the Keyword Generation Model in the Keyword Generation Module: A method based on bidirectional encoding and autoregressive decoding is used as the basic model for keyword generation. During training, the input is the summary text X generated in Step 1, and the output is the target keyword sequence Y. First, the summary text X is converted into an embedding vector, processed by an embedding layer and positional encoding, and then input into the encoder to generate feature representations. The decoder then generates target keywords step by step based on the encoder output and the generated keyword sequence. The model training uses the cross-entropy loss function, including encoder loss and decoder loss, and optimizes the model parameters through backpropagation to make the generated keyword sequence match the real keywords as closely as possible. Step 2.2, Generating the Preliminary Keyword Sequence: The summary X generated in Step 1 is input into the trained keyword generation model, and the decoder part of the model is used to generate the preliminary keyword sequence {g1, g2, ..., g...}. n This process involves progressively predicting each keyword until an end symbol is generated or a preset keyword quantity limit is reached. Step 2.3 involves using an automatically generated keyword sequence to optimize and filter the user-input keyword set, resulting in the final filtered keyword set. The specific steps are as follows:

[0023] Step 2.3.1, User Keyword Input: The module receives a set of keywords related to the abstract, {k1,k2,...,k...}, manually entered by the user. m Step 2.3.2, Automatic Keyword Generation: Utilize a text generation model to automatically generate a keyword sequence {g1, g2, ..., g...} from the abstract X. n Step 2.3.3, Similarity Calculation and Weight Allocation: Calculate the similarity between each automatically generated keyword and each keyword input by the user, using methods such as cosine similarity, to obtain the weight w for each automatically generated keyword. i Step 2.3.4, Keyword Optimization: Based on weight w i The top k keywords with the highest similarity are selected as input for subsequent modules.

[0024] Step 3, Title Generation: Construct an interactive representation of the abstract and keywords, and generate the final title using this representation. This includes the following steps:

[0025] Step 3.1: Convert the keyword set into a query array K, and convert the summary X into a latent representation array L. text The specific steps are as follows:

[0026] Step 3.1.1: Input the keyword set obtained in Step 2 into the text keyword interaction enhancement module, and use the same word embedding model as in Step 2 to embed each keyword k i Vector kw mapped to high-dimensional space iThis forms a vector set {kw1,kw2,…,kw}. n}, and output the query array K as the matrix representation of the vector set; Step 3.1.2, divide the summary X into sentences, and output each word w i Convert to word vector v i Each word vector is combined with the position embedding p. i and type embedding t i The comprehensive embedding representation x is obtained. i To enhance the accuracy of representing sequence and type information; latent representation generation: through an autoencoder, the synthesized embedded representation x i Convert to a potential representation array L text It accurately captures and compresses key information in text;

[0027] Step 3.2, analyze the query array K and the latent representation array L text The relationship between keywords and summary content is analyzed, attention scores are calculated, and feature representations that fuse keywords and summary content are generated based on these scores, resulting in an output array. The specific steps are as follows:

[0028] Step 3.2.1, Construction of Interactive Feature Representation: Analyze the query array K and latent representation array L obtained in Steps 3.1.1 and 3.1.2. text The dynamic relationship between them is calculated by interpreting the query, key, and value. Then, the attention score, Attention_energy, is calculated using the query, key, and value.

[0029] Step 3.2.2, Feature Representation Construction: Finally, the attention score Attention_energy is compared with the latent array L of the main text. text Multiplying the value vector V yields the feature representation of the fused keyword features and the core content of the abstract, i.e., the Output Array, which is used for subsequent title generation.

[0030] Step 3.3 transforms the output array from Step 3.2 into a format suitable for the decoding task through an embedding layer. The embedded array is then fed into a decoding module based on a predefined structure. The decoder progressively generates a sequence of titles, with the probability of each generated word calculated using a softmax function. During training, a cross-entropy loss function is used to optimize the model parameters, ensuring the generated titles are as close as possible to the actual titles. Finally, the title generation module outputs automatically generated structured document titles.

[0031] The beneficial effects of this invention are:

[0032] 1) This invention provides an innovative method for automatically constructing structured document titles. By utilizing deep learning and natural language processing technologies, it overcomes the shortcomings of existing technologies. In particular, when processing documents with complex structures or long lengths, it can effectively generate high-quality titles, which has significant practical significance and application value.

[0033] 2) The method of this invention improves the quality of title generation: by integrating information from the abstract and keywords, the generated title more accurately reflects the core content of the document and has higher relevance and attractiveness; it is suitable for documents with complex structures: this method is designed for structured documents with complex structures and long contexts, and can effectively process long sequences of text and complex structural information; it has a high degree of automation: the entire process is highly automated, reducing manual intervention and improving the efficiency of information processing.

[0034] 3) This invention can be widely applied to the generation of titles for various structured documents, such as technical reports, design documents, research papers, and patent documents, providing a new solution for automated text summarization and title generation. Attached Figure Description

[0035] Figure 1 This is a flowchart of the overall steps of the present invention;

[0036] Figure 2 This is the overall framework diagram of the present invention;

[0037] Figure 3 This is a diagram showing the relationships between the modules of this invention;

[0038] Figure 4 This is a flowchart of the keyword generation model training process in the keyword generation module;

[0039] Figure 5 This is a flowchart of the keyword generation module;

[0040] Figure 6 This is a schematic diagram illustrating the principle of the text keyword interaction enhancement module;

[0041] Figure 7 This is the flowchart for the title generation module. Detailed Implementation

[0042] The technical solution of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0043] A method for automatically constructing structured document titles, such as... Figure 2 As shown, it specifically includes the following:

[0044] Step 1: Build and train a summary generation model. This model automatically generates a summary of a document using the structural information and content features of the document. The specific implementation is as follows:

[0045] (1) Hierarchical encoding and word-level representation: Each word in the structured document is encoded and represented at the word level. i Convert to word vector e i Pre-trained word embedding models (such as Word2Vec or GloVe) are used to obtain word vector representations. Where w i Represents the i-th word in the document, e i The corresponding word vectors; sentence-level representation: using a bidirectional recurrent neural network to process the word vector sequence. Encode the data to obtain a sentence-level representation s. j Among them, s j The vector representation of the j-th sentence, where T is the total number of words in the sentence; paragraph-level representation: further utilizing a bidirectional recurrent neural network to process the sentence-level representation sequence. Encode to obtain paragraph-level representation p k Among them, p k Let M be the vector representation of the k-th paragraph, and M be the total number of sentences in the paragraph.

[0046] (2) Importance weight calculation: The importance weight of each element is calculated at the word level and sentence level, respectively. The specific calculation method is as follows:

[0047] Importance weights of word vectors:

[0048]

[0049] Among them, e i s is the word vector of the i-th word. j It is the sentence vector of the j-th sentence;

[0050] Importance weights of sentence vectors:

[0051]

[0052] Among them, s j p is the sentence vector of the j-th sentence. k It is the paragraph vector of the k-th paragraph.

[0053] Each word vector e i Multiply by its importance weight α i Then sum them up to obtain the updated sentence vector s′. j :

[0054]

[0055] The sentence vector s′ j Multiply by its importance weight β j Then sum them up to obtain the updated paragraph vector p′. k :

[0056]

[0057] (3) Summary Generation: The decoder decodes the multi-level representation and attention weights generated by the encoder to generate summary text X. The decoder uses a sequence-to-sequence model based on an attention mechanism to ensure that the generated summary is coherent and accurately reflects the core content of the document. The specific process includes: Initial State: The initial hidden state of the decoder is the paragraph-level representation p generated by the encoder. k Mapping yields the result; time-step recursion: at each time step t, the decoder receives the output y from the previous step. t-1 and the current hidden state s t-1 Generate a new hidden state s t ; Calculate the context vector c of the current time step. t :

[0058]

[0059] Where N is the number of paragraphs, and p′ is the updated vector of all paragraphs. k Perform summation.

[0060] Output generation: combining hidden state s t and context vector c t The probability distribution P(y) of the current word is generated through the output layer. t |y <t ,X):

[0061] P(y t |y ＜t X) = Softmax(W o [s t ;c t ]+b o )

[0062] Among them, [s t ;c t ] indicates that the hidden state s will be hidden. t and context vector c t Connect them; W o b is the weight matrix of the output layer; o This is the bias vector for the output layer.

[0063] According to the probability distribution P(y) t |y <t (X) Select the generated word y at the current time step. tThrough the above process, the decoder generates one word at each time step until a complete summary text X is generated.

[0064] Model training: The generated summary is calculated using the cross-entropy loss function. The difference between the actual summary X and the real summary X is expressed by the formula:

[0065]

[0066] Where T′ is the total number of words in the abstract; P(y t |y <t (X) represents the generated word y t The probability of.

[0067] The backpropagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters, and the model parameters are updated using an optimization algorithm (such as Adam). Through continuous iteration, the model generates a summary. To improve the accuracy and coherence of the abstract, make it as close as possible to the actual abstract X.

[0068] Step 2: Keyword Generation Module: This module uses the BART model for keyword generation. The BART model consists of a bidirectional encoder and an autoregressive decoder. First, it extracts the summary text generated in Step 1, obtaining a text sequence X. The input text sequence is converted into embedding vectors. These embedding vectors are obtained through a learnable embedding layer, where each word is mapped to a vector in a high-dimensional space. Then, position encoding is added to the embedding vectors to provide the model with information about the word's position in the text. For example... Figure 4 As shown, the specific steps of this process are as follows:

[0069] The input text sequence is X = [x1, x2, ..., x...]. n ], where x i It is the i-th word in the text, and each word x i It is transformed into an embedding vector e through the embedding matrix E. i :

[0070] e i =E[x i ]

[0071] Here, E is the embedding layer, which is a learned parameter matrix that maps words to a high-dimensional space. To enable the model to understand the position of words in the text, positional encoding is added to the embedding vector of each word:

[0072] e′ i =e i +PositionEncoding(i)

[0073] PositionEncoding(i) is a vector associated with position i, typically calculated using a combination of sine and cosine functions.

[0074] Add position-encoded embedding vector e′ i After processing by the encoder, which involves multiple layers of self-attention and feedforward networks, the embedded vectors are transformed into context-rich feature representations C. The encoder consists of multiple identical stacked layers, each containing the following two sub-layers:

[0075] (1) Multi-head self-attention: The input sequence is first transformed into three different vector sets: query, key, and value.

[0076]

[0077] Where, d k is the dimension of the key vector, used to scale the dot product to prevent excessively large dot product values ​​from causing gradient vanishing; Q, K, and V are the query, key, and value matrices, obtained by applying different linear transformations to the input e′:

[0078] Q = e′W Q ;K=e′W K V = e′W V

[0079] Among them, W Q W K W V It is a learnable weight matrix. The self-attention layer determines the influence of each element in the sequence on other elements by calculating the dot product of the query and the key. The attention weights (the output of softmax) are multiplied by the value to generate a weighted representation that incorporates information from the entire sequence.

[0080] (2) Feedforward Fully Connected Network: The output of each self-attention layer passes through a feedforward fully connected network.

[0081] FFN(x) = max(0, xW1+b1)W2+b2

[0082] Where W1 and W2 are the weights of the linear transformation; b1 and b2 are the bias terms; and max(0,z) is the ReLU activation function.

[0083] For each sublayer (self-attention and feedforward networks), the output is normalized by applying the layer after being added to the input through a residual connection:

[0084] LayerNorm(x+Sublayer(x))

[0085] This helps avoid the vanishing gradient problem in deep networks.

[0086] The decoder also consists of multiple identical layers, each of which includes three sub-layers:

[0087] (1) Masked multi-head self-attention: Masking is achieved by setting the value of the future position to infinity (before applying softmax) in the self-attention calculation, preventing the view of future words when generating the current word.

[0088] (2) Encoder-decoder attention: Here the query comes from the previous layer of the decoder, while the key and value come from the output of the encoder, which allows each position of the decoder to access the entire information of the input sequence.

[0089] (3) Feedforward fully connected network: The structure is the same as the feedforward network in the encoder.

[0090] Each sublayer also contains a residual connection and layer normalization.

[0091] The input received by the decoder at each time step j includes the encoder's output C and the keyword embeddings Y generated at all previous time steps. prev ={y1,y2,…,y j-1 The decoder calculates and generates new keywords y based on these inputs. j This continues until the sequence ends (e.g., a special end symbol or a maximum length limit is reached).

[0092] Model training:

[0093] The model's input is X = [x1, x2, ..., x...]. n This is the sequence of the entire paper's main text, where each x... i It is a word in the text; the output is Y = [y1, y2, ..., y m ] is the target keyword sequence, where each y i It is a keyword.

[0094] The loss function consists of two parts, and its calculation formula is as follows:

[0095] (1) Encoder loss:

[0096]

[0097] Where p(x) i |x1,...,x i-1 Let represent the conditional probability of the i-th word given the first i-1 words. This loss function encourages the encoder to learn a strong internal representation of the text to support keyword extraction by the decoder.

[0098] (2) Decoder loss:

[0099]

[0100] Where C is the encoder's context output, p(y j |y1,...,y j-1 C) represents the conditional probability of generating the j-th keyword given the previously generated keywords and the context of the encoder output.

[0101] (3) Total loss:

[0102] L total =L encoder +αL decoder

[0103] Here, α is a hyperparameter used to balance the loss contributions of the encoder and decoder.

[0104] In the process of reasoning, such as Figure 5 As shown, the complete paper text to be analyzed is first converted into a sequence of embedded vectors, which are then positionally encoded and input into the encoder. The encoder processes the text through a multi-layer self-attention mechanism and a feedforward network to generate a high-level representation containing full-text context information. The decoder then uses these representations, combined with the generated keyword sequence (triggered by a special start symbol), to iteratively predict the next keyword. This process continues until a special end symbol is generated or a preset number of keywords is reached. Based on this, the present invention further introduces a keyword filtering mechanism to optimize the keyword quality input into the text keyword interaction enhancement module. The specific implementation steps are as follows:

[0105] Let the set of manually entered keywords be {k1,k2,...,k}. m The set of keywords automatically generated by the model is {g1, g2, ..., g}. n For each manually entered keyword k i (where i = 1, 2, ..., m), calculate its relationship with each automatically generated keyword g. j The cosine similarity between (where j = 1, 2, ..., n) is cos(k) i ,g j Then, for each manually entered keyword k i Calculate a total score S i As shown below:

[0106]

[0107] Next, for all S i Normalization is performed to obtain the final score N for each keyword.i :

[0108]

[0109] Based on these scores, the top k manually entered keywords with the highest scores are selected and used as input to the text keyword interaction enhancement module. This selection process ensures that the selected keywords are highly relevant to the article content and their importance in the model.

[0110] Step 3: Process the main text and generated keywords using the text keyword interaction enhancement module. In this module, the main text serves as the input feature, while the generated keywords serve as the output query feature. This module uses a cross-attention mechanism to deeply analyze the dynamic relationship between the main text and keywords. During this process, the system generates an output feature containing the core viewpoints and key information of the main text based on the interaction between the main text content and keywords. The specific steps are as follows:

[0111] Step 3.1: Transformation of the main text into an input array: Segment the main text of the structured document into sentences {s1, s2, ..., s...} n}, where n is the total number of sentences; each sentence s i Further segmentation into {w i1 ,w i2 ,…,w im}, where m is sentence s i The total number of words; each word is embedded using a pre-trained word embedding model (such as BERT or GloVe). ij Convert to the corresponding word vector v ij For sentence s i Each word w in ij Calculate its position embedding p ij =PositionalEncoding(j), the positional embedding is generated using variants of the sine and cosine functions to encode the relative position information of words in the sentence. To distinguish different types of text (such as summary, body text, conclusion, etc.), type embedding is introduced:

[0112] t ij =TypeEmbedding(type_of(s) i ))

[0113] Among them, type_of(s i ) represents sentence s i The type embedding assigns a unique vector to each text type; for example, a summary might have one embedding, while each part of the body might have another. The final input array X is obtained by transforming the word vector v of each word... ijPosition embedding p ij and type embedding t ij The result is obtained by concatenating the parts of sentence s. i Each word w in ij Its final embedding representation is:

[0114]

[0115] in This represents the vector concatenation operation. The input array X is the concatenated embedding vector x of all words in all sentences. ij The resulting matrix.

[0116] Step 3.2: The overall process of the text keyword interaction enhancement module from input to output is as follows: Figure 6 As shown, the input array is processed by an autoencoder, which transforms the high-dimensional text input vector into a higher-level latent array. This latent array captures key information and internal structure of the input data, providing an information-rich and compressed data representation for subsequent processing. The encoder performs cross-attention between a fixed-size latent array and the input array to obtain the compressed representation. Then, a series of self-attention operations are performed between the latent arrays to obtain L... text This is independent of the input size, so it can stack deeper layers using self-attention modules. Finally, the decoder performs cross-attention between the latent and output arrays to produce a shape with the desired size. The calculation of cross-attention is illustrated below:

[0117] Step 3.3: Input the main text into the keyword generation module to obtain a series of keywords {k1,k2,…,k n}, using a word embedding model to embed each keyword k i Mapping to points in a high-dimensional space forms a vector set {kw1,kw2,…,kw}. n The output query array K is a matrix formed by this set of vectors.

[0118] Step 3.4: Decode the latent array L of the main text using the cross-attention module in the decoder. text Specifically, the latent array L of the main text text The output query array K, consisting of the keywords as keys and values, and the query array as the query, interacts with each other through a cross-attention mechanism to extract key information and construct highly relevant feature representations, resulting in the final output array. The formula is as follows:

[0119] Q = W q ·K; K = W k ·L textV = W v ·L text

[0120] Where Q, K, and V represent query, key, and value, respectively, and W q W k W v It is a learnable weight matrix used to transform arrays into a format suitable for cross-attention processing.

[0121]

[0122] Where, d k This is the dimension of the key vector. This formula calculates the semantic relevance between keyword information and the main text, and normalizes it using the softmax function to obtain the contribution weight of each main text feature to the keyword.

[0123] OutputArray=CrossAttention(Q,K,V)=Attention_energy·V

[0124] CrossAttention represents the final interaction feature representation, which is used as the final output array. This allows the model to retain the contextual information of the keywords while focusing on the parts of the text related to the keywords.

[0125] Step 3.5: As Figure 7 As shown, the output array obtained from the text keyword interaction enhancement module in step 3.4 contains the interaction features of text and keywords, and serves as the input array X for the title generation module. This array is an n×d dimensional matrix, where n represents the sequence length and d represents the feature dimension. The input X is first transformed into a form more suitable for the decoding task through an embedding layer. Subsequently, this array is fed into a decoder based on the Transformer architecture (a deep learning architecture specifically designed for processing sequence data), where each layer performs the following operations:

[0126] (1) Handling of multi-head self-attention mechanism:

[0127] H (l) =Attention(Q) (l) ,K (l) V (l) )

[0128] Among them, Q (l) K (l) V (l) These are query, key, and value matrices, which are derived from the output H of the previous layer. (l-1)(or X for the first layer) through different weight matrices These weight matrices are obtained through transformation. They are the parameters in the model, learned through training.

[0129] (2) Feedforward fully connected network:

[0130]

[0131] Among them, W1 ( l ) and W2 ( l ) These are the weights of the linear transformation, while and These are bias terms. These are also parameters learned through training.

[0132] The decoder output H( L This is used to generate titles word by word. The probability of each word generated in each step is given by the following formula:

[0133] p(y t |y1, y2, ..., y t-1 = softmax(H) (L) W out )

[0134] Among them, W out It is a mapping matrix from the output of the last layer of the decoder to the word probabilities. The softmax function is used to transform the output vector of the decoder into a probability distribution.

[0135] During training, the cross-entropy loss function is used to optimize the model. The specific loss calculation formula is as follows:

[0136]

[0137] in, Let be the t-th word in the actual title, and T be the total number of words in the title. This loss function helps the model learn to generate a word sequence that is as close as possible to the real title.

[0138] Through these steps, the title generation module can effectively transform the combined features of text and keywords into precise titles that describe the core theme of an article, achieving the transformation from deep features to human-readable text.

[0139] To further verify the effectiveness of the method of this invention, the abstracts and keywords of two scientific papers were selected to compare the actual generation effects of different methods, as shown in Table 1. The results show that the title generated by the method of this invention has the highest similarity to the original title. Methods that rely solely on the abstract suffer from inaccurate capture of key information, as shown in the first example in Table 1; due to the limited information in the keywords, titles generated by methods relying solely on the abstract cannot cover all the core content of the text, as shown in the second example in Table 1. The title generation results and similarity results of the two examples demonstrate that the method of this invention can more accurately capture and express the core theme of the article, has higher content coverage, and can effectively improve the quality and relevance of title generation.

[0140] Table 1. Actual generation results of different methods

[0141]

[0142] Automatic Abstract and Keyword Generation: The abstract and keyword generation modules automatically extract the core content and key keywords from documents, improving the efficiency and accuracy of information extraction. Abstract generation fully utilizes the document's structural hierarchy, while keyword generation combines user input with automatically generated keywords, optimizing the keyword selection process.

[0143] Application of cross-attention mechanism: The cross-attention mechanism enables deep interaction between keywords and abstracts, enhancing the understanding of the document's core content. By analyzing the dynamic relationship between keywords and abstracts, the generated feature representations more accurately reflect the document's theme and important information, improving the quality and relevance of title generation.

[0144] The embodiments described in this invention are for reference only and do not imply limitation on the scope of protection of this invention. Based on the basic principles of this invention, various variations and modifications can be made to the embodiments of this invention. These variations and modifications, without departing from the spirit of this invention, should also be considered part of the scope of protection of this invention. Furthermore, all embodiments and modifications included in this invention should be considered equivalent implementations under a broad technical concept.

Claims

1. A method for automatically constructing structured document titles, characterized in that, The ability to automatically generate headings for structured documents with long contexts includes the following steps: Step 1, Generate structured document summaries: Train a function to generate structured document summaries. X The abstract generation model; the specific process of step 1 is as follows: Step 1.1: Perform hierarchical coding on the document and calculate the importance weights of elements at the word level and sentence level respectively, so as to effectively extract content elements that affect the generation of the summary; Step 1.2: Construct a neural network model for summarization. The neural network model uses the importance weights of each element at each level obtained in Step 1.1 to perform a weighted summation of the outputs of the word-level, sentence-level, and paragraph-level encoders to obtain the context vector. Based on the context vector and the hidden state, a sequence-to-sequence decoder is used to generate a document summary. Step 1.3: Using the cross-entropy loss function, the summarization generation model is trained through the error backpropagation algorithm to optimize the model parameters, making the generated summaries closer to the real summaries. X This enables the automatic generation of summaries for structured documents. Step 2, Generate keywords for structured documents: First, train a text generation model, then use the text generation model to generate keywords from the summary. X Generate a keyword sequence; the specific process of step 2 is as follows: Step 2.1, train the keyword generation model in the keyword generation module: adopt a method based on bidirectional encoding and autoregressive decoding as the basic model for keyword generation; during training, the input is the summary text generated in Step 1. X The output is the target keyword sequence. Y ; Step 2.2, Generate a preliminary keyword sequence: This involves generating the abstract from Step 1. X Input the pre-trained keyword generation model and use the model's decoder to generate a preliminary keyword sequence. During the generation of the preliminary keyword sequence, predict each keyword step by step until an end symbol is generated or the preset number of keywords is reached. Step 2.3: Use automatically generated keyword sequences to optimize and filter the set of keywords manually entered by the user, and obtain the final filtered keyword set; Step 3, Title Generation: Construct an interactive representation of the abstract and keywords, and generate the final title through this interactive representation. The specific process of Step 3 is as follows: Step 3.1: Convert the keywords generated in Step 2 into vectors using a word embedding model to form a query array. K At the same time, the abstract X After segmenting into sentences, each word is converted into a word vector, and combined with position and type information, a comprehensive embedding representation is obtained. Then, an array of latent representations is generated through an autoencoder. Step 3.2, Analyze the query array K The relationship between the potential representation array and the attention score is calculated, and a feature representation that integrates keywords and summary content is generated based on the attention score, i.e., the output array. Step 3.3: The output array obtained in Step 3.2 is converted into a form suitable for the decoding task through an embedding layer. Subsequently, the embedded array is input into the decoding module based on a predefined structure. The decoding module generates a sequence of titles step by step, and the probability of each word generated in each step is obtained through the corresponding calculation method. During the training process, the cross-entropy loss function is used to optimize the model parameters so that the generated titles are as close as possible to the real titles. Finally, the title generation module outputs the automatically generated structured document titles.

2. The method for automatically constructing structured document titles according to claim 1, characterized in that, The specific implementation steps of step 1.1 are as follows: Step 1.1.1: Convert each word in the structured document into a word vector; Step 1.1.2: Encode the word vector sequence using a bidirectional recurrent neural network to generate sentence vectors; Step 1.1.3: Encode the sentence vector sequence using a bidirectional recurrent neural network to obtain paragraph vectors; Step 1.1.4: Extract the importance weights of elements at the word, sentence, and paragraph levels, and use these weights to update the corresponding sentence and paragraph vectors.

3. The method for automatically constructing structured document titles according to claim 2, characterized in that, In step 1.1.4, at the word level, the similarity between each word vector and its corresponding sentence vector is calculated to determine the importance of each word in the sentence; at the sentence level, the similarity between each sentence vector and its corresponding paragraph vector is further calculated to determine the relative importance of the sentence in the paragraph. The specific process of step 1.1.4 is as follows: The weight of each word is multiplied by its corresponding word vector to obtain a weighted sentence vector. Then, the weight of each sentence is multiplied by the updated sentence vector to obtain a weighted paragraph vector. The weighted vector representation is used for the calculation of contextual information in subsequent summary generation.

4. The method for automatically constructing structured document titles according to claim 3, characterized in that, The specific process of step 2.1 is as follows: Step 2.1.1: Extract the abstract text. X The vector is converted into an embedding vector, processed by an embedding layer and positional encoding, and then input into the encoder to generate a feature representation. Step 2.1.2: The decoder generates target keywords step by step based on the encoder's output and the generated keyword sequence; Step 2.1.3: The model training uses the cross-entropy loss function, including encoder loss and decoder loss, and optimizes the model parameters through the backpropagation algorithm to make the generated keyword sequence match the real keywords.

5. The method for automatically constructing structured document titles according to claim 4, characterized in that, In step 2.3, keywords are automatically generated from the abstract using a text generation model, and a weighting mechanism is used to filter manually input keywords; the specific implementation steps are as follows: Step 2.3.1, User Keyword Input: The module receives a set of keywords related to the abstract that are manually entered by the user; Step 2.3.2, Automatic Keyword Generation: Utilizing a text generation model to generate keywords from the abstract. X Automatically generate keyword sequences; Step 2.3.3, Similarity Calculation and Weight Allocation: Calculate the similarity between each automatically generated keyword and each keyword input by the user to obtain the weight of each automatically generated keyword; Step 2.3.4, Keyword Optimization: Based on weight, select the keywords with the highest similarity. k These keywords serve as input for subsequent modules.

6. The method for automatically constructing structured document titles according to claim 5, characterized in that, In steps 3.1 and 3.2, features that integrate keyword features and document content are generated in the text keyword interaction enhancement module; the specific implementation steps are as follows: Step 3.1.1: Input the keyword set obtained in Step 2 into the text keyword interaction enhancement module. Use the word embedding model to map each keyword to a vector in a high-dimensional space, forming a vector set, and output the query array. K A matrix representation of a set of vectors; Step 3.1.2, extract the abstract X The text is segmented into sentences, and each word is converted into a word vector. Each word vector is combined with positional embedding and type embedding to obtain a comprehensive embedding representation, which enhances the accuracy of the representation of sequence and type information. Latent representation generation: Through an autoencoder, the comprehensive embedding representation is converted into a latent representation array to accurately capture and compress key information in the text. Step 3.2.1, Construction of Interactive Feature Representation: Analyze the query arrays obtained in Steps 3.1.1 and 3.1.

2. K The dynamic relationship between the arrays and the potential representations is calculated by computed queries, keys, and values ​​for both; attention scores are then calculated using the queries, keys, and values. Step 3.2.2, Feature Representation Construction: Combine the attention score with the value vector of the latent array of the main text. V Multiplying these together yields a feature representation that integrates keyword features and the core content of the abstract, i.e., an output array, which is used for subsequent title generation.

Images