A sentence-level question generation method based on syntax-aware prompt learning

Abstract

This invention discloses a sentence-level question generation method based on syntactic-aware prompt learning. First, a bidirectional syntactic dependency graph is constructed based on a given sentence. Its semantic representation is obtained through a relation-aware attention graph encoder. The encoded vectors are then input into a softmax layer, and the top k vectors are selected as continuous prompts based on probability. The prompts are concatenated to the given source text and the answer using prefix adjustment, and then input into a BERT model for encoding. The encoded result is then fed into a Transformer model for decoding. At each time step of decoding, the syntactic dependency information of the generated text sequence is modeled. This information, combined with the syntactic dependency information of the source sentence, determines the parts that the decoder needs to focus on, assisting in the generation of the current word. Simultaneously, a copying mechanism is introduced to handle situations where the generated word is not in the question vocabulary, allowing the model to directly copy words from the source text.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a sentence-level question generation method based on syntactic awareness prompting learning. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence, natural language processing (NLP) technology has been increasingly widely applied. Natural language here refers to the language used for communication in daily life, such as Chinese and English. NLP refers to the use of a series of technical methods to enable computers to understand and generate natural language; therefore, NLP is divided into two main parts: natural language understanding and natural language generation. Question generation (QG) is a typical task in natural language generation. Question generation refers to automatically generating relevant and fluent natural language questions from a series of data sources (such as text, images, and knowledge bases).

[0003] Current research on question generation primarily focuses on reading comprehension, using factual text as input to generate a natural language question in response to a given answer. This invention, however, investigates a question generation task that uses factual text and the answer as input. The application prospects of question generation tasks are vast. For example, in human-computer interaction, chatbots (Siri, Microsoft Xiaoice, etc.) can engage in dialogue with users or request feedback by asking questions; appropriately posed questions can enhance the user experience. In education, generating targeted questions based on course materials allows for testing students' levels, understanding their knowledge mastery, promoting self-assessment, and reducing teaching pressure. Furthermore, as a dual task of automatic question answering, the QG task can generate a large number of high-quality questions, providing a large dataset for training question-answering models and thus improving their performance. Therefore, research on question generation tasks has significant practical implications and broad application prospects.

[0004] Traditional QG (Quick Question Generation) methods primarily utilize manually constructed rule templates combined with manual annotation, a method that is extremely resource-intensive and results in template-generated questions lacking naturalness and diversity. With the development of deep learning technology, inspired by the application of Sequence-to-Sequence (Seq2seq) models in text generation tasks such as machine translation, research on QG methods incorporating deep learning has garnered significant attention. Using end-to-end deep neural network models can effectively improve the naturalness and diversity of generated questions and achieve better generation results. In the past two years, the emergence of pre-trained language models based on Transformer structures, such as BERT and ERNIE, has had a profound impact on question generation research. These pre-trained language models, trained on large-scale corpora, can accurately understand natural language and express it fluently in natural language form, far outperforming traditional encoder-decoder frameworks in question generation tasks. However, while pre-trained models offer good results for question generation, a significant challenge remains: they cannot effectively utilize the syntactic structure information of the source text. Syntactic information refers to the information obtained after dependency parsing the source sentence, including various syntactic structural relations between entities, such as the compound word "compound," the noun subject "nsubj," and the conjunction "conj." This information helps the encoder incorporate the sentence's dependency relations, generating a more effective source text vector representation. However, since these relations are predefined symbols and exist in a different semantic space than the words already trained on the pre-trained model, directly encoding them using the trained language model will create a semantic gap. Therefore, effectively integrating this type of syntactic structural information into the pre-trained model is a significant challenge.

[0005] Furthermore, existing methods primarily consider only the syntactic structure of the input text, neglecting the syntactic structure of already generated sentences. During decoding, text generated at previous time steps is treated merely as a sequence. However, the syntactic structure of already generated text provides crucial clues for the words to be generated in the next time step. Because sentences often contain fixed collocations, understanding the existing syntactic structure allows for greater attention to relevant words in the source text and the direct copying of corresponding collocations.

[0006] Therefore, this invention proposes a novel question generation model that leverages the characteristics of continuous Prompt learning to integrate syntactic dependency information into a pre-trained model. Furthermore, during the decoding process, the syntactic dependency information of the generated text is modeled and used together with the syntactic dependency information of the source text to guide question generation. Summary of the Invention

[0007] Technical Problem: The technical problem to be solved by this invention is that existing methods have difficulty integrating syntactic information into pre-trained language models and ignore the syntactic information of the generated text sequences. This invention provides a sentence-level question generation method based on syntactic awareness prompting learning.

[0008] Technical Solution: The technical solution adopted by this invention to solve its technical problem is a sentence-level question generation method based on syntactic awareness prompt learning. This method first constructs a corresponding bidirectional syntactic dependency graph based on a given sentence, obtains its semantic representation through a relation-aware attention graph encoder, and inputs the encoded vectors into a softmax layer. The first k vectors are selected as continuous prompts based on probability. The prompts are concatenated to the given source text and answer using prefix adjustment, and then input together into a BERT model for encoding. The encoded result is then fed into a Transformer model for decoding. At each time step of decoding, the syntactic dependency information of the generated text sequence is modeled. This information, combined with the syntactic dependency information of the source sentence, determines the parts that the decoder needs to focus on, assisting in the generation of the current word. Simultaneously, a copying mechanism is introduced to address the situation where the generated word is not in the question vocabulary, allowing the model to directly copy words from the source text.

[0009] The sentence-level question generation method based on syntactic awareness prompting learning of the present invention includes the following steps:

[0010] 1) The given sentence sequence is initially processed by the iterative reinforcement-based referential resolution method and the generative path dependency parsing method. First, a dependency parsing tree is constructed, and then the unidirectional connection is extended into a bidirectional connection to construct a bidirectional syntactic dependency graph.

[0011] 2) For the bidirectional syntactic dependency graph finally obtained in step 1), the syntactic dependency relations between nodes in the graph are first encoded by a relation-aware attention graph encoder. Then, the vector representation of the nodes in the graph is updated based on the syntactic dependency relation vector. Finally, the vector representation of the entire graph is obtained.

[0012] 3) Input the syntactic dependency graph vector obtained in step 2) into the softmax layer, calculate the probability, and extract the top k vectors with the highest probabilities as the prompt vectors for prefix adjustment. Add the prompt vectors as prefixes to the inputs of the encoder and decoder respectively.

[0013] 4) Using BERT combined with the prompt prefix, the given text and answer are encoded to obtain a syntax-aware text vector representation.

[0014] 5) The syntactic-aware text vector representation is fed into the Transformer model for decoding. At each time step of decoding, the syntactic dependency information of the generated text sequence is modeled. This information, together with the syntactic dependency information of the source sentence, determines the parts that the decoder needs to focus on, thus assisting in the generation of the current word.

[0015] 6) A copying mechanism is introduced to address the issue of generated words not being in the question dictionary. This allows the decoder to decide at each decoding time step whether to generate words from the question dictionary or directly copy words from the source text.

[0016] Ultimately, this generates a natural language problem.

[0017] As a further improvement of the present invention, in step 1), for a given sentence sequence, pronouns are first replaced with their referents using an iterative reinforcement-based referential resolution technique, facilitating the fusion of identical entities during subsequent graph construction. In the iterative reinforcement-based referential resolution technique, the present invention first constructs a graph from the sentence sequence, using words in the sentence as graph nodes and relationships between words as edges. Then, coreference links are modeled in the graph structure and iteratively refined based on previous predictions to obtain global dependencies and perform referential resolution.

[0018] As a further improvement of the present invention, in step 1), a generative path dependency parsing method is used to perform dependency parsing on the sentence to generate a syntactic dependency tree. Dependency parsing refers to inputting a sentence X = [x1, ..., x...]. N Output dependency pairs P = [p1, ..., p] N ],in Represents the word x i The dependency relationship, and Each refers to the word x i The j-th head word and their relationships. Based on a generative path dependency parsing method, the dependency structure is converted into a serialized representation conforming to the output format of a pre-trained language model through a serializer. Then, the pre-trained language model is used to parse the dependency relationships of the sentence through sequence generation. Specifically, by using the dependency word x... i Distribute the dependencies among each of its central terms. Decompose into smaller dependent units, thus forming a set of triples. Then, each relationship Use a special mark Replace, where It is a special set of markers for all different relationships. The central word... Replace with its position in the input sentence Finally, all dependent units are concatenated with the segmentation marker [SPT] to obtain the target serialization representation.

[0019] As a further improvement of the present invention, in step 1), when performing dependency analysis on the sentence based on the generative path dependency analysis method to generate a syntactic dependency tree, the present invention, in order to disambiguate, uses the core word... Replace with its position in the input sentence While representing the head word by its position is an important method for head word disambiguation, pre-trained language models are not very good at numerical reasoning and struggle to learn the positional information of each word from scratch. Therefore, this invention injects word positional information into the input sentence to avoid numerical reasoning. Specifically, a positional cue (PP) is injected into each word, transforming the positional encoding problem into generating positional numbers in the input, rather than counting each word. The positional cue is the positional number of each word, containing two special markers: [PID] and [SPT]. [PID] marks the beginning of the positional number and prevents the tokenization algorithm from incorrectly treating the positional cue as part of the previous word. [SPT] separates the positional number from the next word. In this way, the input sentence sequence is represented as (x1[PID]1[SPT]x2[PID]2[SPT]…).

[0020] As a further improvement of the present invention, in step 1), when performing dependency analysis on the sentence based on the generative path dependency analysis method to generate a syntactic dependency tree, the first two steps introduce special tags outside the original vocabulary, including, for example, separation tags [PID] and [SPT]. Before training, these tags are added to the vocabulary, and their corresponding embeddings are randomly initialized from the same distribution as other tags. Since different tags have different semantics, the present invention uses a pre-trained language model to treat these special tags as trainable variables and learns their semantics during training. Specifically, the input sentence is first converted into a sequence with positional cues, which is further input into the pre-trained language model to obtain the sequence output with the highest probability. Finally, the predicted dependency relations are recovered by a deserializer. The deserializer replaces the special tags with the original relations. And use the position in the input sentence The index header is used to recover dependencies from the serialized representation.

[0021] As a further improvement of the present invention, in step 1), in the syntactic dependency tree generated by dependency analysis of the sentence, the nodes are each word in the sentence, and the edges are the syntactic relationships between different words, such as the compound word "compound", the noun subject "nsubj", the conjunction "conj", etc., which are all labels for different edges. However, in this tree structure, only words that are directly related in the sentence are connected, while many words are not directly related. In order to explore the syntactic relationships between two words in the sentence more deeply, this patent proposes to add reverse connections to the syntactic dependency tree to change the unidirectional connection into a bidirectional connection, expand the syntactic dependency tree into a syntactic dependency graph, and establish fully connected communication by expanding the topology of the syntactic dependency tree.

[0022] As a further improvement of the present invention, in step 2), when encoding the syntactic dependency relations between nodes in the graph, the relation between two nodes is usually the shortest relation path between them. Therefore, the present invention treats syntactic relations as relation sequences and uses a bidirectional LSTM (Long Short Term Memory) network to encode these relation sequences.

[0023] As a further improvement of the present invention, in step 2), when encoding a node in the graph, the node's vector representation is calculated and updated by fusing semantic information from its neighboring nodes and semantic information from the syntactic relationships between them. Since the syntactic relationship encoding only encodes the shortest path between two nodes and does not distinguish the direction of the edges, this invention uses a parameter matrix to divide the syntactic relationship encoding vector into a forward syntactic relationship encoding vector and a backward syntactic relationship encoding vector to incorporate edge direction information into the node's vector representation. Because neighboring nodes contribute differently to the node's vector representation, this invention proposes using a syntactic-aware graph attention mechanism to calculate the attention score of different nodes in order to incorporate as much information as possible from neighboring nodes with high relevance to the node. After obtaining the syntactic-aware attention, the information from neighboring nodes is combined with the node's own semantic information to update the node's vector representation.

[0024] As a further improvement of the present invention, in step 4), BERT is used to encode the given text and answer. The input of the text encoder is {Prefix; [CLS]sentence[SEP]answer[SEP]}, where Prefix represents the prompt prefix vector. Prefix adjustment first initializes a trainable matrix to store the prefix parameters. After prefix adjustment, the parameters of the pre-trained language model are frozen, and only the prefix parameters are trained, achieving lightweight fine-tuning. During the optimization process, directly updating the trainable parameters will lead to instability in the training process and a slight decrease in performance. Therefore, a smaller matrix based on a large feedforward neural network is used to reparameterize the matrix. After training, only the prefix parameters are retained, the reprocessed parameters are deleted, and the encoding result of the classification character — [CLS] is used as the encoding result of the text, which contains the representation information of the fused text and answer.

[0025] As a further improvement of the present invention, in step 5), given that the syntactic structure information of the generated text plays an important role in indicating the word to be generated in the next time step, the present invention proposes to model the syntactic information of the generated text sequence. Specifically, firstly, the syntactic dependency graph of the generated sentence sequence and its corresponding vector representation are obtained using the syntactic dependency graph construction and encoding methods in steps 1) and 2). Then, it is matched with the vector representation of the source text and the vector representation of the syntactic structure graph of the source text to guide the generation of the current word.

[0026] As a further improvement of this invention, in step 5), to prevent the decoder from repeatedly paying attention to a certain node in the source syntactic dependency graph during word generation, leading to word repetition, this invention introduces a covering mechanism. However, traditional covering mechanisms calculate the coverage degree for each word in the text sequence, treating the input sequence as a vector set and ignoring the influence of word order, let alone syntactic dependency relations. Therefore, this invention improves upon this by proposing a syntactic-aware covering mechanism. Given that the syntactic dependency graph is evolved from a dependency parsing tree, this invention proposes to additionally consider the covering vectors of its left and right child nodes when calculating the covering vector of the current node, thereby integrating syntactic dependency relations into the calculation of the covering vector. Then, the covering vector is used to constrain the attention at the current time step, alleviating the phenomenon of repeated attention.

[0027] Beneficial effects:

[0028] Compared to existing technologies, this invention has the following advantages: 1) This invention constructs a syntactic dependency graph for a given sentence sequence. Existing question generation methods only utilize dependency parsing trees to obtain syntactic information, while this invention adds reverse connections to the dependency parsing tree, transforming unidirectional connections into bidirectional connections, expanding the syntactic dependency tree into a syntactic dependency graph. This method establishes fully connected communication, which is beneficial for obtaining richer syntactic information. 2) This invention uses a relation-aware attention graph encoder to obtain the semantic representation vector of the syntactic dependency graph, separately encoding syntactic dependency relations and integrating them into the node vector representation, thereby obtaining a semantically richer syntactic dependency graph representation vector. 3) This invention uses continuous prompt learning to model syntactic information. Continuous prompt learning focuses on vectorized expression, does not require specific text, and can learn directly to act on the model's embedding space. Therefore, using prompt learning can better model syntactic dependency relations and effectively solve the semantic gap problem. Furthermore, most current continuous prompt learning methods are randomly initialized and static, while this invention adopts a dynamic, syntactic-aware approach, using syntactic information for prompt selection. 4) This invention models the syntactic dependency information of the generated text sequence during the decoding process and guides the generation of the current word. Existing works generally only consider the syntactic structure information of the input text, ignoring the syntactic structure information of the already generated sentences, treating the text generated in previous time steps merely as a sequence during decoding. However, the syntactic information of the already generated text plays an important role in prompting the word to be generated in the next time step. Because sentences usually contain some fixed collocations, by mastering the existing syntactic dependency information, more attention can be paid to relevant words in the source text, and the corresponding collocation words can be directly copied. 5) This invention designs a syntactic-aware coverage mechanism. To prevent the decoder from repeatedly paying attention to a certain node in the source syntactic structure graph during word generation, leading to word repetition, this invention plans to introduce a coverage mechanism. However, traditional coverage mechanisms calculate the coverage degree for each word in the text sequence, treating the input sequence as a vector set, ignoring the influence of word order, let alone syntactic structure. Therefore, this invention improves upon this by proposing a syntactic-aware coverage mechanism. Since syntactic dependency graphs are derived from dependency parsing trees, this invention proposes to consider the coverage vectors of the left and right child nodes when calculating the coverage vector of the current node, thereby incorporating syntactic structure information into the calculation of the coverage vector. Experimental analysis demonstrates that the sentence-level question generation method based on syntactic awareness prompting learning proposed in this paper improves the quality of generated questions and enhances the effectiveness of question generation. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the basic process of the present invention;

[0030] Figure 2 This is a model framework diagram of the present invention;

[0031] Figure 3 This is the difference between the syntactic dependency graph and the syntactic dependency tree in this invention;

[0032] Figure 4 This is a diagram of the encoder-decoder architecture using continuous cue learning according to the present invention;

[0033] Figure 5 This is a diagram illustrating the model decoding implementation of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the embodiments and the accompanying drawings.

[0035] Example 1: The sentence-level question generation method based on syntactic awareness prompting learning of the present invention includes the following steps:

[0036] 1) A bidirectional syntactic dependency graph is constructed by initially processing a given sentence sequence using an iterative reinforcement-based referential resolution method and a generative path dependency parsing method. Specifically, for a given sentence sequence, the iterative reinforcement-based referential resolution method is first used to replace pronouns with their referents, facilitating the fusion of identical entities during subsequent graph construction. Then, the generative path dependency parsing method is used to perform dependency analysis on the sentences, generating a syntactic dependency tree.

[0037] In the iterative reinforcement-based coreference resolution method, this invention first constructs a graph from the sentence sequence, using words in the sentence as graph nodes and relationships between words as edges. Then, coreference links are modeled within the graph structure and iteratively refined based on previous predictions to obtain global dependencies for coreference resolution. Specifically, given a sentence sequence X = [x1,…,x…],… ... N The constructed coreference graph is the relationship matrix G between the words. Among them, two words x... i and x j The relationship between g i,j ∈{0,1,2}, where 0 indicates no relationship, 1 indicates entity link, and 2 indicates core reference link.

[0038] Entity links are used to identify entities in a sentence sequence. This invention defines entity links in two different ways, depending on whether the graph is the input or output of the model. When the graph is the input, there is a directed link from each entity's label to the beginning of the entity (referred to as the entity head), including the head to itself. When the graph is the output of the model, there is only one directed link from the last label of the entity span to the first label. Both encoding methods uniquely define an entity span. Each entity has a unique set of origins and a unique head.

[0039] Coreference links define the relationships between an entity and each of its predecessors. This invention defines coreference links in two different ways, depending on whether the graph is an input or output of the model. When the graph is the input, there are links from the entity header tag to every entity header in the same cluster. When the graph is the output of the model, an entity should be associated with at least one of its predecessors. If an entity has no predecessor, or corresponds to an entity mentioned for the first time in the text, then it is connected to an empty predecessor. This invention uses all possible connections between entities in an entity cluster as input so that the model receives direct input for each coreference relationship.

[0040] The ultimate goal is to learn the conditional probability distribution p(G|X) of the relation matrix over a given sequence of sentences, and to achieve coreference resolution based on this probability distribution. This distribution is initially derived by assuming that each relation g i,j The independence between them is used as an approximation, and the calculation formula is as follows:

[0041]

[0042] probability p(g) i,j The calculation of |D) is divided into two cases: one is for entity link p m Another type is for core-referenced links p c Entity link p m The probability is calculated as follows:

[0043] p m (g i,j =1|X)=σ(W m ·[h i ,h j ])

[0044] Among them, W m It is a parameter matrix, h i and h j It is the word x i and x j The latent vector. This probability indicates whether there exists an entity in the sentence sequence X that starts at position j and ends at position i.

[0045] Core link p c The probability is calculated as follows:

[0046]

[0047] Among them, W c It is a parameter matrix. A(i) is the word x. i The set of all candidate antecedents includes all entity headers with intervals less than i, and also includes empty entity headers ∈.

[0048] That is, A(i) = {∈,x} k |k k ∈H(D)}, where H(D) is the set of all candidate entity heads in the sentence sequence.

[0049] Then, iterative coreference modeling is used to model the intrinsic dependencies between relationships. In the t-th iteration, based on the coreference graph G obtained in the previous iteration... t-1 Predicting the co-pointing graph G in this round t The conditional probability distribution of the model is defined as follows:

[0050]

[0051] In generative path dependency parsing, the dependency structure is first converted into a serialized representation conforming to the output format of a pre-trained language model. Then, word position information is injected into the input sentence to avoid numerical reasoning. Finally, the sequence with positional cues is input into the pre-trained language model to obtain the output sequence with maximum probability. The predicted dependency relationship is then recovered through a deserializer. The first step involves converting the dependency structure into a serialized representation conforming to the output format of the pre-trained language model. Specifically, this involves converting the dependency structure into a serialized representation conforming to the output format of the pre-trained language model by deserializing the dependency word x. i Distribute the dependencies among each of its central terms. Decompose into smaller dependent units, thus forming a set of triples. Then, each relationship Use a special mark Replace, where It is a special set of markers for all different relationships. The central word... Replace with its position in the input sentence Finally, all dependent units are concatenated with the segmentation marker [SPT] to obtain the target serialization representation.

[0052] The second step involves injecting word position information into the input sentence to avoid performing numerical reasoning. In the first step, this invention, for disambiguation purposes, injects the central word... Replace with its position in the input sentence ​While representing the head word by its position is an important approach to head word disambiguation, pre-trained language models are not well-suited for numerical reasoning and struggle to learn the positional information of each word from scratch. Therefore, this invention injects a positional cue (PP) into each word, transforming the positional encoding problem into generating positional numbers in the input, rather than counting each word. The positional cue is the positional number of each word, containing two special markers: [PID] and [SPT]. [PID] marks the beginning of the positional number and prevents the tokenization algorithm from incorrectly treating the positional cue as part of the previous word. [SPT] separates the positional number from the next word. In this way, the input sentence sequence is represented as (x1[PID]1[SPT]x2[PID]2[SPT]…).

[0053] Finally, sequences are generated using a pre-trained language model. The first two steps introduce special tags outside the original vocabulary, including, for example, separation tags [PID] and [SPT]. These tags are added to the vocabulary before training, and their embeddings are randomly initialized from the same distribution as the other tags. Since different tags have different semantics, this invention uses a pre-trained language model to treat these special tags as trainable variables and learns their semantics during training. Specifically, the input sentence is first converted into a sequence with positional cues, which is then further input into the pre-trained language model to obtain the sequence output with maximum probability. Finally, the predicted dependencies are recovered using a deserializer. The deserializer replaces the special tags with the original relations. And use the position in the input sentence The index header is used to recover dependencies from the serialized representation and obtain the syntactic dependency tree.

[0054] In a syntactic dependency tree, nodes represent each token in a sentence, and edges represent the syntactic relationships between different tokens. For example, the words "compound," "nsubj" (noun subject), and "conj" (conjunction) are all labels for different edges. However, in this tree structure, only directly related words in the sentence are connected, while many words have no direct connection. To more deeply explore the syntactic relationships between two words in a sentence, this patent proposes adding reverse connections to the syntactic dependency tree, transforming unidirectional connections into bidirectional connections, and expanding the syntactic dependency tree into a syntactic dependency graph. This expansion of the syntactic dependency tree's topology establishes fully connected communication. Figure 3As shown, for the sentence "How many points did the Broncos score?", the left side of the graph constructs the corresponding syntactic dependency tree. By adding a back connection to the syntactic dependency tree, a one-way connection is transformed into a two-way connection, forming the syntactic dependency graph on the right side of the graph. Through two-way connections, a word can directly receive and send information to any other word, regardless of whether they are directly connected.

[0055] 2) For the bidirectional syntactic dependency graph finally obtained in step 1), the syntactic dependency relations between nodes in the graph are first encoded by a relation-aware attention graph encoder. Then, the vector representation of the nodes in the graph is updated based on the syntactic dependency relation vector. Finally, the vector representation of the entire graph is obtained.

[0056] When encoding syntactic dependency relations, the relationship between two nodes is usually the shortest path between them. For example, the shortest path sp between node i and node j. i→j =[sp1,…,sp t ,…,sp b+1 ]=[e(i,k1),e(k1,k2),…,e(i,k1)], where, sp t Let represent the t-th edge in the relation path, which consists of b+1 edges in total. Let e(i,k1) represent the label of the edge connecting node i and node k1. 1:b The shortest relation path sp i→j The intermediate nodes on the graph. The relationship between two nodes is usually the shortest path between them. Therefore, this invention treats syntactic relations as relation sequences and uses a bidirectional LSTM (Long Short Term Memory) network to encode these relation sequences. The calculation formula is as follows:

[0057]

[0058]

[0059]

[0060] in, Let r represent the hidden state at time step t-1 in the feedforward LSTM network. ij It is the final relation encoding vector, which is formed by concatenating the last hidden states of the forward LSTM network and the backward LSTM network, representing the syntactic relation between node i and node j.

[0061] For a given node in the graph, its vector representation is calculated and updated by fusing the semantic information of its neighboring nodes and the semantic information of the syntactic relations between them. Since the syntactic relation encoding only encodes the shortest path between two nodes and does not distinguish the direction of the edges, this invention first uses a parameter matrix to encode the syntactic relation encoding vector r... ij Divided into forward syntactic relation encoding vectors r i→j and backward syntactic relation encoding vector r j→i That is, [r] i→j ,r j→i ] = W r r ij .

[0062] Furthermore, the degree of association between a given node and its neighboring nodes varies. Some nodes are closely related, for example, if there is a compound word relationship between the two nodes, thus their contributions to the node's vector representation also differ. To incorporate as much information as possible from neighboring nodes with high association as possible, this invention proposes a syntax-aware graph attention mechanism to calculate the attention score for different nodes. Specifically, for nodes i and j, the attention α between them... ij It is based on their respective initial vectors h i h j and the syntactic relation vector r between them ij The calculation method is as follows:

[0063]

[0064] This computational method leverages syntactic relations to constrain the vector representations of nodes, thereby improving the accuracy and semantic richness of the vector representations. After obtaining syntactic-aware attention, information from neighboring nodes is combined with the node's own semantic information to update the vector representation of node i. The computation method is as follows:

[0065]

[0066] Here, G(i) represents the set of neighboring nodes of node i. Finally, the vector representation h of all nodes in the graph is used. 1:m Inputting into the average pooling layer yields the semantic vector representation g of the entire syntactic dependency graph, calculated as follows:

[0067] as follows:

[0068] g = AvgPooling(h 1:m )

[0069] Here, AvgPooling(·) represents the average pooling function.

[0070] 3) Input the syntactic dependency graph vector finally obtained in step 2) into the softmax layer, calculate the probability, and take out the first k vectors with high probability as the prompt vector for prefix adjustment.

[0071] 4) Using BERT combined with the prompt prefix, the given text and answer are encoded to obtain a syntax-aware text vector representation. For example... Figure 4 As shown, the input to the BERT pre-trained language model is {Prefix; [CLS]sentence[SEP]answer[SEP]}. In this model, there are a total of k prefixes; in the diagram, we assume k = 5. Prefix adjustment first initializes a trainable matrix P. θ This is used to store the prefix parameters. After prefix adjustment, the parameters of the pre-trained language model are frozen, and only the prefix parameters θ are trained, achieving lightweight fine-tuning. During optimization, the trainable parameters P are directly updated. θ This can lead to instability in the training process and a slight decrease in performance, therefore a large feedforward neural network-based MLP is used. θ The smaller matrix P formed θ For matrix P θ To achieve reparameterization, i.e., P θ =MLP θ (P θ After training, only the prefix parameter P is retained. θ Delete reprocessing parameter P θ The encoding result of the classification character [CLS] is used as the encoding result of the text, represented by vector c, which contains the representation information of the fused text and answer.

[0072] 5) The syntax-aware text vector representation is fed into the Transformer model for decoding. At each time step of decoding, the syntax dependency graph of the generated sentence sequence and its corresponding vector representation q are obtained using the syntax dependency graph construction and encoding methods in steps 1) and 2). t The modeling process involves defining the syntactic dependency information of the generated text sequence. Figure 5 As shown, since different information in the source text needs to be focused on when generating different words, an attention mechanism is used to extract key information from the syntactic structure graph of the given source text. At the t-th time step of decoding, the attention-aware source text syntactic dependency graph vector representation g... t c) Vector representation of the source text; q) Vector representation of the syntactic dependency graph of the generated sentence sequence. t The hidden state s of the previous time step t-1 Calculate the hidden state s at the current time step. t And generate the current word y tThe calculation method is as follows:

[0073] s t =Transformer(FFN(c,g) t ,q t ),s t-1 )

[0074] P vocab (y t ) = softmax(s t ,y t-1 )

[0075] Here, FFN(·) represents a fully connected feedforward network layer.

[0076] To prevent the decoder from repeatedly focusing on a single node in the source syntactic dependency graph during word generation, leading to word repetition, this invention introduces a covering mechanism. However, traditional covering mechanisms calculate the coverage degree for each word in the text sequence, treating the input sequence as a vector set and ignoring the influence of word order, let alone syntactic structure. Therefore, this invention improves upon this by proposing a syntactic-aware covering mechanism. Given that the syntactic dependency graph evolves from a dependency parsing tree, this invention proposes to additionally consider the covering vectors of its left and right child nodes when calculating the covering vector of the current node, thereby incorporating syntactic information into the calculation of the covering vector. The calculation process is as follows:

[0077] Cov t,j

[0078] =GRU(Cov) t-1,j ,a t-1,j ,s t-1 ,h j ,Cov t-1,L(i) ,a t-1,L(i) ,Cov t-1,R(i) ,a t-1,R(i) )

[0079] Among them, a t-1,j This is the attention weight of the j-th node in the syntactic structure graph of the source text at time step t-1. As can be seen from this formula, when calculating the coverage vector, this invention considers not only the attention value of the current node, but also the attention values ​​and coverage degree of its left child node L(i) and right child node R(i).

[0080] Then, the attention at the current time step is constrained by the covering vector, and the hidden vectors of all nodes in the graph are calculated. The weighted average value is used to obtain the attention-aware source text syntactic structure graph vector representation g. t ,specific

[0081] The calculation process is as follows:

[0082]

[0083]

[0084] e t，j =λ(s) t-1 h j Cov t，j )

[0085] Here, λ is an activation function, usually implemented using tanh.

[0086] 6) A copying mechanism is introduced to address the issue of generated words not being in the question dictionary. This allows the decoder to decide at each decoding time step whether to generate words from the question dictionary or directly copy words from the source text, ultimately generating a natural language question. This is because the attention weight 'a' is calculated during the generation process. t =(a t,1 ,a t,2 ,…,a t,N This measures the correlation between each input word and a partial decoding state at the current time step, which is the information weight distribution of the input sequence. Therefore, this weight vector a can be directly used. t Consider the probability distribution P of copying words from the input sequence copy (y t ).

[0087] The final output y at the current time step t It can be generated either from the question dictionary or copied from the source text, and the calculation formula is as follows:

[0088] P(y t ) = p gen ·P vocab (y t )+(1-p gen )·P copy (y t )

[0089] p gen =σ(W gen (s t +c+g t +q t +y t-1 ))

[0090] Where, p gen The probability of generating a value from the question dictionary is s, which is the hidden state s of the decoder at the current time step. t c) Vector representation of the source text; g) Vector representation of the source text's syntactic structure graph. tAnd the syntactic structure graph vector representation q of the generated text sequence t Calculated. W gen σ is a trainable parameter matrix, and σ is the sigmoid function.

[0091] The above embodiments are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and equivalent substitutions without departing from the principle of the present invention. All such improvements and equivalent substitutions to the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A sentence-level question generation method based on syntactic awareness prompting learning, characterized in that, The method includes the following steps: 1) The given sentence sequence is initially processed by the iterative reinforcement-based referential resolution method and the generative path dependency parsing method. First, a dependency parsing tree is constructed, and then the unidirectional connection is extended into a bidirectional connection to construct a bidirectional syntactic dependency graph. 2) For the bidirectional syntactic dependency graph finally obtained in step 1), the syntactic dependency relations between nodes in the graph are first encoded by a relation-aware attention graph encoder. Then, the vector representation of the nodes in the graph is updated based on the syntactic dependency relation vector. Finally, the vector representation of the entire graph is obtained. 3) Input the syntactic dependency graph vector finally obtained in step 2) into the softmax layer, calculate the probability, and take out the first k vectors with high probability as the prompt vectors for prefix adjustment. Add the prompt vectors as prefixes to the input of the encoder and decoder respectively. 4) Using BERT combined with the prompt prefix, the given text and answer are encoded to obtain a syntax-aware text vector representation; 5) The syntactic-aware text vector representation is fed into the Transformer model for decoding. At each time step of decoding, the syntactic dependency information of the generated text sequence is modeled. This information, together with the syntactic dependency information of the source sentence, determines the parts that the decoder needs to focus on, thus assisting in the generation of the current word. 6) Introduce a copying mechanism to address the situation where the generated word is not in the question dictionary. This allows the decoder to decide at each decoding time step whether to generate the word from the question dictionary or directly copy the word from the source text, ultimately generating a natural language question.

2. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 1), for a given sentence sequence, pronouns are first replaced with their referents based on iterative reinforcement of the referent resolution technique, so that the same entities can be merged in the subsequent graph construction. In the iterative reinforcement of the referent resolution technique, the sentence sequence is first constructed into a graph, with words in the sentence as graph nodes and the relationships between words as edges. Then, coreference links are modeled in the graph structure and iteratively refined according to previous predictions to obtain global dependencies and perform referent resolution.

3. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 1), a generative path dependency parsing method is used to perform dependency parsing on the sentence, generating a syntactic dependency tree. Dependency parsing refers to inputting a sentence X = [x1, ..., x...]. N Output dependency pairs P = [p1, ..., p] N ],in Represents the word x i The dependency relationship, and Each refers to the word x i The j-th head word and their relationships are analyzed using a generative path dependency parsing method. A serializer converts the dependency structure into a serialized representation conforming to the output format of a pre-trained language model. Then, the pre-trained language model is used to parse the sentence's dependency relationships through sequence generation. Specifically, this involves analyzing the dependency words x... i Distribute the dependencies among each of its central terms. Decompose into smaller dependent units, thus forming a set of triples. Then, each relationship Use a special mark Replace, where It is a special set of markers for all different relationships, with the central word... Replace with its position in the input sentence Finally, all dependent units are concatenated with the segmentation marker [SPT] to obtain the target serialization representation.

4. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 1), when performing dependency analysis on the sentence based on the generative path dependency analysis method to generate a syntactic dependency tree, in order to disambiguate, the headword is... Replace with its position in the input sentence Injecting positional hints (PP) into each word transforms the positional encoding problem into generating positional numbers in the input. The positional hint is the positional number of each word, which contains two special markers, [PID] and [SPT]. [PID] marks the beginning of the positional number and prevents the tokenization algorithm from mistakenly treating the positional hint as part of the previous word. [SPT] separates the positional number from the next word. In this way, the input sentence sequence is represented as (x1[PID]1[SPT]x2[PID]2[SPT]...).

5. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 1), when performing dependency analysis on the sentence based on the generative path dependency analysis method to generate a syntactic dependency tree, the first two steps introduce special tags outside the original vocabulary, including separation tags [PID] and [SPT]. Before training, these tags are added to the vocabulary, and their corresponding embeddings are randomly initialized from the same distribution as other tags. Since different tags have different semantics, a pre-trained language model is used to treat these special tags as trainable variables, and their semantics are learned during training. Specifically, the input sentence is first converted into a sequence with positional cues, which is then further input into the pre-trained language model to obtain the sequence output with the highest probability. Finally, the predicted dependency relations are recovered through a deserializer, which replaces the special tags with the original relations. And use the position in the input sentence The index header is used to recover dependencies from the serialized representation and obtain the syntactic dependency tree.

6. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 2), when encoding the syntactic dependency relationships between nodes in the graph, the relationship between two nodes is usually the shortest path between them. The syntactic relationship is regarded as a relation sequence, and bidirectional LSTM (Long Short Term Memory) is used to encode these relation sequences.

7. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 2), when encoding a node in the graph, the node's vector representation is calculated and updated by fusing semantic information from its neighboring nodes and semantic information from the syntactic relations between them. Since the syntactic relation encoding only encodes the shortest path between two nodes and does not distinguish the direction of the edge, in order to incorporate the direction information of the edge into the node's vector representation, the syntactic relation encoding vector is divided into a forward syntactic relation encoding vector and a backward syntactic relation encoding vector using a parameter matrix. Since the contribution of neighboring nodes to the node's vector representation is different, in order to incorporate as much information as possible from neighboring nodes with a high degree of association with the node, a syntactic-aware graph attention mechanism is proposed. The attention scores of different nodes are calculated, and after obtaining the syntactic-aware attention, the information from neighboring nodes is combined with its own semantic information to update the node's vector representation.

8. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 4), the prompt is used as a prefix, and BERT is used to encode the given text and answer. Therefore, the input of the text encoder is {Prefix; [CLS]sentence[SEP]answer[SEP]}, where Prefix represents the prompt prefix vector. Prefix adjustment first initializes a trainable matrix to store the prefix parameters. After prefix adjustment, the parameters of the pre-trained language model are frozen, and only the prefix parameters are trained, achieving lightweight fine-tuning. In the optimization process, directly updating the trainable parameters will lead to instability in the training process and a slight decrease in performance. Therefore, a smaller matrix based on a large feedforward neural network is used to reparameterize the matrix. After training, only the prefix parameters are retained, the reprocessed parameters are deleted, and the encoding result of the classification character [CLS] is used as the encoding result of the text, which contains the representation information of the fused text and answer.

9. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 5), given that the syntactic structure information of the generated text plays an important role in prompting the word to be generated in the next time step, it is proposed to model the syntactic information of the generated text sequence. Specifically, the syntactic dependency graph of the generated sentence sequence and its corresponding vector representation are obtained by using the syntactic dependency graph construction and encoding method in steps 1) and 2). Then, it is matched with the vector representation of the source text and the vector representation of the syntactic structure graph of the source text to guide the generation of the current word.

10. The sentence-level question generation method based on syntactic awareness prompting learning according to claim 1, characterized in that, In step 5), when calculating the coverage vector of the current node, the coverage vectors of its left and right child nodes are also considered, thereby incorporating syntactic dependency relations into the calculation of the coverage vector. Then, the coverage vector is used to constrain the attention of the current time step, alleviating the phenomenon of repeated attention.

Images