A subject text entity extraction and keyword generation method and device

Abstract

The application discloses a subject text entity extraction and keyword generation method and device, the method comprising: obtaining a global feature matrix, a local feature matrix and a head-tail enhanced feature matrix according to a plurality of character-level word-level splicing feature vectors of an original text sequence; using a multi-head self-attention mechanism to perform feature fusion on the matrices to obtain a fused feature matrix, and obtaining an existing knowledge point set by decoding the fused feature matrix; the existing knowledge point set comprises entity knowledge points existing in the original text; using BART and SoftMask to generate word embedding vectors containing semantic information of the original text according to the original text sequence, and obtaining a generated knowledge point set based on the word embedding vectors; the generated knowledge point set comprises entity knowledge points that do not exist in the original text and are associated with the original text; and screening according to a word importance score, the existing knowledge point set and the generated knowledge point set to obtain a more accurate and comprehensive key knowledge point set.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a method and device for subject text entity extraction and keyword generation. Background Art

[0002] Knowledge acquisition is a core step in building a disciplinary knowledge graph. It refers to the process of automatically extracting information from multi-source heterogeneous data to obtain knowledge elements such as entities, attributes, and their relationships, thereby forming an ontological representation of knowledge. In the field of education, a large amount of data exists in unstructured form, such as subject textbooks and test questions, which contains a wealth of information and knowledge. Automatically extracting and mining valuable entity knowledge points and relationships from this unstructured text data and storing them in a structured form plays a vital role in building knowledge graphs in the education field, thereby further promoting the development of smart education.

[0003] Extensive research has been conducted on knowledge acquisition from educational texts. Using the textbook "Principles of Computer Organization" as an example, Zhang Huinan et al. employed a relatively mature BiLSTM-CRF (bidirectional long short-term memory network-conditional random field) model combined with manual verification to extract knowledge points, providing knowledge acquisition for the construction of subject knowledge graphs. Using the textbook "Linear Algebra" as an example, Yang Wenxia et al. employed a "pre-training + fine-tuning" strategy to extract entities and relationships, thereby acquiring knowledge for subject knowledge graphs. However, unlike general domain texts, educational texts typically contain a wealth of specialized knowledge and information. Current research still relies on general domain text recognition methods, which presents challenges in identifying and extracting educational texts, such as polysemy and difficulty identifying rare or unknown knowledge points. Furthermore, current research on knowledge acquisition from subject knowledge graphs primarily focuses on identifying entity knowledge points within text content, while limited attention is paid to mining text content. Subject texts, in addition to the various semantic information found in general texts, also contain a wealth of subject domain knowledge and possess strong knowledge relevance. Consequently, implicit knowledge points within subject texts have not been fully and comprehensively extracted and utilized.

[0004] Entity extraction, also known as named entity recognition (NER) in natural language processing, started relatively late in China compared to overseas research. Furthermore, Chinese has complex word structure and lacks spaces between words, making it more challenging than English NER. However, commonly used NER methods both domestically and internationally fall into three categories: rule-based and dictionary-based methods, traditional machine learning-based methods, and deep learning-based methods.

[0005] In the education field, named entity recognition (NER) technology primarily extracts proper nouns from teaching resources such as textbooks and test questions. Based on the entity categories of these proper nouns, these text resources are then sequenced to create corresponding labels. During the screening of text resources based on knowledge points, proper nouns appearing in the text are abstracted into entities, and the text resources are then associated with specific knowledge points for subsequent teaching resource retrieval.

[0006] Currently, mainstream NER technology is based on deep learning. The main idea behind this approach is to use each word in a given text as input to a model. Using a deep neural network, the model learns the representation of each word and the semantic information of the text, and predicts whether each word belongs to a specific entity category. However, text in the educational field is highly specialized and contains a lot of specialized terminology. Text vectorization models based on general domains cannot fully capture the semantic information of educational text. Therefore, deep learning research methods that incorporate the specific characteristics of the educational field have attracted widespread attention.

[0007] The existing methods of text entity extraction in the field of education mainly include three aspects: the first is the entity extraction method based on rules and dictionaries, the second is the entity extraction method based on machine learning models such as support vector machines and hidden Markov models, and the third is the entity extraction method based on deep learning.

[0008] Rule- and dictionary-based entity extraction methods are highly dependent on these rules and dictionaries. Therefore, for educational texts containing a large number of specialized terms, the quality of the constructed rules and dictionaries plays a decisive role in entity recognition. Furthermore, researchers need to spend most of the entire entity recognition process manually compiling key technologies such as rules and dictionaries. Furthermore, the scope of manually compiled rules and dictionaries is limited, and different fields require different rules and domain dictionaries. Therefore, this method has poor generalization and reusability.

[0009] Compared with rule- and dictionary-based entity recognition methods in the education field, the above-mentioned machine learning-based methods can automatically identify entities in texts through feature engineering. However, these methods only focus on flat-structured subject entities, which is inconsistent with the characteristics of nested entities in Chinese educational texts. Researchers are required to spend more than 80% of their time and energy on feature engineering work such as data preprocessing, cleaning, and part-of-speech tagging. In addition, problems such as the difficulty of modeling and sparse data dimensions still exist.

[0010] Deep learning research methods that integrate the specific characteristics of educational disciplines are the mainstream. For example, an improved BiLSTM-CRF model based on domain characteristics is constructed, integrating the root vector features of radicals for chemical entity recognition; and a BERT (Bidirectional Encoder Representations from Transformers)-BiLSTM-CRF model that integrates an attention mechanism for recognizing Chinese poetry.

[0011] However, in educational texts, the vocabulary in the subject area is relatively professional, and the above-mentioned existing entity recognition and generation methods have shortcomings in terms of the accuracy of entity recognition and generation. Summary of the Invention

[0012] In order to solve the above problems existing in the prior art, the present invention provides a method and device for subject text entity extraction and keyword generation based on neural network.

[0013] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0014] In a first aspect, the present invention provides a method for extracting subject text entities and generating keywords, the method comprising:

[0015] Generate an original text sequence according to the original text;

[0016] A global feature matrix, a local feature matrix, and a head-end reinforcement feature matrix are obtained based on multiple character-level and word-level splicing feature vectors of the original text sequence, and a multi-head self-attention mechanism is used to perform feature fusion on the global feature matrix, the local feature matrix, and the head-end reinforcement feature matrix to obtain a fused feature matrix, and a set of existing knowledge points is obtained by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then performing a splicing operation; the set of existing knowledge points includes entity knowledge points existing in the original text;

[0017] Based on the original text sequence, using BART and SoftMask to generate a word embedding vector containing the semantic information of the original text, and obtaining a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text;

[0018] The key knowledge point set is obtained by screening according to the word importance score, the existing knowledge point set and the generated knowledge point set.

[0019] Optionally, the multiple character-level word-level concatenation feature vectors are obtained by:

[0020] Extracting multiple word-level feature vectors from the original text sequence using BERT;

[0021] Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0022] Multiple word-level concatenated feature vectors are obtained by concatenating word-level feature vectors and character-level feature vectors.

[0023] Optionally, obtaining a set of existing knowledge points by decoding the fusion feature matrix includes:

[0024] The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

[0025] Optionally, a global feature matrix, a local feature matrix, and a head-to-tail reinforcement feature matrix are obtained based on multiple character-level and word-level concatenated feature vectors of the original text sequence, including:

[0026] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level splicing feature vectors;

[0027] Obtain a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level splicing feature vectors;

[0028] A local feature matrix is ​​obtained according to the multiple character-level word-level splicing feature vectors by introducing a convolution module with a self-attention mechanism.

[0029] Optionally, the BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

[0030] In a second aspect, the present invention provides a device for extracting subject text entities and generating keywords, the device comprising:

[0031] An original text sequence generation module, used to generate an original text sequence according to the original text;

[0032] An existing knowledge point set determination module is configured to obtain a global feature matrix, a local feature matrix, and a head-end reinforcement feature matrix based on multiple character-level and word-level splicing feature vectors of the original text sequence, perform feature fusion on the global feature matrix, the local feature matrix, and the head-end reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and obtain an existing knowledge point set by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing a splicing operation after performing character-level feature extraction and word-level feature extraction on the original text sequence; and the existing knowledge point set includes entity knowledge points existing in the original text;

[0033] A generated knowledge point set determination module is configured to generate a word embedding vector containing semantic information of the original text using BART and SoftMask based on the original text sequence, and obtain a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text;

[0034] The key knowledge point set screening module is used to screen the key knowledge point set according to the word importance score, the existing knowledge point set and the generated knowledge point set to obtain the key knowledge point set.

[0035] Optionally, the multiple character-level word-level concatenation feature vectors are obtained by:

[0036] Extracting multiple word-level feature vectors from the original text sequence using BERT;

[0037] Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0038] Multiple word-level concatenated feature vectors are obtained by concatenating word-level feature vectors and character-level feature vectors.

[0039] Optionally, the existing knowledge point set determination module obtains the existing knowledge point set by decoding the fusion feature matrix, including:

[0040] The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

[0041] Optionally, the existing knowledge point set determination module obtains a global feature matrix, a local feature matrix, and a head-to-tail reinforcement feature matrix based on multiple character-level and word-level concatenated feature vectors of the original text sequence, including:

[0042] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level splicing feature vectors;

[0043] Obtain a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level splicing feature vectors;

[0044] A local feature matrix is ​​obtained according to the multiple character-level word-level splicing feature vectors by introducing a convolution module with a self-attention mechanism.

[0045] Optionally, the BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

[0046] The present invention provides a subject text entity extraction and keyword generation method in which a knowledge point set is obtained by decoding a fusion feature matrix, and the fusion feature matrix is ​​obtained by a global feature matrix, a local feature matrix and a head-tail reinforcement feature matrix obtained by concatenating feature vectors of multiple character-level and word-level in the original text sequence. Therefore, the present invention introduces information from multiple angles in the process of entity knowledge point recognition, thereby improving the accuracy of entity recognition. BART can be used to improve the understanding of general semantic information, and the softMask mechanism is used to distinguish the importance of different words in the original text in a "soft masking" manner, thereby enhancing the semantic information perception ability and semantic control ability, and improving the accuracy of generating entity knowledge points that do not exist in the original text and are associated with the original text. Under the condition that the accuracy of the existing knowledge point set and the generated knowledge point set is higher, a more accurate and comprehensive set of key knowledge points is obtained.

[0047] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of a method for subject text entity extraction and keyword generation provided by an embodiment of the present invention;

[0049] Figure 2 is a schematic structural diagram of an entity extraction module provided by an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of the structure of the knowledge generation module provided by an embodiment of the present invention;

[0051] Figure 4 This is a flow chart of word embedding using the SoftMask technology provided by an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of a process for obtaining a set of key knowledge points provided by an embodiment of the present invention;

[0053] Figure 6It is a structural diagram of a subject text entity extraction and keyword generation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0055] In order to solve the problem of low accuracy of existing entity recognition and generation methods in the field of education, the embodiment of the present invention provides a subject text entity extraction and keyword generation method, see Figure 1 , Figure 1 This is a flow chart of a method for subject text entity extraction and keyword generation provided by an embodiment of the present invention, which specifically includes the following steps:

[0056] Step S101: Generate an original text sequence according to the original text.

[0057] In the embodiments of the present invention, raw text refers to directly input natural language text. For example, "a process is the basic unit for resource allocation and scheduling in a system." A raw text sequence refers to a form in which the raw text is converted into a processable form. Typically, the raw text can be broken down into a word-level sequence.

[0058] Step S102, obtaining a global feature matrix, a local feature matrix and a head-to-tail reinforcement feature matrix based on multiple character-level and word-level splicing feature vectors of the original text sequence, performing feature fusion on the global feature matrix, the local feature matrix and the head-to-tail reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and obtaining a set of existing knowledge points by decoding the fused feature matrix; wherein, the character-level and word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then using a splicing operation; the set of existing knowledge points includes entity knowledge points existing in the original text.

[0059] In an embodiment of the present invention, a specific method for obtaining a character-level word-level splicing feature vector includes: extracting features for each Chinese character or letter from the original text sequence to obtain multiple character-level feature vectors, then extracting features for each word from the original text sequence to obtain multiple word-level feature vectors, and then splicing the byte feature vectors and the word-level feature vectors to obtain multiple character-level word-level splicing feature vectors. Because the character-level word-level splicing feature vector can reflect both character-level information and word-level information, the information it represents is richer.

[0060] In an embodiment of the present invention, a global feature matrix, a local feature matrix and a head-to-tail enhanced feature matrix can be obtained based on multiple character-level word-level splicing feature vectors. Among them, the global feature matrix contains the global information of the entire original text sequence, the local feature matrix contains the local context information of the original text sequence, and the head-to-tail enhanced feature matrix is ​​obtained by enhancing the features of the beginning and end parts of each entity, which is intended to capture the boundary information of each entity. In an embodiment of the present invention, an entity refers to a word with a specific meaning in a text, and specifically can be a proper noun in a text in the field of education.

[0061] The use of a multi-head self-attention mechanism can be used to more comprehensively capture the potential multiple semantic associations in a text sequence. Therefore, in an embodiment of the present invention, a multi-head self-attention mechanism is used to perform feature fusion on the global feature matrix, the local feature matrix, and the head-to-tail reinforcement feature matrix to obtain a fused feature matrix.

[0062] In an embodiment of the present invention, obtaining the existing knowledge point set by decoding the fused feature matrix includes decoding the fused feature matrix according to a preset classification label to obtain the existing knowledge point set consisting of entity knowledge points existing in the original text.

[0063] Preset classification labels serve as a standard for decoding and can include entity labels and non-entity labels. Entity labels identify portions of the text that are actual knowledge entities. Non-entity labels identify other general vocabulary or functional words in the text, such as conjunctions and particles. Therefore, by decoding the fused feature matrix based on the preset classification labels, we obtain a set of existing knowledge points consisting of the entity knowledge points present in the original text.

[0064] Step S103, based on the original text sequence, use BART and SoftMask to generate a word embedding vector containing the semantic information of the original text, and obtain a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text.

[0065] BART is a denoising autoencoder built using a sequence-to-sequence model. BART can be used to embed words in original text sequences. The introduction of SoftMask allows the model to make more accurate predictions based on contextual information while retaining the word vector representation of the original word. SoftMask can be used to weight word embeddings and selectively focus on certain words or parts, such as entities. By assigning a weight to each word, the accuracy of predictions can be improved based on the importance of the word or the contextual relationship.

[0066] Based on this, according to BART and SoftMask, a word embedding vector containing the semantic information of the original text can be generated according to the original text sequence, and based on the word embedding vector with high weight, entity knowledge points that do not exist in the original text and are associated with the original text are generated according to methods such as similarity calculation to obtain a set of generated knowledge points.

[0067] Step S104 , filtering to obtain a key knowledge point set based on the word importance score, the existing knowledge point set, and the generated knowledge point set.

[0068] In an embodiment of the present invention, the importance degree between entity knowledge points can be obtained based on word importance scores, existing knowledge point sets, and generated knowledge point sets, and a key knowledge point set can be obtained by screening.

[0069] In an embodiment of the present invention, the existing knowledge point set is obtained by decoding the fused feature matrix, and the fused feature matrix is ​​obtained by obtaining a global feature matrix, a local feature matrix and a head-tail reinforcement feature matrix based on multiple character-level word-level splicing feature vectors of the original text sequence. Therefore, the embodiment of the present invention introduces information from multiple angles in the process of entity knowledge point recognition, thereby improving the accuracy of entity recognition. BART can be used to improve the understanding of general semantic information, and the softMask mechanism is used to distinguish the importance of different words in the original text in a "soft occlusion" manner, thereby enhancing the semantic information perception ability and semantic control ability, and improving the accuracy of generating entity knowledge points that do not exist in the original text and are associated with the original text. Under the condition that the accuracy of the existing knowledge point set and the generated knowledge point set is higher, a more accurate and comprehensive set of key knowledge points is obtained.

[0070] In one implementation, the specific steps of obtaining the set of existing knowledge points can be implemented by the entity extraction module, see Figure 2 , Figure 2 This is a schematic diagram of the structure of the entity extraction module provided by an embodiment of the present invention. The entity extraction module may include an input layer, an educational information enhancement unit, a feature extraction layer, a multi-head self-attention layer, a decoding layer, and an output layer. The specific process of entity extraction based on this entity extraction module is as follows:

[0071] The original text sequence is input into the input layer. The original text sequence is obtained based on the original text. For example, the original text can be: Process is the basic unit for resource allocation and scheduling in the system. The original text sequence is X = {x1, x2, x3, ..., x n}, i = 1, 2, ..., n, n is the total number of words in the original text. When the original text is a process, which is the basic unit for resource allocation and scheduling in the system, n = 19.

[0072] In an embodiment of the present invention, BERT and a bidirectional maximum matching algorithm are introduced into the educational information enhancement unit.

[0073] In an embodiment of the present invention, multiple character-level word-level concatenation feature vectors are obtained by:

[0074] Use BERT to extract multiple word-level feature vectors from the original text sequence;

[0075] Multiple word-level feature vectors are extracted from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0076] Multiple word-level concatenated feature vectors are obtained by concatenating word-level feature vectors and character-level feature vectors.

[0077] For the input original text sequence X={x1,x2,x3,...,x n}, using BERT to extract word-level features can be based on x i Generate word-level feature vector t with rich information i .

[0078] By using the bidirectional maximum matching algorithm, the input original text sequence can be segmented and matched with the preset dictionary entities. The forward matching and reverse matching results are compared, and the segmentation result with the smallest number of segments is selected. The feature vector is constructed by one-hot encoding to obtain the dictionary feature D = d1, d2, ..., d n . i With the dictionary feature d i Splicing to get the word-level splicing feature vector w i ,

[0079] The feature extraction layer includes a global feature extraction module, an entity boundary enhancement module and a local feature extraction module.

[0080] In an embodiment of the present invention, a global feature matrix, a local feature matrix, and a head-to-tail reinforcement feature matrix are obtained based on multiple character-level and word-level concatenation feature vectors of an original text sequence, including:

[0081] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level splicing feature vectors;

[0082] According to the global feature matrix and multiple character-level word-level concatenation feature vectors, the head-tail reinforcement feature matrix is ​​obtained through two LSTM layers;

[0083] The local feature matrix is ​​obtained based on multiple character-level word-level splicing feature vectors by introducing a convolution module with a self-attention mechanism.

[0084] The global feature extraction module includes a BiGRU, or bidirectional GRU (Gated Recurrent Unit). The GRU's unique structure extracts features from multiple character-level and word-level concatenated feature vectors, preserving key and useful information. The bidirectional GRU also extracts contextual input information in both directions, enriching semantic features.

[0085] However, since the use of BiGRU will ignore some local features, a local feature extraction module is introduced in the embodiment of the present invention. The local feature extraction module includes a convolution module that introduces a self-attention mechanism. Specifically, the self-attention mechanism is introduced at the output end of two convolutional layers CNN of different sizes to enhance the extraction of local information and obtain a local feature matrix. The specific calculation formula of the convolution layer is as follows:

[0086] r q =ReLU(convld(w i ));

[0087] Where r q is the output result of the convolution layer, w i is the character-level word-level concatenation feature vector, convld(·) represents convolution, ReLU(·) is the nonlinear activation function, and q represents the convolution kernel size.

[0088] In educational texts, there is no explicit word boundary information. Therefore, in the embodiment of the present invention, the entity boundary information of words is enhanced to improve the model's ability to extract entity features. Specifically, two single-layer LSTM networks are introduced into the entity boundary enhancement module to predict the head entity and the tail entity, and obtain the head and tail enhanced feature matrix. start Can be used to predict head entities, LSTM end It can be used to predict the tail entity and enhance the boundary information of the entity's head and tail positions by fusing it with global and local features. The head entity refers to the first word in an entity, and the tail entity refers to the last word in an entity.

[0089] In an embodiment of the present invention, a multi-head self-attention layer is introduced to first fuse the input global feature matrix, local feature matrix and head-to-tail reinforcement feature matrix to obtain an initial fusion matrix, and then the multi-head self-attention mechanism is used to extract features from the initial fusion matrix from three different spatial angles through three attention heads, Head1, Head2 and Head3, to obtain information feature matrices of three different spaces, and then the information feature matrices of the three different spaces are spliced ​​through the full connection layer Full Connection to obtain a fusion feature matrix.

[0090] In an embodiment of the present invention, the multi-head self-attention mechanism is an improvement based on self-attention, focusing more on obtaining the internal associations of information. Specifically, this mechanism maps the same entity multiple times. Each time it is mapped to a space, the entity is given a new meaning. In other words, feature extraction is performed on the data from different perspectives. Finally, the information features of different spaces are spliced ​​together to form the final fused feature representation. By inputting the output of the feature extraction layer into this layer, the model's ability to obtain key information can be indirectly improved, the waste of feature information can be avoided, and the prediction efficiency of the model can be improved.

[0091] In an embodiment of the present invention, the decoding layer can decode the fused feature matrix according to the preset classification labels to obtain a set of existing knowledge points consisting of entity knowledge points present in the original text. The decoding layer can be a CRF (conditional random field). The CRF can combine context-related constraints to proofread the correlation between labels, thereby reducing the probability of outputting incorrect labels. The preset classification labels can include entity labels and non-entity labels.

[0092] Finally, the output layer outputs the results, where O represents the non-entity knowledge point in the original text under the non-entity label, and B-Know and I-Know represent the entity knowledge points in the original text under the entity label. Specifically, B-Know represents the first word of the entity knowledge point, and I-Know represents the last word of the entity knowledge point.

[0093] The set of existing knowledge points can be obtained by using B-Know and I-Know, wherein the set of existing knowledge points includes entity knowledge points existing in the original text.

[0094] In an embodiment of the present invention, an educational information enhancement unit that integrates Bert word-level features and external preset dictionary entity features enhances semantic representation and is more suitable for professional field texts; and the feature extraction layer pays more attention to local features on the basis of global features and enhances the feature extraction capability by strengthening the boundary information of the entity's head and tail positions; finally, through the multi-head self-attention layer, the discriminant features of the entity are extracted from multiple angles, which helps to improve the fitting ability of the overall model, thereby improving the accuracy of subject text entity extraction.

[0095] In an embodiment of the present invention, BERT is used to fuse the left and right context information to enhance the semantic representation of the word and alleviate the problem of polysemy; a preset external dictionary in the field of education is built, and a bidirectional maximum matching strategy is introduced to obtain distributed dictionary feature representation, thereby improving the model's recognition accuracy for rare or unknown knowledge entities; at the feature extraction level, attention is paid to global features, and a bidirectional gated recurrent unit BiGRU is introduced to better obtain long-sequence semantic features of sentence dependencies and solve the gradient vanishing problem. Compared with the traditional LSTM network, GRU has fewer parameters, faster convergence speed, and lower computational complexity than LSTM. At the same time, GRU has relatively high efficiency in utilizing previous and next information, is not easily dominated by close context, and is not easily ignored by key information at a distance. Therefore, GRU is more efficient and has more advantages in global feature extraction; attention is paid to local features, and by introducing a self-attention mechanism based on the convolutional layer, more emphasis is placed on key knowledge-guided entity recognition; because there is no explicit word boundary information in education texts, two single-layer LSTM networks are introduced to predict the head entity and the tail entity, obtain the head and tail features of the entity, and strengthen the boundary information of the head and tail positions of the entity by fusing with global and local features. The fused results are fed into a multi-head self-attention layer to extract discriminative features of entities from multiple perspectives, helping to improve the overall model's fitting capabilities. This is in contrast to traditional attention mechanisms, which lack consideration of features from different perspectives and are susceptible to data noise. Finally, a conditional random field (CRF) model is used to generate a globally optimal sequence. Compared to existing technologies, this embodiment of the present invention better identifies polysemy in the education field and achieves higher recognition accuracy for rare or unknown knowledge entities.

[0096] In one implementation, the specific steps of obtaining and generating a knowledge point set can be implemented by a knowledge generation module, see Figure 3 , Figure 3 This is a schematic diagram of the structure of the knowledge generation module provided by an embodiment of the present invention. The knowledge generation module includes a word embedding layer, a BART encoding layer, a BART decoding layer, and a pointer generation network. BART is a Transformer that combines contextual information and autoregressive properties. The specific process of acquiring and generating a knowledge point set based on this knowledge generation module is as follows:

[0097] The importance scores of words can be obtained in the multi-head self-attention layer in the entity extraction module. After the sigmoid function transformation, the importance scores of each word can be converted into the range of [0,1], thereby obtaining the keyword probability score p.

[0098] In the word embedding layer, the application of SoftMask technology enables the knowledge generation module to focus on the key part of the original sentence, that is, the entity part, while reducing its attention to the non-keyword part, that is, the non-entity part. For sourcetext, the original text sequence X = {x1, x2, ..., x n}, whose length is n, and the vector representation Emb of the original text sequence X is obtained through BART word embedding technology x ={Emb x1 ,Emb x2 ,…,Emb xn Combined with the importance scores of the words obtained above, we can get the entity probability score p of the original text sequence X = {p1, p2, ..., p n}. Apply SoftMask technology and introduce a special symbol [MASK] word embedding vector Emb MASK The original meaning of the [MASK] tag is to mask the words in the actual input original text sequence, that is, to directly use the [MASK] tag to replace certain words in the input original text, thereby playing a direct masking role. Since the [MASK] tag itself is used to replace words, the word itself can also be considered as a special word. Therefore, [MASK] will exist in the vocabulary of the knowledge generation module as a "word", so [MASK] will also correspond to a word embedding representation recorded as Emb MASK , the word vector representation will also be updated as the network parameters are trained. Figure 4 , Figure 4 This is a flow chart of word embedding using SoftMask technology provided by an embodiment of the present invention. SoftMask uses a form of "soft masking" to represent the original original sentence word vector. x , adjusted to its word vector Emb with [MASK] MASK The weighted average of the original sentence is p and (1-p), representing the probability that each word in the original text is key knowledge, that is, an entity, and non-key knowledge, that is, a non-entity. The adjusted original sentence embedding is expressed as SoftEmb, which is the softmasked embedding of the original sentence. SoftEmb is calculated by multiplying the original sentence embedding by the corresponding element in p and multiplying the [MASK] embedding by the corresponding element in (1-p), and then adding the corresponding elements. The specific calculation formula is expressed as:

[0099] SoftEmb x =p*Emb x +(1-p)*Emb MASK ;

[0100] This approach, based on importance scores and using the [MASK] word vector to weightedly mask non-important parts, reduces information about these parts. At the same time, words with less "soft masking" have a greater relative amount of information. In this way, softmask smoothly distinguishes the primary and secondary parts of the original sentence.

[0101] In an embodiment of the present invention, the BART encoding layer is composed of 12 superimposed Transformer Encoder Layers (a neural network model structure encoder layer). This part of the structure can be regarded as similar to the BERT model. Its input is text Input Embedding and Positional Encoding is added to generate the word vector of the text to ensure that each Transformer Encoder Layer can take into account the relative position information of the text to a certain extent while performing parallel operations.

[0102] Among them, the Transformer Encoder Layer consists of a multi-head attention mechanism and a fully connected layer. The multi-head attention mechanism consists of multiple self-attention mechanisms. The self-attention mechanism is implemented by W Q 、W K and W V Three preset matrices transform the word vectors of the original text into the Q, K, and V vectors required by the self-attention mechanism, and then calculate the Attention value of the original text. The calculation formula is as follows:

[0103]

[0104] Among them, d k is the dimension of the K vector, and T represents the transpose of the matrix. To prevent performance from being affected by too many network layers, the model concatenates the attention values ​​of each character rather than feeding them directly into the fully connected layer. Instead, it performs multiple Add & Norm (residual connections) and Layer Norm (normalization) operations to obtain the hidden layer vector, i.e., the hidden state of the BART encoding layer. This ultimately converts the input text into an embedding vector containing its semantic information.

[0105] The input of the BART decoding layer is the entity knowledge points present in the original text, that is, generating words, and the output is the hidden state of the BART decoding layer. The output end of the BART decoding layer and the output end of the BART encoding layer are both connected to the pointer generation network. After the BART decoding layer outputs the BART decoding layer hidden state, in the pointer generation network, the BART decoding layer hidden state is combined with the BART encoding layer hidden state to generate an attention distribution, that is, a context vector. The decoding layer hidden state is combined with the context vector and fed through two linear layers to generate a vocabulary distribution. After that, the decoding layer hidden state will be similar to the BART encoding layer hidden state of the original text vocabulary pointed to by its corresponding attention distribution, and then the generation probability parameter P of the PGN module is calculated based on this similarity and other parameters. gen , and ultimately decides whether to generate the vocabulary. Since the distribution of the BART model word vectors in space is not uniform, but conical, the cosine similarity calculation method cannot be simply used. The word vector coordinate basis generated by BART is not a standard orthogonal basis, that is, there is anisotropy, and the word vector needs to be converted into a standard orthogonal basis. The embodiment of the present invention selects the whitening (BERT-whitening) method, which is to convert all word vectors into vectors with a mean of 0 and a covariance matrix of a unit matrix.

[0106] When the similarity is high enough, the value of the generation probability parameter will increase accordingly, and the knowledge generation module will be more inclined to retain the generated vocabulary. Otherwise, the model will be more inclined to copy the vocabulary in the original text. In this way, when the knowledge generation module generates vocabulary with different semantics from the original text, it will discard the vocabulary and copy the vocabulary of the original text instead. Finally, the attention distribution and vocabulary distribution generate the final distribution under the action of the copy mechanism, and the generated knowledge point set is obtained, which improves the accuracy and stability of the text summary generated by the knowledge generation module. Among them, 1-P gen represents the non-keyword probability score, that is, the non-entity probability score, P gen Represents the keyword probability score, that is, the entity probability score.

[0107] In an embodiment of the present invention, the importance score of each word in the original text obtained is used as auxiliary keyword information for the knowledge generation module; the keyword information is then combined with the softMask mechanism to distinguish the primary and secondary parts of the original text sentence in a "soft masking" manner, thereby realizing the semantic information perception ability of the knowledge generation model and enhancing the semantic control ability; the large language pre-trained BART generation model is used to improve the understanding of general semantic information, and at the same time, the pointer generation network is integrated after the BART decoding layer to reduce the probability of the knowledge generation model generating semantically incorrect vocabulary that is inconsistent with the original text. Compared with the existing technology, the embodiment of the present invention can pay more attention to the importance of text words for knowledge generation in the field of education, and at the same time can reduce the probability of generating semantically incorrect vocabulary that is inconsistent with the original text.

[0108] See also Figure 5 , Figure 5 This is a flow chart of obtaining a key knowledge point set provided by an embodiment of the present invention. For the original text data set, i.e., the original text sequence, after obtaining an existing knowledge point set through knowledge recognition and a generated knowledge point set through knowledge generation, the existing knowledge point set is combined with the word importance score as the first part of the input, and the generated knowledge point set is combined with the attention score as the second part of the input for knowledge fusion. Then, based on the existing knowledge point set and the word importance score, the importance of words under different classification labels relative to the subject text words can be distinguished; based on the generated knowledge point set and the attention score, the importance of the generated knowledge points can be distinguished, thereby obtaining a key knowledge information set, i.e., a key knowledge point set, through screening. Among them, the word importance score is extracted by fusing the global feature matrix, the local feature matrix, and the head-tail reinforcement feature matrix from the multi-head self-attention layer to obtain a fused feature matrix.

[0109] Different classification labels are assigned different importance coefficients. For example, knowledge related to theorems and rules and generated knowledge has higher importance coefficients, while knowledge related to laboratory equipment has slightly lower importance. The existing knowledge point set and the generated knowledge point set are then filtered, and the knowledge points with high similarity between the two sets are selected using the minimum edit distance similarity combined with the Jaccard similarity method. The selected knowledge points appear in both sets, indicating that they are more important than other knowledge in the subject text. Therefore, these knowledge points are assigned higher importance coefficients. The relative importance of the remaining knowledge points that were not selected is calculated based on the importance coefficients of different categories and their own importance scores or attention scores. Finally, in the final filtering step, knowledge points with extremely low importance are removed, and the final set of key knowledge points is formed through knowledge fusion.

[0110] Based on the same inventive concept, the present invention also provides a subject text entity extraction and keyword generation device, see Figure 6 , Figure 6 This is a schematic diagram of the structure of a subject text entity extraction and keyword generation device provided by an embodiment of the present invention, the subject text entity extraction and keyword generation device includes:

[0111] The original text sequence generating module 601 is used to generate an original text sequence according to the original text;

[0112] The existing knowledge point set determination module 602 is configured to obtain a global feature matrix, a local feature matrix, and a head-end reinforcement feature matrix based on multiple character-level and word-level splicing feature vectors of the original text sequence, perform feature fusion on the global feature matrix, the local feature matrix, and the head-end reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and obtain an existing knowledge point set by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then performing a splicing operation; the existing knowledge point set includes entity knowledge points existing in the original text;

[0113] A generated knowledge point set determination module 603 is configured to generate a word embedding vector containing semantic information of the original text using BART and SoftMask based on the original text sequence, and obtain a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text but are associated with the original text;

[0114] The key knowledge point set screening module 604 is used to screen the key knowledge point set according to the word importance score, the existing knowledge point set and the generated knowledge point set to obtain the key knowledge point set.

[0115] In an embodiment of the present invention, the existing knowledge point set is obtained by decoding the fused feature matrix, and the fused feature matrix is ​​obtained by obtaining a global feature matrix, a local feature matrix and a head-tail reinforcement feature matrix based on multiple character-level word-level splicing feature vectors of the original text sequence. Therefore, the embodiment of the present invention introduces information from multiple angles in the process of entity knowledge point recognition, thereby improving the accuracy of entity recognition. BART can be used to improve the understanding of general semantic information, and the softMask mechanism is used to distinguish the importance of different words in the original text in a "soft occlusion" manner, thereby enhancing the semantic information perception ability and semantic control ability, and improving the accuracy of generating entity knowledge points that do not exist in the original text but are associated with the original text. Under the condition that the accuracy of the existing knowledge point set and the generated knowledge point set is higher, a more accurate and comprehensive set of key knowledge points is obtained.

[0116] Optionally, the multiple character-level word-level concatenation feature vectors are obtained by:

[0117] Extracting multiple word-level feature vectors from the original text sequence using BERT;

[0118] Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding;

[0119] Multiple word-level concatenated feature vectors are obtained by concatenating word-level feature vectors and character-level feature vectors.

[0120] Optionally, the existing knowledge point set determination module obtains the existing knowledge point set by decoding the fusion feature matrix, including:

[0121] The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

[0122] Optionally, the existing knowledge point set determination module obtains a global feature matrix, a local feature matrix, and a head-to-tail reinforcement feature matrix based on multiple character-level and word-level concatenated feature vectors of the original text sequence, including:

[0123] The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level splicing feature vectors;

[0124] Obtain a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level splicing feature vectors;

[0125] A local feature matrix is ​​obtained according to the multiple character-level word-level splicing feature vectors by introducing a convolution module with a self-attention mechanism.

[0126] Optionally, the BART includes a BART encoding layer and a BART decoding layer; an output end of the BART decoding layer and an output end of the BART encoding layer are both connected to a pointer generation network.

[0127] It should be noted that the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0128] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0129] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the word "comprising" does not exclude other components or steps, "one" or "a" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0130] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0131] It should be noted that the device of the embodiment of the present invention is a device that applies the above-mentioned neural network-based subject text entity extraction and keyword generation method. All embodiments of the above-mentioned neural network-based subject text entity extraction and keyword generation method are applicable to the device and can achieve the same or similar beneficial effects.

[0132] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for subject text entity extraction and keyword generation, characterized in that: The subject text entity extraction and keyword generation method includes: Generate an original text sequence according to the original text; A global feature matrix, a local feature matrix, and a head-end reinforcement feature matrix are obtained based on multiple character-level and word-level splicing feature vectors of the original text sequence, and a multi-head self-attention mechanism is used to perform feature fusion on the global feature matrix, the local feature matrix, and the head-end reinforcement feature matrix to obtain a fused feature matrix, and a set of existing knowledge points is obtained by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing character-level feature extraction and word-level feature extraction on the original text sequence and then performing a splicing operation; the set of existing knowledge points includes entity knowledge points existing in the original text; Based on the original text sequence, using BART and SoftMask to generate a word embedding vector containing the semantic information of the original text, and obtaining a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text; The key knowledge point set is obtained by screening according to the word importance score, the existing knowledge point set and the generated knowledge point set.

2. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: The multiple character-level word-level concatenation feature vectors are obtained by: Extracting multiple word-level feature vectors from the original text sequence using BERT; Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding; Multiple word-level concatenated feature vectors are obtained by concatenating word-level feature vectors and character-level feature vectors.

3. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: The existing knowledge point set is obtained by decoding the fusion feature matrix, including: The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

4. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: A global feature matrix, a local feature matrix, and a head-to-tail reinforcement feature matrix are obtained based on multiple character-level and word-level concatenated feature vectors of the original text sequence, including: The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level splicing feature vectors; Obtain a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level splicing feature vectors; A local feature matrix is ​​obtained according to the multiple character-level word-level splicing feature vectors by introducing a convolution module with a self-attention mechanism.

5. The subject text entity extraction and keyword generation method according to claim 1 is characterized in that: The BART includes a BART encoding layer and a BART decoding layer; the output end of the BART decoding layer and the output end of the BART encoding layer are both connected to a pointer generation network.

6. A device for subject text entity extraction and keyword generation, characterized in that: The subject text entity extraction and keyword generation device includes: An original text sequence generation module, used to generate an original text sequence according to the original text; An existing knowledge point set determination module is configured to obtain a global feature matrix, a local feature matrix, and a head-end reinforcement feature matrix based on multiple character-level and word-level splicing feature vectors of the original text sequence, perform feature fusion on the global feature matrix, the local feature matrix, and the head-end reinforcement feature matrix using a multi-head self-attention mechanism to obtain a fused feature matrix, and obtain an existing knowledge point set by decoding the fused feature matrix; wherein the character-level and word-level splicing feature vectors are obtained by performing a splicing operation after performing character-level feature extraction and word-level feature extraction on the original text sequence; and the existing knowledge point set includes entity knowledge points existing in the original text; A generated knowledge point set determination module is configured to generate a word embedding vector containing semantic information of the original text using BART and SoftMask based on the original text sequence, and obtain a generated knowledge point set based on the word embedding vector; the generated knowledge point set includes entity knowledge points that do not exist in the original text and are associated with the original text; The key knowledge point set screening module is used to screen the key knowledge point set according to the word importance score, the existing knowledge point set and the generated knowledge point set to obtain the key knowledge point set.

7. The subject text entity extraction and keyword generation device according to claim 6 is characterized in that: The multiple character-level word-level concatenation feature vectors are obtained by: Extracting multiple word-level feature vectors from the original text sequence using BERT; Extracting multiple word-level feature vectors from the original text sequence using a bidirectional maximum matching algorithm and one-hot encoding; Multiple word-level concatenated feature vectors are obtained by concatenating word-level feature vectors and character-level feature vectors.

8. The subject text entity extraction and keyword generation device according to claim 6 is characterized in that: The existing knowledge point set determination module obtains the existing knowledge point set by decoding the fusion feature matrix, including: The fusion feature matrix is ​​decoded according to preset classification labels to obtain an existing knowledge point set consisting of entity knowledge points existing in the original text.

9. The subject text entity extraction and keyword generation device according to claim 6 is characterized in that: The existing knowledge point set determination module obtains a global feature matrix, a local feature matrix, and a head-to-tail reinforcement feature matrix based on multiple character-level and word-level splicing feature vectors of the original text sequence, including: The global feature matrix is ​​obtained by BiGRU based on multiple character-level word-level splicing feature vectors; Obtain a head-to-tail reinforcement feature matrix through two LSTM layers according to the global feature matrix and the multiple character-level word-level splicing feature vectors; A local feature matrix is ​​obtained according to the multiple character-level word-level splicing feature vectors by introducing a convolution module with a self-attention mechanism.

10. The subject text entity extraction and keyword generation device according to claim 6, characterized in that: The BART includes a BART encoding layer and a BART decoding layer; the output end of the BART decoding layer and the output end of the BART encoding layer are both connected to a pointer generation network.

Images