A linear structure-based information extraction method, device and medium

Abstract

The present application discloses a linear structure-based information extraction method, device, and medium. The method includes: designing a linear structure encoding, jointly representing entity information and entity relationships, setting a prompt template, training a pre-trained language model to obtain a generative language model, using an attention residual structure in the training of the pre-trained language model, obtaining natural language data, encoding the corresponding input sequence based on the linear structure encoding, encoding it through an encoder, and decoding it through a decoder according to the input order of the input sequence, obtaining a first prediction probability of a target vocabulary output by the decoder, obtaining a second prediction probability of the target information, and obtaining the target information based on the second prediction probability. The method realizes the joint extraction of entity information and relationship information, effectively integrates entity information and relationship information, and makes the two information extraction tasks have the same underlying operation in the training of the model, thereby enhancing the dependency between tasks.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and specifically to a method, device, and medium for extracting information based on a linear structure. Background Art

[0002] With the deepening of research in the field of natural language processing, information extraction tasks have become an important branch field. With the increasing application of Internet technology, the number of scientific literature published has increased exponentially, and most scientific research results are saved in numerous scientific literature in the form of unstructured electronic documents. As new methods and data are constantly proposed, some scientific research results are needed as support or comparison. The comparison of different methods in different scientific research results requires a lot of manpower costs.

[0003] Traditional information extraction methods usually adopt a pipeline-based approach, which divides multiple information extraction tasks into separate subtasks and requires training different models for different subtasks. This method severs the connection between subtasks, resulting in the model being unable to learn the dependencies between subtasks. At the same time, the errors of the previous task will be passed on to subsequent tasks, resulting in the overall performance of the model deteriorating. Therefore, this method is not suitable for long texts such as scientific literature with long distances between entities. Summary of the Invention

[0004] In order to solve the above problems, this application proposes an information extraction method based on linear structure, including:

[0005] Designing a linear structure code to jointly represent entity information and entity relationships, and setting a prompt template to guide a pre-trained language model to generate information corresponding to the linear structure code through the prompt template;

[0006] Training the pre-trained language model according to the prompt template to obtain a generative language model;

[0007] In the training of the pre-trained language model, an attention residual structure is used, and the residual block in the attention residual structure is placed in the self-attention head of the multi-head attention layer of the pre-trained language model to achieve training optimization of the pre-trained model;

[0008] Acquire natural language data, obtain the corresponding input sequence based on the linear structure encoding in the generative language model, encode the input sequence through an encoder, and decode it through a decoder according to the input order of the input sequence to obtain a first prediction probability of the target vocabulary output by the decoder, obtain a second prediction probability of the target information based on the first prediction probability, and obtain the target information based on the second prediction probability.

[0009] On the other hand, the present application also proposes an information extraction device based on a linear structure, comprising:

[0010] at least one processor; and,

[0011] a memory communicatively connected to the at least one processor; wherein,

[0012] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the linear structure-based information extraction method as described in the above example.

[0013] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: the linear structure-based information extraction method described in the above example.

[0014] This application proposes an information extraction method based on linear structure, which can bring the following beneficial effects:

[0015] A linear structure is designed, and a prompt template is used to correspond to the linear structure. This linear structure combines the extraction results of the named entity recognition task and the extraction results of the inter-entity relationship extraction task, and uniformly encodes the extraction results. Compared with the method of directly marking and extracting in natural language, the extraction using this linear structure can be more accurate and more suitable for application in long text data.

[0016] By training a generative language model with linear structured data, tasks that are not text generation tasks are transformed into text generation tasks, achieving the joint extraction of entity information and relationship information, effectively integrating entity information and relationship information, and making the two information extraction tasks have the same underlying operations in the training of the model, thereby enhancing the dependency between tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a flow chart of a linear structure-based information extraction method in an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of a joint extraction model based on a parameter sharing method in an embodiment of the present application;

[0020] Figure 3This is a schematic diagram of a joint extraction model based on a sequence annotation method in an embodiment of the present application;

[0021] Figure 4 This is a schematic diagram of a joint extraction model based on a generative method in an embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of the linear structure encoding format in the embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of the prompt template structure in the embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the encoder layer structure in an embodiment of the present application;

[0025] Figure 8 This is a schematic diagram of the decoder layer structure in an embodiment of the present application;

[0026] Figure 9 This is a schematic diagram of the Post-LN Transformer structure in an embodiment of the present application;

[0027] Figure 10 This is a schematic diagram of the Pre-LN Transformer structure in an embodiment of the present application;

[0028] Figure 11 This is a schematic diagram of the attention residual structure in the embodiment of this application;

[0029] Figure 12 This is a schematic diagram comparing the data efficiency of different models in the embodiments of this application;

[0030] Figure 13 This is a schematic diagram of an information extraction device based on a linear structure in an embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0033] like Figure 1 As shown, the embodiment of the present application provides an information extraction method based on a linear structure, comprising:

[0034] S101: Design a linear structure encoding to jointly represent entity information and entity relationships, and set a prompt template to guide the pre-trained language model to generate linear structure information through the prompt template.

[0035] Information extraction involves two main tasks: named entity recognition (NER) and inter-entity relationship extraction. Named entity recognition involves identifying entities in unstructured text and classifying them into predefined entity types. Inter-entity relationship extraction builds on entity recognition by determining entity boundaries and entity types and then determining the relationships between them. Once the relationships between entities are determined, information is extracted to obtain structured data corresponding to the text, which is then stored in a graph structure.

[0036] In traditional pipeline information extraction methods, multiple information extraction tasks are divided into separate subtasks, which breaks the connection between the subtasks. As a result, the model cannot learn the dependencies between the subtasks during training, resulting in poor overall model performance. To address the problems arising from pipeline information extraction methods, methods based on parameter sharing and sequence labeling have been proposed in the prior art.

[0037] Among them, based on the parameter sharing method, entity and relationship information are modeled separately during the information extraction process, such as Figure 2 As shown in , the subtasks are implemented through the encoding layer of the shared joint model, that is, the shared encoding layer, loss1 and loss2 are the errors generated by the subtasks, and loss is the total error. During the model training process, back propagation is used to update the shared parameters of the encoding layer, so that the model can learn the dependencies between subtasks and reduce the error transmission between tasks. However, during the training process of the shared joint model, the entity recognition and relationship classification processes in the information extraction task are not synchronized. This method does not achieve the true joint extraction of entity information and entity relationships. Sequence labeling-based methods, such as Figure 3 As shown in the figure, during the model training process, by using a unified decoding layer to decode entity information and entity relationships at the same time, true entity recognition and relationship classification information sharing can be achieved. However, due to the cumbersome label design and decoding process, this method is not practical. In order to solve the problems in the above-mentioned prior art, this application proposes an information extraction method based on linear structure, such as Figure 4 As shown in the figure, a generative language model is trained, where Encoder is the encoder and Decoder is the decoder.

[0038] Before training the pre-trained language model for information extraction, the natural language inference model also needs to be pre-trained with some data processing to obtain the pre-trained language model. Specifically, natural language data is collected from published scientific literature. Using dataset construction methods (such as the Crocodile dataset construction method), entity and relationship types are defined. Long text data is then labeled and classified according to entity and relationship types. After classification, extraction is performed and the extracted entity and relationship information is integrated to obtain a standard dataset.

[0039] For example, "The convolutional neural network extracts local features in the image by using convolutional layers and pooling layers, and combines and classifies features through multiple convolutional layers and fully connected layers." For a long text data, according to the entity type and relationship type, the internship and relationship information in this long text data is extracted, including entity information: convolutional neural network, convolutional layer, pooling layer, image, local features, multiple convolutional layers, fully connected layer, features; relationship information: "The convolutional neural network extracts local features in the image by using convolutional layers", "The pooling layer combines and classifies features through multiple convolutional layers and fully connected layers". The entity information and relationship information are integrated to obtain a standard data set, which is: "The convolutional neural network extracts local features in the image by using convolutional layers and pooling layers, and combines and classifies features through multiple convolutional layers and fully connected layers."

[0040] Furthermore, a standard dataset is input into a natural language inference model. Through relevant processing at the natural language inference model's decoding layer, the predicted probabilities of the relationships between entities in the standard dataset are obtained. A pre-set probability value is then set, and relationships with predicted probabilities below the pre-set probability value are filtered out to eliminate noise information such as annotations in the standard dataset. For example, a pre-set probability value of 0.75 is set to filter out relationships with predicted probabilities below 0.75, thereby filtering out relationships that do not exist in the data. Based on the filtered standard dataset, the natural language inference model is trained to process the natural language data into structured source text sequences. After this training is completed, negative examples are added to the model. For example, in the sentence "This Must Be the Place" is a song by the new wave band Talking Heads, released in November 1983, the relationship (This Must Be the Place, performer, Talking Heads) exists. The relationship "work for" (which does not exist in the sentence) is selected as a negative example and input into the model to train the model's generalization ability. After the above training is completed, a pre-trained language model is obtained.

[0041] Furthermore, if Figure 5 As shown, based on the traditional structure of representing entity information and entity relationships, such as tuples and triples, a linear structure encoding is designed to integrate the linear structures of tuples and triples, and the format of the linear structure encoding is ((entity type 1: entity name 1 (relationship type 1: relationship 1) (relationship type 2: relationship 2)) (entity type 2: entity name 2)). For example, for a piece of information, its tuple structure is (Entity-Type, Span): (Method, BiDAF), and its triple structure is (Entity, Relation-Type, Entity): (BiDAF, USED-FOR, QuestionAnswering). The structure based on the linear structure encoding is ((Method: BiDAF (USED-FOR: QuestionAnswering))). This linear structure encoding includes reversibility. It not only represents entity information and entity relationships in a structured manner based on a unified format, but also minimizes the number of tags that need to be decoded. It can effectively integrate entity information and entity relationships, realize the joint extraction of entities and relationships by the model, and avoid problems such as error propagation and neglect of dependencies between subtasks that occur in pipeline methods.

[0042] like Figure 6 As shown, a prompt template is set. The prompt template corresponds to the linear structure, guiding the pre-trained language model to generate information corresponding to the linear structure encoding. The prompt template includes two fields corresponding to entity information and relationship information: EntityType entity type field and RelationType relationship type field, such as entity type "person" and "subject", and relationship type "location". Entity is the entity information to be extracted, and Relation is the relationship information to be extracted. The prompt template can contain weak supervision signals about the task and can better use label semantics.

[0043] S102: Training the pre-trained language model according to the prompt template to obtain a generative language model.

[0044] The natural language data is preprocessed through the pre-trained language model to obtain the corresponding source text sequence. The source text sequence is combined with the prompt template to obtain the corresponding linear structure encoding information, and this information is used as the input column and input into the pre-trained language model to achieve the training effect of the model.

[0045] Specifically, if Figure 7As shown, the input sequence is input into the encoder, and the encoder obtains the position sequence according to the input order of the input sequence, and at the same time embeds the text information and position information of the input sequence into the multi-head attention layer in the encoder, and processes it based on the multi-head attention mechanism. The attention mechanism uses attention weights to weight the input content, and then adds the weighted input to the original input to obtain the final output. The encoder is based on the formula H = {h1,…,h |p| ,h |p+1| ,…,h |p+x|}Calculate the hidden vector of the input sequence, transform the features of the input sequence into the hidden vector, and represent the features of the input sequence through the hidden vector, where X={x1,x2,…,x |x|} is the source text sequence, Input={p1,p2,…,p |p| ,x1,x2,…,x |x|} is the input sequence encoded by the prompt template combined with the linear structure.

[0046] Furthermore, based on the formula H=Encoder(p1,p2,…,p |p| ,x1,x2,…,x |x| ), we get the encoder layer part with multi-head attention mechanism. Figure 8 As shown, the decoder follows the order of the position sequence based on the formula y i , The hidden vector of each character in the input sequence is predicted character by character to obtain the structure of the entire output sequence, where i is the i-th step of the prediction, y i is the predicted i-th character, is the state of the decoder at step i.

[0047] Furthermore, the decoder obtains a first prediction probability of the target vocabulary based on the decoded characters and the target characters. After decoding the entire input sequence, it obtains the entire output sequence. Based on the output sequence and the target information, it obtains a second prediction probability of the target information. The model continues training to reduce the gap between the output sequence and the target information and improve the second prediction probability to achieve optimal model training. After training is complete, a generative language model is obtained that can jointly extract entity information and entity relationships from natural language data. This allows the model to better utilize the shared knowledge and dependencies between the named entity recognition task and the inter-entity relationship extraction task, effectively solving the problems of error accumulation and loss of interaction that are prone to occur during the extraction process.

[0048] S103: In the training of the pre-trained language model, an attention residual structure is used, and the residual block in the attention residual structure is placed in the self-attention head of the multi-head attention layer of the pre-trained language model to achieve training optimization of the pre-trained model.

[0049] Among them, in the training process of the pre-trained language model, such as Figure 9 As shown in the figure, Attention is a multi-head attention layer and LN is normalization. In order to solve the problem of gradient disappearance caused by the encoder layer link, there are generally two structures for processing. The first structure is the Post-LN Transformer structure. Residual connection and layer normalization are used after the multi-head attention layer and the feedforward neural network layer. The layer normalization formula is LayerNorm(x+f(x)), x is the input tensor, that is, the input sequence, and f(x) is the introduced function. It is used for model applications in different linear transformations, nonlinear transformations and other complex operations. It is determined according to the specific application scenario. By introducing the f(x) function, additional operations can be introduced while maintaining normalization, thereby increasing the nonlinear ability of the model and improving the expression ability and performance of the model. However, the layer normalization in this structure will weaken the role of the residual block and increase the risk of gradient disappearance again.

[0050] like Figure 10 As shown in the figure, the second structure is the Pre-LN Transformer structure, which puts the layer normalization before the multi-head attention layer and the feedforward neural network layer. The input sequence is normalized before passing through the multi-head attention layer. This structure may cause the information of the input sequence to disappear, resulting in inaccurate calculated hidden vectors of the input sequence. The model is constantly adjusting the information deviation during the training process, which makes model training more difficult.

[0051] Therefore, based on the Post-LN Transformer structure, in order to solve the problems that arise in the Post-LN Transformer structure, such as Figure 11 As shown in Figure 1, a residual block PrevAttention is added to the self-attention head in the multi-head attention layer of the encoder to solve the problem that the weakening of the residual block caused by the layer normalization process increases the risk of gradient vanishing again.

[0052] Specifically, adding a residual block to the self-attention head of the multi-head attention layer can enable each head in the multi-head attention layer to better capture the original features of the input sequence and retain more detailed information. Through the processing of the multi-head attention mechanism, the hidden vector of the input sequence, that is, the input feature, is obtained. At the same time, the output feature of the multi-head attention layer is also obtained. The input feature is added to the output feature to obtain the result of the residual connection. At the same time, the attention score of the multi-head attention layer after the residual connection is obtained. Based on the attention score of the multi-head attention layer, the formula The output of the multi-head attention layer of the next layer is calculated. Where Q is the query matrix of the input sequence, K is the key matrix of the input sequence, and V is the value matrix of the input sequence. The matrix W is calculated by linear transformation of the input sequence. Q 、W V 、W K Calculated, d k It is a hyperparameter defined artificially to prevent the size of Q and K from being too large, based on ResMultiHead(Q,K,V,Prev)=Cincat(head1,…,head h )W O , convert Q, K, and V into h sub-queries, sub-keys, and sub-values, and obtain the output attention score of the multi-head attention layer by connecting multiple attention heads and calculating with the final weight matrix. In Formula 2-5, the query, key, and value matrices are converted into h sub-queries, sub-keys, and sub-values, which are then input into the dot product attention model respectively. After that, multiple attention heads are connected and calculated with the final weight matrix to obtain the multi-head attention output. Using this method, not only can the accuracy of the input sequence information be guaranteed, but the gradient of the subsequent layers is also further reduced, making the gradients of the previous and next layers more synchronized, which can further optimize the training of the pre-trained language model.

[0053] S104: Acquire natural language data, obtain the corresponding input sequence based on the linear structure encoding in the generative language model, encode the input sequence through an encoder, and decode it through a decoder according to the input order of the input sequence to obtain a first prediction probability of the target vocabulary output by the decoder.

[0054] This step is the usage step after the model training is completed. Specifically, natural language data is obtained and input into the generative language model. The generative language model performs linear structure preprocessing on the natural language data based on the prompt template to obtain an input sequence. The input sequence is input into the encoder of the model. The encoder encodes the input sequence and also obtains the attention score corresponding to each character of the input sequence, that is, the attention weight.

[0055] Furthermore, the decoder decodes the input sequence according to the position sequence order of the input sequence, predicts a first prediction probability of the target vocabulary based on the attention score, integrates and calculates the second prediction probability of the target information based on the first prediction probability of the target vocabulary, and obtains the target information based on the second prediction probability.

[0056] In order to verify the training effect of the model, corresponding experiments are set up in the embodiment of the present application for verification. Specifically, a data set is obtained, and the real entity information and entity relationships in the data set are annotated using the method of domain experts. The data set is divided into a training set, a test set and a validation set. The specific information is as follows:

[0057] As shown in Table 1:

[0058]

[0059] Table 1

[0060] The dataset contains six annotated real-world scientific entity types (Task, Material, Metric, Method, Other-Scientific Term, and Generic), and seven relationship types (Compare, Part-of, Conjunction, Evaluate-for, Feature-of, Used-for, and Hyponym-Of). These seven relationship types include two symmetrical relationship types (Conjunction and Compare) and five directional relationship types. The model specifies key parameters, as shown in Table 2.

[0061]

[0062] Table 2

[0063] Where Num_hidden_layer is the number of layers of the encoder and decoder, Num_attention_head is the number of heads in the multi-head self-attention layer, Hidden_size is the size of the hidden layer, FF Hidden_size is the size of the network hidden layer, Layer_norm_epsilon is the normalized error value, Hidden_dropout is the loss ratio of the hidden layer, and Attention_dropout is the loss ratio of the attention mechanism.

[0064] After the data set is input into the generative language model, the corresponding output content is obtained, which is the experimental result of the experiment set in this embodiment, as shown in Table 3:

[0065]

[0066]

[0067] Table 3

[0068] The experimental results in the embodiments of the present application selected the information extraction results of multiple types of models for comparison, mainly comparing the precision and recall of information extraction, as well as the evaluation index F1 that takes both into account. The evaluation criteria are: in the named entity recognition task, if the predicted entity boundary and the predicted entity type are both correct, the predicted entity is considered to be correct; in the relationship extraction task, if the predicted boundaries of the two entities and the predicted relationship type are both correct, the predicted relationship is considered to be correct. In the multi-relationship extraction task, if the boundaries of the four entities are correct and the relationship types of the entities are correct, the predicted multi-relationship is considered to be correct. According to Table 3, the generative language model trained in this application achieved better results in named entity recognition tasks and linear relationship extraction tasks. The TANL model in Table 3 is similar to the generative language model, which converts information extraction into a text generation task. However, since the TANL model generates content by adding natural semantically enhanced labels to the complete input document, its ability to extract relationships in longer texts is poor. The generative language model uses a designed linear structure encoding to process the data before using it as an input sequence, and the output sequence is also output based on the linear structure, which can shorten the distance between entities and thus handle the relationship between long-distance entities.

[0069] At the same time, in the experiments of the embodiments of the present application, in order to verify the influence of the symbol pairs used in the linear structure coding on the information extraction effect, and based on the comparison results obtained by the prompt template between the digital labels and the semantic labels, as shown in Table 4, random digital labels are used to replace the original labels with semantic information for experiments.

[0070]

[0071] Table 4

[0072] After the experiment is completed, the data efficiency comparison of the template using digital labels and semantic labels is obtained, as shown in Table 5, and the full test set is used for evaluation based on different proportions of training data (1%, 10%, 20%, 50%, 100%), as shown in Table 5. Figure 12 As shown in Figure 2, data comparisons of various models at different scales are obtained. The experimental results show that by encoding labels as natural language words, knowledge can be effectively transferred from pre-trained language models.

[0073]

[0074] Table 5

[0075] like Figure 13 As shown, the embodiment of the present application also proposes an information extraction device based on a linear structure, including:

[0076] at least one processor; and,

[0077] a memory communicatively connected to the at least one processor; wherein,

[0078] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a linear structure-based information extraction method as described in any of the above embodiments.

[0079] An embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be: the linear structure-based information extraction method described in any of the above embodiments.

[0080] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0081] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0082] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0087] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0088] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0090] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A linear structure-based information extraction method, characterized in that: include: Design a linear structure code to jointly represent entity information and entity relationships, and set a prompt template to guide the pre-trained language model to generate information corresponding to the linear structure code through the prompt template, specifically including: Based on the structure of binary and triplet, a linear structure encoding is designed to represent entity and relationship information. The format of the linear structure encoding is: ((first entity type: first entity name (first relationship type: first relationship) (second relationship type: second relationship)) (second entity type: second entity name)); Setting a prompt template, and encoding the linear structure by the prompt template to guide the pre-trained language model to generate linear structure information by the prompt template, wherein the prompt template includes an EntityType entity type field and a RelationType relationship type field; Training the pre-trained language model according to the prompt template to obtain a generative language model; In the training of the pre-trained language model, an attention residual structure is used, and the residual block in the attention residual structure is placed in the self-attention head of the multi-head attention layer of the pre-trained language model to achieve training optimization of the pre-trained model; Acquire natural language data, obtain a corresponding input sequence based on the linear structure encoding in the generative language model, encode the input sequence through an encoder, and decode it through a decoder according to the input order of the input sequence to obtain a first prediction probability of a target vocabulary output by the decoder, obtain a second prediction probability of target information based on the first prediction probability, and obtain the target information based on the second prediction probability.

2. The method according to claim 1, characterized in that Before designing the linear structure code to jointly represent entity information and entity relationships, and setting a prompt template to guide the pre-trained language model to generate information corresponding to the linear structure code through the prompt template, the method further includes: Collecting natural language data, and constructing a standard dataset based on the natural language data using a dataset construction method; Based on the standard data set, a natural language inference model is pre-trained to obtain a pre-trained language model, and based on the standard data set, a source text sequence is output.

3. The method according to claim 2, characterized in that Pre-training the natural language inference model based on the standard data set to obtain a pre-trained language model specifically includes: Inputting the standard data set into a natural language inference model, calculating predicted probabilities of relationships between entities in the standard data set through a decoding layer in the natural language inference model, filtering out relationships whose predicted probabilities are lower than a preset probability value, and training the natural language inference model based on the filtered standard data set; After the training is completed, negative sample data is added, and based on the negative sample data, the trained natural language inference model is retrained. After the training is completed, a pre-trained language model is obtained.

4. The method according to claim 1, wherein In the training of the pre-trained language model, an attention residual structure is used, and the residual block in the attention residual structure is placed in the self-attention head of the multi-head attention layer of the pre-trained language model to achieve training optimization of the pre-trained model, specifically including: In the pre-trained language model, an attention residual structure is used, residual connections are performed based on the residual blocks in the attention residual structure, and layer normalization is performed to achieve gradient stability between model layers during the training process of the pre-trained language model; At the same time, the residual block is placed in the self-attention head of the multi-head attention layer of the pre-trained language model for residual connection, and based on the attention score obtained by the encoder layer, the probability distribution of the attention score is calculated through a normalization function and output through the multi-head attention layer to achieve training optimization of the pre-trained model.

5. The method according to claim 4, characterized in that The attention scores obtained based on the encoder layer are calculated through a normalization function to obtain a probability distribution of the attention scores, which is output through the multi-head attention layer, specifically including: The multi-head attention layer calculates the correlation between each position in the input sequence and other positions, obtains the correlation value of the position, and obtains the corresponding attention weight based on the correlation value, thereby obtaining the corresponding attention score; The attention scores are processed using a Softmax function to obtain a probability distribution of the input sequence, and the probability distribution is output through the multi-head attention layer.

6. The method according to claim 1, characterized in that The acquiring of natural language data, encoding the corresponding input sequence based on the linear structure in the generative language model, encoding the input sequence by an encoder, and decoding the input sequence by a decoder according to the input order of the input sequence, and obtaining a first prediction probability of a target vocabulary output by the decoder, specifically includes: Acquiring natural language data, preprocessing it using the generative language model to obtain a source text sequence corresponding to the natural language data, and performing linear structured processing on the source text sequence based on the prompt template to obtain the corresponding input sequence; The input sequence is input into an encoder, encoded by the encoder, and decoded by a decoder according to the input order of the input sequence, to obtain a first prediction probability of a target vocabulary output by the decoder.

7. The method according to claim 6, characterized in that The encoding by the encoder and the decoding by the decoder according to the input order of the input sequence to obtain the first prediction probability of the target vocabulary output by the decoder specifically include: Encode the input sequence by the encoder and calculate a hidden vector of the input sequence; Based on the hidden vector, a decoder is used to decode the input sequence according to the input order of the input sequence, and an output structure of the input sequence is predicted to obtain a first prediction probability of the target vocabulary.

8. An information extraction device based on linear structure, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the linear structure-based information extraction method as described in any one of claims 1 to 7.

9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured as: the linear structure-based information extraction method according to any one of claims 1 to 7.

Images