An entity relationship joint extraction method and system of a government affair service convenient hotline

Abstract

The application provides an entity relationship joint extraction method and system of a government affair service convenient hotline, and regards the government affair service convenient hotline case entity relationship extraction as a triple extraction problem, that is, first, a head entity is extracted, then a tail entity and a relationship are extracted through head entity related information, specifically, first, a target text of specific content in the hotline case is coded, then the head entity is extracted through a single-layer pointer network coding mode, then the tail entity and the relationship are extracted through a multi-layer pointer network coding mode combined with context coding information and head entity position coding information. The method can improve the accurate analysis and processing capability of the government affair service convenient hotline data, and better meet the intelligent demand of the government affair service convenient hotline.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to an entity relationship joint extraction method and system for a government service convenience hotline. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present application and do not necessarily constitute prior art.

[0003] With the sharp increase in government service convenience hotline data, accurate analysis and processing of data are imminent. At present, as an important channel for responding to people's livelihood appeals, the intelligentization of the government service convenience hotline is even more important. The intelligentization of the government service convenience hotline requires the ability to accurately analyze and process data, and the knowledge graph has advantages in these two aspects, but before constructing the government service convenience hotline knowledge graph, the non-structured hotline data needs to be subjected to knowledge extraction operation, and in this process, the joint extraction method of entity relationship needs to be used to extract the entity and relationship in the city hotline data, and store the extracted entity and relationship to form a knowledge graph, so as to achieve accurate analysis and processing of city hotline data.

[0004] At present, the entity relationship extraction method is divided into pipeline method and joint extraction method. The pipeline entity relationship extraction method first identifies the entity, and then classifies the relationship between the entities, such as the method combining conditional random field and convolutional neural network to extract entities and then extract relationships, but this method also has limitations, mainly having the following three shortcomings: 1) entity redundancy: the entities extracted by entity recognition are not necessarily related when paired, which increases the computational complexity and increases the error rate; 2) interaction missing: there is a connection and dependency relationship between entity recognition and relationship classification; 3) error propagation: the errors of entity recognition will affect the performance of relationship classification.

[0005] The joint extraction method combines the two tasks of entity recognition and relationship classification, which can effectively solve the problem of relationship overlap by sharing parameters for joint learning, but most of them use the heavyweight pre-training language model BERT to encode the feature of the text data to be extracted, and the BERT model has a large number of parameters, which consumes a lot of memory during training, which is not conducive to the training speed and stability of the model; at the same time, the NSP task in the BERT model has a low accuracy rate in completing the downstream task of multiple sentence input. SUMMARY

[0006] In order to solve the problems in the prior art, the application provides an entity relationship joint extraction method and system for a government service convenient hotline, and the entity relationship extraction of a government service convenient hotline case is regarded as a triple extraction problem, that is, a head entity is extracted first, and then a tail entity and a relationship are extracted through head entity related information, specifically, a pre-training language model ALBERT is used to encode the target text of the specific content in the hotline case, a single-layer pointer network coding mode is used to extract the head entity, context coding information and head entity position coding information are combined, and finally a multi-layer pointer network coding mode is used to extract the tail entity and the relationship. The method effectively reduces the parameters in the model, reduces the memory cost during training, and effectively improves the training speed and stability of the model. Meanwhile, the hotline data coding result is shared by the head entity extraction, tail entity and relationship extraction tasks, which can effectively alleviate the error propagation problem caused by the non-joint extraction mode.

[0007] In order to achieve the above object, the application adopts the following technical scheme:

[0008] The first aspect of the application provides an entity relationship joint extraction method for a government service convenient hotline.

[0009] An entity relationship joint extraction method for a government service convenient hotline comprises the following steps:

[0010] Obtain government service convenient hotline text data and input it into a pre-training language model ALBERT;

[0011] The pre-training language model ALBERT extracts features from the input data, mines deep semantic features, and obtains coding vectors with rich context information;

[0012] The coding vectors are extracted to extract the head entity, and then the tail entity and the relationship are extracted according to the relationship category, and finally output in the form of triplets;

[0013] The pre-training language model ALBERT is composed of an embedding layer and a Transformer layer, the embedding layer maps each character into a word vector through a dictionary, and inputs it into the concatenated Transformer layer, captures the information at the syntax and semantic levels through pre-training, encodes the language knowledge contained in the text into the Transformer encoder in the form of parameters.

[0014] Further, the obtained government service convenient hotline data is unstructured, and there is a large amount of repeated data and part of invalid data, so the data needs to be de-duplicated and screened first, and then the data is labeled, the head entity position and the head entity label are marked in the first step, and the tail entity position and the corresponding relationship are marked in the second step.

[0015] Further, the pre-trained language model ALBERT is used to encode the text data to be extracted, and the encoding result is shared by the head entity extraction, tail entity and relation extraction.

[0016] Further, the triple extraction layer comprises a head entity extraction layer and a tail entity and relation joint extraction layer.

[0017] Further, the head entity extraction layer uses a pointer network decoding method to extract the head entity, specifically:

[0018] The label of the head entity is divided into a start label and an end label.

[0019] The encoding vector output by the Transformer encoder layer is input into two identical binary pointer labelers to predict the start position and end position of the head entity in the sentence, and the head entity is obtained.

[0020] The entity type of the head entity is marked.

[0021] Further, the start position and end position of the head entity corresponding to the label are determined by 0 / H pointer: in the start label of the head entity, H represents the start position of the head entity in the sentence; in the end label of the head entity, H represents the end position of the head entity in the sentence.

[0022] Further, in the tail entity and relation joint extraction layer, the tail entity is obtained by combining the position information, entity type of the head entity and the prediction of the tail entity position information.

[0023] The second aspect of the application provides an entity and relation joint extraction system for a government service and convenient hotline.

[0024] An entity and relation joint extraction system for a government service and convenient hotline comprises:

[0025] An input module is configured to obtain government service and convenient hotline text data and input the data into a pre-trained language model ALBERT.

[0026] An encoding module is configured to perform feature extraction on the input data by the pre-trained language model ALBERT, mine deep semantic features, and obtain encoding vectors with rich context information.

[0027] A triple extraction module is configured to extract the head entity from the encoding vectors, extract the tail entity and relation according to the relation category, and finally output in the form of triple.

[0028] The pre-training language model ALBERT is composed of an embedding layer and a Transformer layer, the embedding layer maps each character into a word vector through a dictionary, and is input into the concatenated Transformer layer, and through pre-training, the information at the syntax and semantic levels is captured, and the language knowledge contained in the text is encoded into the Transformer encoder in the form of parameters.

[0029] The third aspect of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the steps of the entity relationship joint extraction method of the government service convenient hotline according to the first aspect of the present application.

[0030] The fourth aspect of the present application provides an electronic device, which includes a memory, a processor and a program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the entity relationship joint extraction method of the government service convenient hotline according to the first aspect of the present application.

[0031] Compared with the prior art, the beneficial effects of the present application are:

[0032] 1、The method, system, medium or electronic device of the present application adopts a pre-training language model ALBERT instead of a heavy pre-training language model BERT to perform feature coding on the text data to be extracted, effectively reduces the parameters in the BERT model through embedding layer parameter factorization and cross-layer parameter sharing, greatly reduces the memory cost during training, and effectively improves the training speed and stability of the model; at the same time, the SOP task is used to replace the NSP task in BERT, which improves the accuracy of the downstream task of multiple sentence input.

[0033] 2、The method, system, medium or electronic device of the present application, the hot line data coding result obtained by the ALBERT pre-training language model is shared by the head entity extraction, tail entity and relationship extraction tasks, which can effectively alleviate the error propagation problem caused by the non-joint extraction method. BRIEF DESCRIPTION OF DRAWINGS

[0034] The drawings accompanying the specification of the present application form a part of the present application and serve to provide a further understanding of the present application, the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application.

[0035] Figure 1 The flowchart of the entity relationship joint extraction method of the government service convenient hotline provided by the embodiment 1 of the present application.

[0036] Figure 2 The entity relationship joint extraction model diagram of the government service convenient hotline provided by the embodiment 1 of the present application.

[0037] Figure 3 is a schematic diagram of an ALBERT model structure provided by Embodiment 1 of the present application.

[0038] Figure 4 is a feature embedding representation diagram of an ALBERT embedding layer provided by Embodiment 1 of the present application.

[0039] Figure 5 is a schematic diagram of a Transformer encoding structure provided by Embodiment 1 of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application pertains.

[0042] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component, and / or combinations thereof.

[0043] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0044] Embodiment 1

[0045] As shown in Figure 1 Embodiment 1 of the present application provides an entity relationship joint extraction method for a government service convenience hotline, comprising the following steps:

[0046] First, in the input layer, the government service convenience hotline text data is obtained, the hotline data obtained is subjected to a data cleaning operation, the hotline text is extracted and labeled, and then input into the pre-training language model ALBERT;

[0047] Then, in the encoding layer, the pre-training language model ALBERT is used to extract features from the input sentence input in the input layer, to mine deep semantic features and obtain encoding vectors with rich context information;

[0048] Finally, in the triple extraction layer, the pointer network decoding method is used to extract the head entity from the encoding vector, and the tail entity and the relationship are extracted according to the relationship category, and finally output in the form of triplets.

[0049] In detail, the following is included:

[0050] As shown in the accompanying Figure 1 The process of the entity and relation joint extraction method of the government service hotline is shown in the accompanying

[0051] 1. Input layer: The hotline data obtained is subjected to data cleaning operation, and then the hotline text is extracted and labeled.

[0052] First, the government service hotline data of Jinan in 2020 is obtained; then the hotline data obtained is subjected to data cleaning operation to obtain appropriate hotline text information; then the labeling rules of entities and relations are set, and the hotline text is labeled according to the set rules. The specific steps are to mark the head entity and the head entity label first, and then mark the tail entity corresponding to each head entity and the corresponding relation; finally, the labeled data is divided into training set, test set and validation set for subsequent model training.

[0053] In the first part of data cleaning, the government service hotline data obtained is unstructured, and there are a lot of repeated hotlines in the data, so the repeated data needs to be removed to reduce the redundancy of the data; then there are some invalid data in the government service hotline data, which needs to be selected. The effective data in the data is filtered out. After the above de-duplication and filtering operation, the data is labeled, the first step is to mark the head entity position and the head entity label, and the second step is to mark the tail entity position and the corresponding relation.

[0054] As shown in the accompanying Figure 2 First, an empty matrix of size 2*N is generated in the first step, where N is the length of the text to be extracted. In the first dimension, the start and end positions of the entity in the sentence are marked as H, and the remaining positions are marked as "0", i.e. in "Mr. Yang", the position of "Yang" is marked as H, which represents the "citizen" entity type; in the second dimension, the end position of the entity in the sentence is marked as H, and the remaining positions are marked as "0". As shown in the accompanying Figure 3As shown, the position of "sheng" in "Yang Mr." is marked as H. In the second step, the relationship type is marked as G1, G2, G1 and G2 respectively represent the address of the hotline event and the content of the hotline. The position of "Ji" in "Jinan City Sixth People's Hospital" is marked as G1, and the position of "Yuan" in "Jinan City Sixth People's Hospital" is marked as G1. For the relationship type G2, the position of "fa" in "found a viral patient" is marked as G2, and the position of "zhe" in "found a viral patient" is marked as G2.

[0055] 2. Encoding layer: using ALBERT pre-training language model to extract features of the input sentence input by the input layer, to mine deep semantic features and get encoding vectors with rich context information.

[0056] As shown in Figure 2 , first, the original text sentence is represented by sequence Y = (y1, y2,..., y n ), where y1 to y n represent the positions of the first to the nth word in the sentence, and n represents the length of the hotline case text. Then, the ALBERT pre-training language model is used instead of the heavy pre-training language model (Bidirectional Encoder Representations from Transformers, BERT) to encode the text data to be extracted. The ALEBRT model improves the BERT model in the following two points: 1) The ALBERT model effectively reduces the parameters in the BERT model through embedding layer parameter factorization (Factorized Embedding Parameterization) and cross-layer parameter sharing (Cross-layer Parameter Sharing), greatly reducing the memory cost during training, and effectively improving the training speed and stability of the model. 2) ALBERT proposes a sentence order prediction (Sentence Order Prediction, SOP) task to replace the next sentence prediction (Next Sentence Prediction, NSP) task. In order to make up for the NSP task in BERT proposed by Yang et al., the effect of the multi-sentence input downstream task is improved. The core of the SOP task is to evaluate the coherence between sentences and produce an inter-sentence coherence loss. For multi-sentence input downstream tasks, SOP can improve the accuracy by about 2% compared to NSP. At the same time, the encoding results of the hotline data by the ALBERT pre-training language model are shared by the head entity extraction, tail entity and relationship extraction tasks. Among them, the joint extraction method with shared encoding parameters can alleviate the error propagation problem caused by the non-joint extraction method.

[0057] As shown in Figure 3 , the model structure of ALBERT is composed of embedding layer and Transformer layer. The embedding layer maps each character into a word vector through a dictionary, and inputs into the concatenated Transformer layer. Through pre-training, the information at the syntax and semantic levels is captured to encode the language knowledge contained in the text into the Transformer encoder in the form of parameters. The steps of pre-training are: determining the pre-trained model to be loaded and installing the dependent package; loading the tokenizer of the pre-trained model; loading the pre-trained model with header; using the model to obtain the result. Pre-training is mainly to adjust the model parameters in advance to reduce the time for subsequent model training. The size of the output result is: (torch.Size([1, 25]), torch.Size([1, 25])).

[0058] As shown in Figure 4 , the input of the ALBERT embedding layer is the linear sequence of words, which supports single sentence text and sentence pair text. The beginning of the sentence is represented by the symbol [CLS], and the end of the sentence uses the symbol [SEP] to separate the two input sentences. The three components of the ALBERT embedding layer have different functions, among which the word block embedding is to convert Chinese characters into word vectors, the position embedding is to encode the position information of the characters into feature vectors, and the sentence segment embedding is used to distinguish the two sentences.

[0059] As shown in Figure 5 , the Transformer model was first proposed by the Google team in 2017, which is a Seq2Seq (Sequence to Sequence) model based on Self-Attention. The Transformer layer is mainly composed of multiple Encoders and Decoders. The Transformer encoding structure is stacked by multiple identical network layer structures. Each network layer fuses the position information of the words and characters by combining the "multi-head self-attention mechanism layer" and the "feedforward network layer" two sub-network layers. The "Add&Norm" layer adds the input and output of the network layer and performs normalization processing, and then uses residual connection between the two sub-network layers.

[0060] Among them, the Encoder is stacked by N identical network layers, mainly including four layers: the first layer is a multi-head attention mechanism layer; the second layer is a residual network layer; the third layer is a common feed forward neural network layer for integrating the position information of words; and the fourth layer is an Add&Norm layer for adding the input and output of the layer and performing normalization processing. The Decoder adds an Encoder-Decoder-attention layer on the basis of the Encoder to help the current node obtain the key content to be focused on.

[0061] In the Transformer encoder, the most important module is the multi-head attention mechanism, and the calculation formula is as shown below.

[0062] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W o (1)

[0063] head i =Attention(QW i Q ,KW i K ,VW i V ) (2)

[0064]

[0065]

[0066] Among them, W o is an additional weight matrix, which can compress the dimension of the concatenated matrix to the size of the sequence length; Q, K, and V represent the query, key, and value vectors of each word in the input sequence, respectively; W i Q , W i K , and W i V are the weight matrices of Q, K, and V, respectively; d k represents the dimension of the query and key vectors of each word; Softmax(·) is a normalization activation function, and z represents an N-dimensional row vector.

[0067] 3. Triple extraction layer: the encoded vector is first extracted by using a pointer network decoding method to extract the head entity, then the tail entity is extracted according to the pre-labeled relationship category, and finally output in the form of a triple.

[0068] Output as: triple format such as

Head entity, relationship, tail entity

Mr. Yang, hotline incident address, the sixth people's hospital

[0069] 3.1 Head entity extraction layer

[0070] As shown in Figure 2 , the label of the head entity is divided into start label and end label, which is used to determine the starting position of the head entity. In order to enhance the dependence of the model and alleviate the problem of gradient explosion, the characteristics of the head entity need to be considered when extracting the head entity. In this paper, the head entity features are used as conditions to adopt the layer normalization LN method to more effectively fuse the head entity features and sentence vectors. The head entity features are according to the entity type of the head entity, and the sentence vector is the vector form of the sentence where the entity is located. The calculation method is shown in formulas (5) and (6).

[0071]

[0072]

[0073] Wherein, is the corresponding ith input feature in the input vector H0, μ is the mean, σ is the variance, ε is a small constant greater than 0, and α and β are training parameters.

[0074] Subsequently, the output vector of the Transformer encoder layer is input into two identical binary pointer markers to predict the start and end positions of the head entity in the sentence, respectively. The 0 / H pointer determines whether the label corresponds to the start and end position of the head entity. In the start label of the head entity, H represents the start position of the head entity in the sentence; similarly, H in the end label represents the end position of the head entity in the sentence. The specific operation is shown in formulas (7) and (8).

[0075]

[0076]

[0077] Wherein, is the ith input feature of the sentence encoding H1, and respectively represent the probability that the character label at position i is the start and end position of the head entity. If the probability exceeds a certain value, the position is marked as H, otherwise it is marked as 0. W s_start and W s_end are training weights, b s_start and b s_endis the bias vector, S(·) is the sigmoid activation function.

[0078] Finally, the entity type of the head entity needs to be labeled. The entity types of the government service convenience hotline text are diverse, and different entity types correspond to various relationship types. Therefore, the entity type needs to be labeled, as shown in equation (9) and equation (10).

[0079]

[0080]

[0081] wherein, label start (z i ) and label end (z i ) represent the label information of character z i , which respectively represent the start and end of the head entity type; Q represents the set threshold value. When label start = label end , it means that the corresponding text content of this interval is the labeled head entity. As shown in equation (9), the labeled head entity is “Mr. Yang”, and the corresponding entity type is “citizen”. Figure 2

[0082] 3.2 Tail entity relationship joint extraction layer

[0083] Tail entity annotation is used to mark the position, entity type and relationship category of the head entity and the corresponding tail entity.

[0084] In the tail entity relationship joint extraction layer, the labeled information of the head entity is combined to extract the tail entity and the relationship between the head entity and the tail entity. In the tail entity relationship joint extraction layer, the labeled information of the head entity is combined to determine the relationship category related to the head entity, then the tail entity is determined according to the relationship category, and finally the relationship between the head entity and the tail entity is determined by combining the head and tail entity information, to complete the joint extraction of the tail entity and the relationship. Therefore, in order to strengthen the connection between layers, the CLN network layer is embedded to encode the annotation of the head entity E j and the sentence encoding H0 to obtain a new sentence encoding H2, as shown in equation (11) and equation (12):

[0085]

[0086]

[0087] wherein, and ​For the input vector H0 and H2 corresponding to the i-th input feature of the input feature, μ is the mean, σ is the variance, ε is a minimum constant greater than 0, and α and β are training parameters, w α and w β is an initialized transformation matrix.

[0088] The preprocessing operation of the data is completed before the data is extracted, and the entities and relations in the data are labeled. After the head entity is extracted, the tail entity is extracted according to the relation type.

[0089] The tail entity is labeled by using the pointer labeling method. First, a plurality of relations are predefined, and two binary pointer labelers are established under each relation. The actual operation is similar to the head entity labeling, and the difference is that the input vector is changed to the sentence vector CLN fused with the head entity feature. The detailed operation under a specific relation is shown in formula (13) and formula (14).

[0090]

[0091]

[0092] wherein, is the i-th input feature of the sentence encoding H2, and respectively represent the probability that the character mark at position i is the start position and the end position of the head entity. If the probability exceeds a certain value, the position is marked as H, otherwise it is marked as 0. W o_start and W o_end are training weights, b o_start and b o_end are bias vectors, and S(·) is a sigmoid activation function.

[0093] Finally, the entity type of the tail entity also needs to be labeled. The tail entity can be obtained by combining the position information of the head entity, the entity type and the prediction of the position information of the tail entity. All entity types are labeled before data extraction. After the head entity is extracted, the position of the tail entity is determined and extracted by searching the tail entity related to the head entity. The relationship between the head entity and the tail entity has been labeled before extraction and stored in the dictionary. When extracting, the entity appearing in the dictionary can be used to determine. As shown in formula (15), the tail entity is “Jinan Sixth People's Hospital”, and the relationship type between the head entity and the tail entity is “hotline event address”. Figure 2

[0094] Embodiment 2

[0095] The embodiment 2 of the application provides an entity relation joint extraction system for a government service convenience hotline, which comprises:

[0096] ​The input module is configured to: obtain government service convenience hotline text data, perform data cleaning operation on the obtained hotline data, extract hotline text and perform labeling, and then input into a pre-training language model ALBERT;

[0097] The encoding module is configured to: the pre-training language model ALBERT extracts features from input data, mines deep semantic features, and obtains encoding vectors rich in context information.

[0098] The triple extraction module is configured to: use a pointer network decoding method to extract head entities from the encoding vectors, then extract tail entities according to the relationship categories, and finally output in the form of triples.

[0099] The pre-training language model ALBERT is composed of an embedding layer and a Transformer layer, the embedding layer maps each character into a word vector through a dictionary, and inputs into a concatenated Transformer layer, captures information at the syntax and semantic levels through pre-training, encodes language knowledge contained in the text into the Transformer encoder in the form of parameters.

[0100] The working method of the system is the same as the entity relationship joint extraction method of the government service convenience hotline provided in Embodiment 1, and will not be repeated here.

[0101] Embodiment 3

[0102] Embodiment 3 of the present application provides a computer readable storage medium having a program stored thereon, which is executed by a processor to implement the steps in the entity relationship joint extraction method of the government service convenience hotline as described in Embodiment 1 of the present application, the steps being:

[0103] Obtain government service convenience hotline text data, perform data cleaning operation on the obtained hotline data, extract hotline text and perform labeling, and then input into a pre-training language model ALBERT;

[0104] The pre-training language model ALBERT extracts features from input data, mines deep semantic features, and obtains encoding vectors rich in context information.

[0105] The pointer network decoding method is used to extract head entities from the encoding vectors, then extract tail entities according to the relationship categories, and finally output in the form of triples.

[0106] The pre-training language model ALBERT is composed of an embedding layer and a Transformer layer, the embedding layer maps each character into a word vector through a dictionary, and the word vector is input into the concatenated Transformer layer, the pre-training is used to capture information at the syntax and semantic levels, language knowledge contained in the text is encoded into the Transformer encoder in the form of parameters.

[0107] The detailed steps are the same as the entity relationship joint extraction method of the government service convenient hotline provided in Embodiment 1, and will not be described here.

[0108] Embodiment 4:

[0109] Embodiment 4 of the present application provides an electronic device, comprising a memory, a processor and a program stored in the memory and executable on the processor, wherein the processor executes the program to realize the steps in the entity relationship joint extraction method of the government service convenient hotline as described in the first aspect of the present application, and the steps are:

[0110] The government service convenient hotline text data is acquired, the acquired hotline data is subjected to a data cleaning operation, the hotline text is extracted and labeled, and then input into the pre-training language model ALBERT;

[0111] The pre-training language model ALBERT extracts features from the input data, mines deep semantic features, and obtains encoding vectors with rich context information;

[0112] The pointer network decoding mode is used to extract the head entity from the encoding vectors, then the tail entity is extracted according to the relationship category, and finally the output is in the form of a triple;

[0113] The pre-training language model ALBERT is composed of an embedding layer and a Transformer layer, the embedding layer maps each character into a word vector through a dictionary, and the word vector is input into the concatenated Transformer layer, the pre-training is used to capture information at the syntax and semantic levels, language knowledge contained in the text is encoded into the Transformer encoder in the form of parameters.

[0114] The detailed steps are the same as the entity relationship joint extraction method of the government service convenient hotline provided in Embodiment 1, and will not be described here.

[0115] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer usable program code.

[0116] The embodiments of methods, apparatuses (systems) and computer program products according to the present application can be described in the general context of method steps and processes, which can be implemented in one embodiment by a program of instructions being executed in a computer system on a machine such as the computer systems executing the innovation. The program is not limited in scope to this machine; that is, virtually any machine that is associated with any data that can be read and used by this machine is nevertheless within the scope of the present application. Further, it is recognized that the innovations can be implemented together with other methods or as independent or separate methods. Figure 1 one or more functions specified in a flow or multiple flows and / or block or blocks. Figure 1 means for carrying out one or more functions specified in a flow or multiple flows and / or block or blocks.

[0117] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more functions specified in a flow or multiple flows and / or block or blocks. Figure 1 means for carrying out one or more functions specified in a flow or multiple flows and / or block or blocks.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more functions specified in a flow or multiple flows and / or block or blocks. Figure 1 means for carrying out one or more functions specified in a flow or multiple flows and / or block or blocks.

[0119] Those skilled in the art can understand that all or part of the flow of the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the flow of the above-mentioned embodiment method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0120] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for jointly extracting entity relationships for a government service hotline, characterized in that, Includes the following steps: Obtain the text data of the government service hotline and input it into the pre-trained language model ALBERT; The pre-trained language model ALBERT extracts features from the input data, mines deep semantic features, and obtains encoding vectors rich in contextual information. Extract the head entity from the encoded vector, then extract the tail entity and relation according to the relation category, and finally output it in the form of triples; The triplet extraction layer includes a head entity extraction layer and a tail entity relation joint extraction layer. The head entity extraction layer extracts head entities using a pointer network decoding method. Specifically, it uses head entity features as conditions and employs a layered normalization (LN) method to fuse the head entity features with the sentence vectors containing the entities, transforming the head entity features into vector form. ; ;in, For input vectors The corresponding number in the middle Each input feature The mean, For variance, It is a very small constant greater than 0. and These are the training parameters; the labels of the head entities are divided into start and end labels; the encoded vector output by the Transformer encoder layer is input into two identical binary pointer labelers to predict the start and end positions of the head entities in the sentence, respectively, thus obtaining the head entities; specific operations: ; ;in, Encoding sentences The Each input feature and Representing positions respectively The character is marked as the probability of the start and end positions of the header entity; if the probability exceeds a certain value, the position is marked as H, otherwise it is marked as 0; and For training weights, and For bias vectors, Use the sigmoid activation function; mark the entity type of the head entity; ; ;in, and Character The tag information, which represents the start and end of the header entity type respectively; Q represents the set threshold; when When this occurs, it indicates that the corresponding text content of this interval is the header entity of the tag; The tail entity relation joint extraction layer combines the labeling information of the head entity to determine the relation category associated with the head entity, then determines the tail entity based on the relation category, and labels the entity type of the tail entity; finally, it combines the head and tail entity information to determine the relationship between the head and tail entities, completing the joint extraction of tail entities and relations; an embedded CLN network layer is used to encode the labeling of the head entity. and sentence encoding Feature fusion is performed to obtain a new sentence code. Calculation method: ; ;in, and For input vectors and The corresponding number in the middle Each input feature The mean, For variance, It is a very small constant greater than 0. and These are training parameters. and Transformation matrix for initialization; annotation of tail entity: ; ;in, Encoding sentences The Each input feature and Representing positions respectively The character is marked as the probability of the start and end positions of the header entity; if the probability exceeds the set value, the position is marked as H, otherwise it is marked as 0; and For training weights, and For bias vectors, It is the sigmoid activation function; The pre-trained language model ALBERT consists of an embedding layer and a Transformer layer. The embedding layer maps each character to a character vector using a dictionary, which is then input into the concatenated Transformer layer. Through pre-training, it captures grammatical and semantic information and encodes the language knowledge contained in the text into the Transformer encoder in the form of parameters.

2. The method for jointly extracting entity relationships for government service hotlines as described in claim 1, characterized in that, The data obtained from the government service hotline is unstructured, containing a large amount of duplicate data and some invalid data. It is necessary to first perform deduplication and filtering on the data, and then label the data. The first step is to mark the head entity position and head entity label, and the second step is to mark the tail entity position and corresponding relationship.

3. The method for jointly extracting entity relationships for government service hotlines as described in claim 1, characterized in that, The pre-trained language model ALBERT is used to encode features of the extracted text data. The encoding results are shared for both head entity extraction, tail entity extraction, and relation extraction tasks.

4. The method for jointly extracting entity relationships for government service hotlines as described in claim 1, characterized in that, The 0 / H pointers determine the start and end positions of the header entity corresponding to the tag: in the start tag of the header entity, H represents the start position of the header entity in the sentence; in the end tag of the header entity, H represents the end position of the header entity in the sentence.

5. The method for jointly extracting entity relationships for government service hotlines as described in claim 1, characterized in that, In the tail entity relation joint extraction layer, the tail entity is obtained by combining the location information of the head entity, the entity type, and the prediction of the tail entity's location information.

6. A system for jointly extracting entity relationships for a government service hotline, characterized in that, include: The input module is configured to: obtain text data from the government service hotline and input it into the pre-trained language model ALBERT; The encoding module is configured to: use the pre-trained language model ALBERT to extract features from the input data, mine deep semantic features, and obtain encoding vectors rich in contextual information; The triple extraction module is configured to extract the head entity from the encoded vector, then extract the tail entity and relation according to the relation category, and finally output it in the form of triples. The triplet extraction module includes a head entity extraction layer and a tail entity relation joint extraction layer. The head entity extraction layer extracts head entities using a pointer network decoding method. Specifically, it uses head entity features as conditions and employs a layered normalization (LN) method to fuse the head entity features with the sentence vectors containing the entities, transforming the head entity features into vector form. ; ;in, For input vectors The corresponding number in the middle Each input feature The mean, For variance, It is a very small constant greater than 0. and These are the training parameters; the labels of the head entities are divided into start and end labels; the encoded vector output by the Transformer encoder layer is input into two identical binary pointer labelers to predict the start and end positions of the head entities in the sentence, respectively, thus obtaining the head entities; specific operations: ; ;in, Encoding sentences The Each input feature and Representing positions respectively The character is marked as the probability of the start and end positions of the header entity; if the probability exceeds a certain value, the position is marked as H, otherwise it is marked as 0; and For training weights, and For bias vectors, Use the sigmoid activation function; mark the entity type of the head entity; ; ;in, and Character The tag information, which represents the start and end of the header entity type respectively; Q represents the set threshold; when When this occurs, it indicates that the corresponding text content of this interval is the header entity of the tag; The tail entity relation joint extraction layer combines the tagging information of the head entity to determine the relation category related to the head entity, then determines the tail entity based on the relation category, and tags the entity type of the tail entity. Finally, the relationship between the head and tail entities is determined by combining the head and tail entity information, thus completing the joint extraction of the tail entity and its relationship. An embedded CLN network layer is used to encode the annotations of the head entities. and sentence encoding Feature fusion is performed to obtain a new sentence code. Calculation method: ; ;in, and For input vectors and The corresponding number in the middle Each input feature The mean, For variance, It is a very small constant greater than 0. and These are training parameters. and Transformation matrix for initialization; annotation of tail entity: ; ;in, Encoding sentences The Each input feature and Representing positions respectively The character is marked as the probability of the start and end positions of the header entity; if the probability exceeds the set value, the position is marked as H, otherwise it is marked as 0; and For training weights, and For bias vectors, It is the sigmoid activation function; The pre-trained language model ALBERT consists of an embedding layer and a Transformer layer. The embedding layer maps each character to a character vector using a dictionary, which is then input into the concatenated Transformer layer. Through pre-training, it captures grammatical and semantic information and encodes the language knowledge contained in the text into the Transformer encoder in the form of parameters.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the entity relationship joint extraction method for the government service hotline as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the entity relationship joint extraction method for government service hotlines as described in any one of claims 1-5.

Images