Diversity and ambiguity semantic recognition and connection method based on sequence labeling

Abstract

The invention relates to a natural language data processing technology, in particular to a diversity and ambiguity semantic recognition and connection method based on sequence labeling. The method comprises the following steps: firstly, representing position boundary information of words in an entity by adopting a BIESO tag; entity categories, relation category labels and diversity labels are added on the basis of entity position boundary labels so as to reflect the relation between entities. According to the method, a joint extraction task for constructing the electric power knowledge graph is converted into a sequence labeling problem, meanwhile, a sequence labeling strategy considering the complex entity relation overlapping condition is provided, the overlapping relation extraction problem in traditional sequence labeling is solved by adding entity relation categories and enriching entity roles, and the method has the advantages of being high in practicability and high in practicability. Accurate prediction of entities and relation types is achieved, the extraction accuracy is effectively improved, and an accurate and professional electric power knowledge graph can be built more easily.

Description

Technical Field

[0001] The present invention relates to natural language data processing technology, and in particular to a semantic recognition and penetration method based on the diversity and ambiguity of sequence annotation. Background Art

[0002] Knowledge graph is a knowledge representation and management method based on semantic network, which can effectively organize, store and query large-scale, multi-source and heterogeneous knowledge data, and provide strong support for deep mining and intelligent analysis of knowledge. Marketing audit problem clue identification technology based on knowledge graph can make up for the technical shortcomings in the field of marketing audit, realize the extraction, identification and connection of marketing audit information, thereby effectively improving the efficiency and accuracy of identifying potential business risks in power marketing, and providing auditors with more accurate problem clues and more reliable decision support. At present, there is little research on the construction method of power marketing audit information knowledge graph at home and abroad. In the process of traditional knowledge graph construction, the overlapping entity relationship recognition model performs poorly in the extraction of marketing audit business entity relationships, and there are problems such as incomplete extracted information and low accuracy; and due to the complex semantics of power marketing professional terms and the uneven distribution of audit text information features, the recognition accuracy of traditional semantic recognition and connection algorithms is low. Semantic recognition and penetration refers to the process of unifying, standardizing and disambiguating entities and relationships of different sources, forms and contexts during the construction of the knowledge graph. For example, different expressions such as "transformer", "transformer equipment" and "transformer device" all refer to the same entity. This process is crucial to improving the quality and usability of the knowledge graph. However, due to the complex semantics of professional terms in the field of power marketing and the uneven distribution of audit text information features, traditional rule-based or statistical semantic recognition and penetration algorithms often cannot meet the needs of audit business. The construction of marketing audit knowledge graphs lacks methodological guidance and best practice experience, resulting in inaccurate knowledge graph construction. Therefore, it is of great practical need to study the construction method of marketing audit knowledge graphs based on overlapping entity relationship extraction and semantic recognition technology.

[0003] In order to solve the problem of insufficient information interaction in the joint extraction model based on shared parameters, related studies often use sequence labeling methods to design a unified decoder. The encoded sentence sequence is directly decoded to obtain sequence labeling information, thereby obtaining relation triplets. The core idea of ​​the joint extraction sequence labeling strategy is to regard the extraction task as a sequence labeling problem, in which each word or subword in the text is regarded as a marker in the sequence, and the extracted entities, relations, or events correspond to the marked labels. The goal is to identify the specified entities from the text and annotate the boundaries and categories of each entity. As an extended sequence labeling strategy, the "BIESO" label can more finely mark entity boundaries and individual tokens; the specific scheme of this label is as follows:

[0004] The position boundary label "B" indicates the beginning of the entity; the position boundary label "I" indicates the middle position of the entity; the position boundary label "E" indicates the end of the entity; the position boundary label "S" indicates a single entity; the position boundary label "O" indicates other types of entities that are not within the extraction range.

[0005] The advantage of this labeling strategy is that it can more accurately represent the entity boundary, so the present invention further improves the "BIESO" labeling strategy. Summary of the invention

[0006] The present invention is based on the overlapping entity relationship extraction technology of sequence annotation, designs a marketing search text information knowledge extraction algorithm, realizes the joint extraction of overlapping entity relationships of search information through the sequence annotation method, and solves the problem of incomplete extraction and low accuracy of power marketing audit information. The specific technical solution is as follows:

[0007] A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation, firstly adopts "BIESO" tags to represent the position boundary information of words in entities; and also includes the following process:

[0008] S01: Based on the entity location boundary labels, entity category labels, relationship category labels and diversity labels are added to reflect the relationship between entities.

[0009] Preferably, S01 is followed by the following process:

[0010] S02: Use the labeled data set to train the BERT model to further improve the model's recognition ability.

[0011] Preferably, S02 is followed by the following process:

[0012] S03: After model training, for the extracted entities and relationships, if there are diverse labels, combined with the cosine similarity calculation formula, the entity vector and similarity label are used as the basis for judgment to integrate the entities.

[0013] Preferably, the entity type tags include: tag "CUS", representing customer; tag "AR", representing region; tag "EM", representing equipment; tag "STA", representing station area; tag "AT", representing acquisition terminal; tag "FEE", representing receivable service fee; tag "OPE", representing service;

[0014] The relationship type labels include: label "PAY", representing payment; label "BP", representing business processing; label "BUY", representing purchase; label "USE", representing use; label "OL", representing overlapping relationship;

[0015] The diversity labels include: label “D”, representing diversity; label “A”, representing ambiguity.

[0016] Preferably, the BERT model training includes the following process:

[0017] The input of the BERT model is divided into three parts: word embedding, paragraph embedding, and position embedding;

[0018] For a given token sequence S = [s1, s2, s3, ..., sn3, sn2] of length n2, add special tokens [CLS] and [SEP] at the beginning and end of the sequence, and convert each token in the sequence into its corresponding word vector ei to obtain the word embedding matrix E;

[0019] Assign a position vector pi to each position in the sequence, and obtain the position embedding matrix P containing all positions;

[0020] Paragraph embedding is used to distinguish two sentences and is used only once in a sequence, denoted as [SEP]. The paragraph embedding vector si[SEP] indicates the beginning of a paragraph. For each token si in the sequence, its input ii is composed of word embedding, position embedding, and paragraph embedding, as shown in the following formula:

[0021]

[0022] In the formula, e i represents the word embedding vector of the i-th word, p i represents the position embedding vector of the i-th word, s i The paragraph embedding vector representing the paragraph to which the i-th word belongs;

[0023] The obtained input representation I = [i1, i2, ..., in1, in] is fed to the Transformer layer of the Bert model, and each token si gets a context-aware vector representation xi:

[0024] X=[x1,x2,…,xn];

[0025] The BERT module was pre-trained using the power marketing audit corpus constructed in S01, and the PowerBERT model for the marketing audit domain was trained in combination with the professional dictionary in the power marketing domain.

[0026] Preferably, during the training process of the PowerBERT model, adversarial training is introduced to introduce small perturbations, i.e. adversarial samples, into the input data to enhance the model's resistance to noise and attacks, thereby improving the model's robustness and generalization ability; the specific process is as follows:

[0027] Based on PGD adversarial training, a gradient accumulation strategy is proposed. In multiple forward propagation and back propagation processes, the model parameters are not updated immediately, but the gradients are accumulated, as shown in the following formula:

[0028]

[0029] Where, the generated adversarial sample is perturbed_input, Π β(input,ε) is a projection operator that constrains the generated adjoint samples to be within a radius of ε near the original input. Input is the original input, α is the step size, is the loss function for a given target, grad (t) is the gradient of the model parameter θ, and grad_accumulator represents the accumulated gradient;

[0030] After K iterations, the average of the accumulated gradients is calculated and the model parameters are updated using the average gradient, as shown in the following formula:

[0031]

[0032] Where avg_grad is the average value of the accumulated gradient, θ(K) is the model parameter at the Kth iteration, and η is the model learning rate;

[0033] The improved GAPGD optimization objective is as follows:

[0034]

[0035] Preferably, the subsequent step further includes the following process:

[0036] The text data is input into the pre-trained PowerBERT module in sentences to obtain high-quality word vectors, and then the BiLSTM layer is used to learn the contextual features of the power text. The BiLSTM layer will capture the contextual dependencies in the sequence through bidirectional processing to further enhance the feature representation.

[0037] Preferably, the PowerRoberta layer of the BiLSTM model converts power text into an embedding vector, and a sequence of words can be represented as:

[0038] X*=[x*1,x*2,…,x*η],

[0039] Where x*t∈Rd is the d-dimensional word vector corresponding to the t-th word in the sentence after being processed by the PowerBERT layer, and η represents the length of the given sentence;

[0040] The update formula of the BiLSTM left-to-right recurrent neural network layer is:

[0041]

[0042] The update formula for the right-to-left recurrent neural network layer is:

[0043]

[0044] Finally, we will and Connected together, it represents the coded information of marketing audit vocabulary t, expressed as The calculation formula for the final output of the BiLSTM neural network is:

[0045]

[0046] Where f and g are activation functions, W, V, and U are weight matrices, b and c are bias matrices, and ht is the state to be activated.

[0047] The present invention transforms the joint extraction task of constructing the electric power knowledge graph into a sequence labeling problem, and proposes a sequence labeling strategy that takes into account the overlapping of complex entity relationships. By adding entity relationship categories and enriching entity roles, the problem of overlapping relationship extraction in traditional sequence labeling is solved, and accurate prediction of entity and relationship types is achieved, which effectively improves the extraction accuracy and is more conducive to building an accurate and professional electric power knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the marking strategy in the embodiment.

[0049] Figure 2 Schematic diagram of the BERT model input part in the embodiment.

[0050] Figure 3 This is a schematic diagram of the PowerBERT layer architecture and input representation in the power marketing audit-specific field in the embodiment.

[0051] Figure 4 2 is a structural diagram of the PowerBERTGAPGDBiLSTMCRF model in the embodiment.

[0052] Figure 5 2 is a structural diagram of the PowerBERT pre-training model in the embodiment. DETAILED DESCRIPTION

[0053] Take a work order reply as an example: "The radio transmission center uses dual-circuit power supply and needs to pay a high-reliability power supply fee." For example, "radio transmission center" and "dual-circuit power supply" have a usage relationship, and at the same time have a payment relationship with "high-reliability power supply fee." However, the traditional sequence labeling strategy cannot solve the problem of relationship overlap.

[0054] To address this problem, the entity types and relationship category labels annotated by the present invention are shown in the following table:

[0055] Entity diversity and ambiguous labels

[0056]

[0057] The labels defined in the present invention are composed of four parts: "entity position entity type relationship category entity role". First, the "BIESO" label is used to represent the position boundary information of words in the entity. At the same time, entity category and relationship category labels are added on the basis of the entity position boundary label to reflect the relationship between entities. In addition, a new label type "Overlapping" abbreviated as "OL" is defined to mark overlapping relationships, and match them according to the corresponding overlapping entities and relationships. If the entity is the head entity (Head) in the triple, its entity role is represented as "H", and if it is the tail entity (Tail), it is represented as "T". In addition, non-entity labels are denoted as "O". Therefore, the label of the overlapping situation can be composed of "entity position entity type OL entity role" to solve the problem of extracting overlapping relationships in sequence labeling. An example of labeling strategy is as follows Figure 1 shown.

[0058] In this embodiment, "Radio Transmitting Center" belongs to both "Use" and "Payment" relationships, that is, there is an overlapping relationship problem, so its relationship category label is expressed as an OL overlapping relationship label, where "HH" means that "Radio Transmitting Center" is both the head entity of the "Use" (USE) relationship and the head entity of the "Payment" (PAY) relationship.

[0059] PowerBERT encoding layer construction in the exclusive field of power marketing auditing: The BERT model uses the MaskedLanguageModel strategy for pre-training. During the training process, BERT randomly masks part of the vocabulary in the input text, and then requires the model to predict these masked words based on the context. This masking strategy enables the model to learn the semantic information of vocabulary from the bidirectional context, so as to better understand the meaning of words in different contexts. At the same time, BERT also learns sentence-level relationships through the NextSentencePrediction task. The training goal is to determine whether two paragraphs of text are continuous sentences, so that BERT can show strong semantic understanding and reasoning capabilities in downstream tasks.

[0060] The input of the BERT model is divided into three parts: word embedding, paragraph embedding, and position embedding, such as Figure 2As shown. For a given tag sequence S = [s1, s2, s3, …, sn3, sn2] of length n2, special tags [CLS] and [SEP] are added at the beginning and end of the sequence, and each tag in the sequence is converted into its corresponding word vector ei to obtain the word embedding matrix E. A position vector pi is assigned to each position in the sequence to obtain the position embedding matrix P containing all positions. Paragraph embedding is used to distinguish two sentences and is usually used only once in a sequence, denoted as [SEP]. The paragraph embedding vector si[SEP] indicates the beginning of the paragraph. For each tag si in the sequence, its input ii is composed of word embedding, position embedding and paragraph embedding, as shown in the following formula

[0061]

[0062] In the formula, e i represents the word embedding vector of the i-th word, p i represents the position embedding vector of the i-th word, s i The paragraph embedding vector representing the paragraph to which the i-th word belongs;

[0063] The obtained input representation I = [i1, i2, …, in1, in] is fed to the Transformer layer of the Bert model, and each token si gets a context-aware vector representation xi: X = [x1, x2, …, xn].

[0064] The BERT module was pre-trained using the power marketing audit corpus, and the PowerBERT model for the marketing audit domain was trained using a professional dictionary in the power marketing domain. The model architecture and input representation are as follows: Figure 3 shown.

[0065] By analyzing the text data of various policy documents such as power market data, power usage data, power price implementation policies, and business environment policies, this paper constructs the PowerBERT model in the exclusive field of marketing auditing, and introduces adversarial training in the extraction process to further improve the accuracy and robustness of power entity relationship extraction. On this basis, the PowerBERTGAPGDBiLSTMCRF combination model is constructed to achieve accurate extraction of global features of Chinese marketing auditing texts. The model structure diagram is shown in the figure below. Figure 4 shown.

[0066] During the BERT training process, adversarial training is introduced to introduce small perturbations (adversarial samples) into the input data to enhance the model's resistance to noise and attacks, thereby improving the model's robustness and generalization ability. PGD (Projected Gradient Descent) is a gradient-based adversarial sample generation method. At each iteration, the input sample x is perturbed along the gradient direction of the loss function L(x,θ), and the perturbation r is generated according to the gradient direction. Since each iteration requires forward propagation and backpropagation, the computational cost is significantly increased.

[0067] To this end, a gradient accumulation strategy is proposed based on PGD adversarial training. In multiple forward propagation and back propagation processes, the model parameters are not updated immediately, but the gradients are accumulated, as shown in the following formula:

[0068]

[0069] Where, the generated adversarial sample is perturbed_input, Π β(input,ε) is a projection operator that constrains the generated adjoint samples to be within a radius of ε near the original input. Input is the original input, α is the step size, is the loss function for a given target, grad (t) is the gradient of the model parameter θ, and grad_accumulator represents the accumulated gradient;

[0070] After K iterations, the average of the accumulated gradients is calculated and the model parameters are updated using the average gradient, as shown in the following formula:

[0071]

[0072] Where avg_grad is the average value of the accumulated gradient, θ(K) is the model parameter at the Kth iteration, and η is the model learning rate;

[0073] The improved GAPGD optimization objective is as follows:

[0074]

[0075] The model can learn from multiple different norm constraints. Compared with the PGD algorithm with a single norm constraint, the generalization error is smaller, which enhances the generalization ability of the model.

[0076] The improved GAPGD adversarial training mentioned above virtually expands the original data by K times without increasing additional computing overhead, thereby improving the utilization rate of gradients in training. At the same time, after multiple iterations and accumulated gradients, the model can learn from multiple different norm constraints. Compared with the PGD algorithm with a single norm constraint, the generalization error is smaller, thereby enhancing the generalization ability of the model.

[0077] The PowerBERT model trained with the improved adversarial training strategy converts the input power text sequence into word vectors, sentence vectors, and position quantities, which are then input into the model for extraction. The input marketing audit text is modeled using the self-attention mechanism and the fully connected layer, and effective information that is more in line with the current task objectives is selected from a large amount of text information, greatly improving the accuracy of the model in identifying entity boundaries. In order to further learn the features of marketing audit text information, the text data is input into the pre-trained PowerBERT module in units of sentences to obtain high-quality word vectors, and then the BiLSTM layer is used to learn the contextual features of the power text. The BiLSTM layer will capture the contextual dependencies in the sequence through bidirectional processing to further enhance the feature representation.

[0078] Through a large number of practical applications, it is found that deep learning LSTM can achieve effective storage and update of text context information and maintain the dependency between data. However, LSTM cannot encode information from tail to head, while the BiLSTM model can achieve bidirectional semantic dependency to obtain more comprehensive text information. The BiLSTM model controls the flow and computation of information through three gates: input gate, forget gate, and output gate. The PowerRoberta layer converts power text into an embedding vector, so a sequence of words can be represented as X*=[x*1,x*2,…,x*η], where x*t∈Rd is the d-dimensional word vector corresponding to the tth word in the sentence processed by the PowerBERT layer, and η represents the length of the given sentence.

[0079] The update formula of the BiLSTM left-to-right recurrent neural network layer is:

[0080]

[0081] The update formula for the right-to-left recurrent neural network layer is:

[0082]

[0083] Finally, we will and Connected together, it represents the coded information of marketing audit vocabulary t, expressed as The calculation formula for the final output of the BiLSTM neural network is:

[0084]

[0085] Where f and g are activation functions, W, V, and U are weight matrices, b and c are bias matrices, and ht is the state to be activated.

[0086] Although the BiLSTM layer takes context information into account, it does not take into account the dependencies between labels, and cannot guarantee accurate prediction results at each output layer. Therefore, prediction results that do not meet the constraints of the training set will appear, such as: the first-letter label of a class of entities is only allowed to be connected to the non-first-letter label of the same class of entities, and the non-first-letter labels of other classes of entities are not allowed to appear directly, specifically: the first-letter label "BEM" of electric power and electrical equipment can only be followed by the same class of middle or end labels "I / EEM" or "O", and other classes of entities such as customer non-first-letter labels "ICUS" are not allowed to be connected. Therefore, for the problem of entity and relationship extraction in the field of marketing audit, the present invention adopts the CRF model to learn constraint rules on the basis of BiLSTM feature learning, obtains the global optimal label sequence by considering the dependencies between adjacent entity labels, and corrects the output of the BiLSTM network layer, thereby reducing the possibility of the model outputting illegal sequences, making up for the shortcomings of the BiLSTM layer, avoiding unreasonable label prediction, and realizing accurate extraction of entities and relationships in the field of marketing audit.

[0087] The feature vector sequence h = (h1, h2, ... hn) output by the BiLSTM layer, where hi represents the feature vector of the i-th tag. The CRF layer maps the sequence h to a label sequence y = (y1, y2, ..., yn), where yi is the label of the i-th tag. The scoring function of the CRF model is defined as:

[0088]

[0089] Where W yi is with label y i The corresponding weight vector, T yi,yi+1 is the label y i and i+1 The higher the value of the scoring function, the greater the probability that the given input sequence and label sequence will appear at the same time.

[0090] During training, the CRF layer learns the parameters W and T by maximizing the log-likelihood function:

[0091]

[0092] When predicting, the CRF layer selects the optimal label sequence y* by decoding:

[0093]

[0094] According to the labels in the optimal label sequence y*, the corresponding tags are decoded into the starting position, the ending position and the type of the entity. For example, the label BCUS represents the starting position of the customer entity, and the label ECUS represents the ending position. The relationship category label of each entity is identified from the label sequence, and the role of each entity in the relationship is assigned according to the role identifier in the label, that is, H is assigned to the head entity and T is assigned to the tail entity.

[0095] Taking the sequence labeling process of "transformer" as an example, first, the vector generated by PowerBERT pre-training is passed through BiLSTM to obtain the hidden vector h1h3 of each character. The values ​​of each dimension of the vector correspond to the scores of each power entity label. The comprehensive score is calculated based on the score of each label and the transfer score between different labels in the CRF layer, and then the label path with the largest current comprehensive score is taken as the optimal label path, such as Figure 5 As shown by the red line in the middle, the optimal label is the final annotation result of "transformer", namely "BEMH, IEMH, EEMH", indicating that the entity type of "transformer" is "EM", and the entity boundary of "transformer" and the entity role are identified as the tail entity, thus realizing the entity extraction of "transformer" in the power text.

[0096] After all entities are identified, the head entity and the tail entity are paired according to their role labels (H, T) to form candidate entity pairs. For each entity pair (Hi, Ti), the corresponding relationship part in the label sequence is checked. The relationship category will be clearly marked in the label sequence. The extracted entity and relationship categories are used to construct relationship triples to complete the joint extraction of marketing audit entity relationships.

Claims

1. A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation, firstly using "BIESO" tags to represent the position boundary information of words in entities; characterized in that, It also includes the following processes: S01: Entity category and relationship category labels are added based on entity location boundary labels to reflect the relationship between entities.

2. A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 1, characterized in that: S01 is followed by the following process: S02: Use the labeled data set to train the BERT model to further improve the model's recognition ability.

3. A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 2, characterized in that: S02 is followed by the following process: S03: After model training, for the extracted entities and relationships, if there are diverse labels, combined with the cosine similarity calculation formula, the entity vector and similarity label are used as the basis for judgment to integrate the entities.

4. The semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 1, characterized in that: The entity type tags include: tag "CUS", representing customer; tag "AR", representing region; tag "EM", representing equipment; tag "STA", representing station area; tag "AT", representing acquisition terminal; tag "FEE", representing receivable service fee; tag "OPE", representing service; The relationship type label includes: The label "PAY" represents payment; the label "BP" represents business processing; the label "BUY" represents purchase; the label "USE" represents use; and the label "OL" represents an overlapping relationship.

5. The semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 2, characterized in that: The BERT model training includes the following process: The input of the BERT model is divided into three parts: word embedding, paragraph embedding, and position embedding; For a given token sequence S = [s1, s2, s3, ..., sn3, sn2] of length n2, add special tokens [CLS] and [SEP] at the beginning and end of the sequence, and convert each token in the sequence into its corresponding word vector ei to obtain the word embedding matrix E; Assign a position vector pi to each position in the sequence, and obtain the position embedding matrix P containing all positions; Paragraph embedding is used to distinguish two sentences and is used only once in a sequence, denoted as [SEP]. The paragraph embedding vector si[SEP] indicates the beginning of a paragraph. For each token si in the sequence, its input ii is composed of word embedding, position embedding, and paragraph embedding, as shown in the following formula: In the formula, e i represents the word embedding vector of the i-th word, p i represents the position embedding vector of the i-th word, s i The paragraph embedding vector representing the paragraph to which the i-th word belongs; The obtained input representation I = [i1, i2, ..., in1, in] is fed to the Transformer layer of the Bert model, and each token si gets a context-aware vector representation xi: X=[x1,x2,…,xn]; The BERT module was pre-trained using the power marketing audit corpus constructed in S01, and the PowerBERT model for the marketing audit domain was trained in combination with the professional dictionary in the power marketing domain.

6. A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 5, characterized in that: During the training process of the PowerBERT model, adversarial training is introduced to introduce small perturbations, i.e. adversarial samples, into the input data to enhance the model's resistance to noise and attacks, thereby improving the model's robustness and generalization ability. The specific process is as follows: Based on PGD adversarial training, a gradient accumulation strategy is proposed. In multiple forward propagation and back propagation processes, the model parameters are not updated immediately, but the gradients are accumulated, as shown in the following formula: Where, the generated adversarial sample is perturbed_input, Π β(input,ε) is a projection operator that constrains the generated adjoint samples to be within a radius of ε near the original input. Input is the original input, α is the step size, is the loss function for a given target, grad (t) is the gradient of the model parameter θ, and grad_accumulator represents the accumulated gradient; After K iterations, the average of the accumulated gradients is calculated and the model parameters are updated using the average gradient, as shown in the following formula: Where avg_grad is the average value of the accumulated gradient, θ(K) is the model parameter at the Kth iteration, and η is the model learning rate; The improved GAPGD optimization objective is as follows:

7. A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 6, characterized in that: The subsequent process also includes the following: The text data is input into the pre-trained PowerBERT module in sentences to obtain high-quality word vectors, and then the BiLSTM layer is used to learn the contextual features of the power text. The BiLSTM layer will capture the contextual dependencies in the sequence through bidirectional processing to further enhance the feature representation.

8. A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 7, characterized in that: The PowerRoberta layer of the BiLSTM model converts power text into an embedding vector. A sequence of words can be represented as X*=[x*1,x*2,…,x*η], Where x*t∈Rd is the d-dimensional word vector corresponding to the t-th word in the sentence after being processed by the PowerBERT layer, and η represents the length of the given sentence; The update formula of the BiLSTM left-to-right recurrent neural network layer is: In the formula, The update formula for the right-to-left recurrent neural network layer is: Finally, we will and Connected together, it represents the coded information of marketing audit vocabulary t, expressed as The calculation formula for the final output of the BiLSTM neural network is: Where f and g are activation functions, W, V, and U are weight matrices, b and c are bias matrices, and ht is the state to be activated.

9. A semantic identification and penetration method based on the diversity and ambiguity of sequence annotation according to claim 8, characterized in that: It also includes the following processes: Based on BiLSTM feature learning, the CRF model is used to learn constraint rules. By considering the dependency between adjacent entity labels, the global optimal label sequence is obtained and the output of the BiLSTM network layer is corrected, thereby reducing the possibility of the model outputting illegal sequences. The feature vector sequence h = (h1, h2, ... hn) output by the BiLSTM layer, where hi represents the feature vector of the i-th tag; the CRF layer maps the sequence h to a label sequence y = (y1, y2, ..., yn), where yi is the label of the i-th tag; the scoring function of the CRF model is defined as: Where W yi is with label y i The corresponding weight vector, T yi,yi+1 is the label y i and i+1 The higher the value of the scoring function, the greater the probability that the given input sequence and label sequence will appear at the same time; During training, the CRF layer learns the parameters W and T by maximizing the log-likelihood function: When predicting, the CRF layer selects the optimal label sequence y* by decoding: y * =argmax y s(h,y), According to the labels in the optimal label sequence y*, decode the corresponding tokens into the starting position, ending position and type of the entity; First, the vector generated by PowerBERT pre-training is passed through BiLSTM to obtain the hidden vector h1h3 of each character. The value of each dimension of the vector corresponds to the score of each power entity label. The comprehensive score is calculated based on the score of each label and the transfer score between different labels in the CRF layer, and then the label path with the largest current comprehensive score is taken as the optimal label path. After all entities are identified, the head entity and the tail entity are paired according to their role labels (H, T) to form candidate entity pairs; for each entity pair (Hi, Ti), the corresponding relationship part in the label sequence is checked, and the relationship category will be clearly marked in the label sequence. Using the extracted entity and relationship categories, the relationship triples are constructed to complete the joint extraction of entity relationships.

Images