Entity extraction method and system based on dynamic window attention mechanism

Abstract

The invention proposes an entity extraction method and system based on a dynamic window attention mechanism, and the method comprises the steps: data collection, data preprocessing, and automatic labeling training, and during the automatic labeling training, based on the dynamic window attention mechanism, through improving the attention extraction weight of a TMN frame, the automatic labeling training is carried out. Trigger training, entity recognition model training and entity recognition model testing are carried out. From the perspective, the problem of key information omission or redundancy possibly caused by a fixed window is avoided by introducing a dynamic window attention mechanism, so that the model is more efficient when processing long sentences and complex sentence patterns, and the text information capture capability and the feature extraction effect of the model are remarkably improved. Besides, by combining the similarity matching optimization between the trigger and the sentence, the distinguishing capability of the model in a trigger classification task is remarkably improved, so that the distinguishing of a matched sample and an unmatched sample is more accurate.

Description

Technical Field

[0001] The present invention relates to an entity extraction method and system based on a dynamic window attention mechanism, and belongs to the field of information extraction. Background Art

[0002] In the era of big data, complex information contains rich value. The database operation and maintenance knowledge graph helps operation and maintenance engineers to dig deeper into the relationships and abnormal conditions between organizations, help operation and maintenance personnel to find abnormal points in the database faster, and provide efficient and fundamental solutions. In the process of building the operation and maintenance knowledge graph, entity extraction is a crucial step. In the process of entity extraction, most of the data sources are unstructured information, such as text, logs, etc., with unclear and irregular structures; in addition, these texts contain a large number of domain nouns with semantic conflicts, which brings considerable burden and obstacles to entity extraction.

[0003] Existing entity extraction methods mainly rely on sequence annotation models based on deep learning, such as Long Short-Term Memory (LSTM) and Conditional Random Field (CRF). Although these deep learning-based extraction methods can automatically learn feature representations in text and handle complex semantic relationships, they still have the following shortcomings:

[0004] (1) Most entity extraction models in the prior art use a fixed-size window for feature extraction, which is difficult to adapt to different text lengths and complexities. Especially when processing long or complex sentences, it may result in missing key information or introducing redundant information.

[0005] (2) Existing methods have difficulty effectively distinguishing whether triggers and sentences match. Traditional cross entropy loss and matching loss based on Euclidean distance are not stable enough in high-dimensional space, which may lead to poor matching results. At the same time, when dealing with mismatched samples, existing methods often only consider simple mismatched samples and ignore mismatched samples that are difficult to distinguish. This approach cannot effectively improve the generalization ability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is a system architecture diagram of the present invention.

[0007] Figure 2 This is a flow chart of the trigger training phase of the present invention Summary of the invention

[0008] In order to solve the problems existing in the prior art, the present invention proposes an entity extraction method based on a dynamic window attention mechanism, which comprises the following steps:

[0009] (1) Data collection: collecting database logs, application logs, user interaction logs, and API call records;

[0010] (2) Data preprocessing: cleaning, filtering and converting the various data collected above;

[0011] (3) Automatic labeling training;

[0012] Furthermore, step (3) is based on the dynamic window attention mechanism and performs trigger training, entity recognition model training and entity recognition model testing by improving the attention extraction weights of the TMN framework.

[0013] Furthermore, during the trigger training in step (3), the specific steps are as follows:

[0014] 1) Calculate dynamic window size

[0015] a) Calculate the hidden state h t The probability distribution of where h t,i is the hidden state h at time step t t In the i-th dimension, P(h t,i ) is the hidden state h at time step t t The probability distribution of the i-th position in; d is the dimension of the hidden state;

[0016] b) Calculate information entropy Where H(h t ) is the hidden state h at time step t t Information entropy of

[0017] c) Normalized information entropy Where max(H(h t )) is the maximum value of information entropy;

[0018] d) Calculate the mapping to the window size Where D max is the preset maximum window size, D t is the dynamic window size at time step t.

[0019] 2) Training attention query

[0020] a) For trigger classification training, calculate the loss function Among them, Pr(type(e)|g k θ TC ) is the entity type type(e) predicted by the model. Given the input vector g k The probability under which e is the entity and θ TCis the model parameter of the trigger classifier; γ is the hyperparameter that controls the strength of the adjustment factor; ε is the smoothing parameter, and N is the total number of categories;

[0021] b) Calculate the matching contrast loss function Among them, cos(g s ,g t ) is the sentence vector g s and trigger vector g t The cosine similarity between matched =1, the sample is a matching sample; l matched When =0, the sample is a mismatched sample; m is the boundary value between whether the sample matches or not.

[0022] Based on the above method, the present invention proposes an entity extraction system based on a dynamic window attention mechanism, which includes a data acquisition module, a data preprocessing module, and an automatic labeling module. Specifically:

[0023] (1) The data acquisition module includes the following submodules:

[0024] Multi-channel data integration submodule: This submodule collects database logs, application logs, user interaction logs, and API call records;

[0025] Sensor data integration submodule: This submodule combines database performance indicators, system monitoring data, etc. with external data sources;

[0026] Real-time data stream processing submodule: This submodule uses Kafka stream data processing technology to realize real-time data collection and processing.

[0027] (2) Data preprocessing: cleaning, filtering and conversion of collected data, including the following submodules:

[0028] Data cleaning submodule: This submodule removes duplicate data, missing values, and outliers through algorithms and rules, and uses Spark tools to clean data;

[0029] Data conversion and standardization submodule: This submodule performs word segmentation and stop word removal on text data;

[0030] Dataset division and verification submodule: This submodule divides the processed data into training set, verification set and test set;

[0031] (3) Automatic labeling training module.

[0032] Furthermore, the automatic annotation training module is based on the dynamic window attention mechanism and performs trigger training, entity recognition model training and entity recognition model testing by improving the attention extraction weights of the TMN framework.

[0033] Furthermore, when the trigger training is performed in the automatic labeling training module, the specific steps are as follows:

[0034] 1) Calculate dynamic window size

[0035] a) Calculate the hidden state h t The probability distribution of where h t,i is the hidden state h at time step t t In the i-th dimension, P(h t,i ) is the hidden state h at time step t t The probability distribution of the i-th position in; d is the dimension of the hidden state;

[0036] b) Calculate information entropy Where H(h t ) is the hidden state h at time step t t Information entropy of

[0037] c) Normalized information entropy Where max(H(h t )) is the maximum value of information entropy;

[0038] d) Calculate the mapping to the window size Where D max is the preset maximum window size, D t is the dynamic window size at time step t.

[0039] 2) Training attention query

[0040] a) For trigger classification training, calculate the loss function Among them, Pr(type(e)|g k θ TC ) is the entity type type(e) predicted by the model. Given the input vector g k The probability under which e is the entity and θ TC is the model parameter of the trigger classifier; γ is the hyperparameter that controls the strength of the adjustment factor; ε is the smoothing parameter, and N is the total number of categories;

[0041] b) Calculate the matching contrast loss function Among them, cos(g s ,g t ) is the sentence vector g s and trigger vector g t The cosine similarity between matched =1, the sample is a matching sample; l matched When =0, the sample is a mismatched sample; m is the boundary value between whether the sample matches or not.

[0042] The present invention enhances the robustness of the model to noise and reduces the sensitivity to label noise by using hyperparameters and smoothing parameters, thereby significantly improving the accuracy of classification tasks and the distinguishing ability of the model in matching training. The improved contrast loss function enhances the ability to distinguish whether samples are matched, ensuring that there is a clear distinction between mismatched samples and matched samples.

[0043] In summary, the present invention avoids the omission or redundancy of key information that may be caused by fixed windows by introducing a dynamic window attention mechanism, making the model more efficient when processing long sentences and complex sentences, and significantly improving the model's ability to capture text information and feature extraction effect. In addition, by combining the similarity matching optimization between triggers and sentences, the model's ability to distinguish in trigger classification tasks is significantly improved, making the discrimination of matching samples and non-matching samples more accurate. DETAILED DESCRIPTION

[0044] Example 1

[0045] The present invention proposes an entity extraction method based on a dynamic window attention mechanism, which comprises the following steps:

[0046] (1) Data collection;

[0047] (2) Data preprocessing;

[0048] (3) Automatic labeling training;

[0049] Furthermore, in step (3), the method of extracting attention weights in the trigger training phase in the trigger matching network (TMN) framework is improved based on the dynamic window attention mechanism.

[0050] (1) Dynamic window size calculation:

[0051] In view of the problems in the prior art of insufficient context capture capability, fixed windows that may bring a large amount of irrelevant noise information, and limited ability to process complex sentences, the present invention proposes a new dynamic window adjustment mechanism that can be dynamically adjusted according to the degree of confusion of the hidden state. The specific steps are as follows:

[0052] 1) Calculate the hidden state h t The probability distribution of where h t,i represents the hidden state h at time step t t In the i-th dimension, P(h t,i ) represents the hidden state h at time step t t The probability distribution of the i-th position in ; d represents the dimension of the hidden state.

[0053] 2) Calculate information entropy Where H(h t ) represents the hidden state h at time step t t Information entropy.

[0054] 3) Normalized information entropy Where max(H(h t )) is the maximum value of information entropy and is used for normalization processing.

[0055] 4) Mapping to window size Where D max is the preset maximum window size, D t is the dynamic window size at time step t.

[0056] (2) In the training attention query stage, the weights of the trigger and the surrounding information of the short sentence level text are calculated using the Local Attention mechanism and Global Attention respectively. The trigger vector g is extracted t and sentence vector g s Finally, they are fed into a multi-classifier for entity type prediction and optimized through a joint loss function to obtain the optimal representation.

[0057] In view of the above problems, the present invention proposes:

[0058] 1) For trigger classification training, the loss function is as follows:

[0059]

[0060] Among them, Pr(type(e)|g k θ TC ) represents the entity type type(e) predicted by the model given the input vector g k The probability under which e represents the entity and θ TC is the model parameter of the trigger classifier; γ is a hyperparameter used to control the strength of the adjustment factor. ε is the smoothing parameter, N represents the total number of categories, and in label smoothing, it is used to evenly distribute ε to all categories.

[0061] 2) Define the matching contrast loss function as follows:

[0062]

[0063] Among them, cos(g s ,g t ) represents the sentence vector g s and trigger vector g t The cosine similarity between matched Indicates whether the sample is a label of the matching sample, l matched=1 means the sample is a matching sample, that is, the sentence and the trigger are matched, otherwise l matched =0 indicates that the sample is a mismatched sample; m indicates the boundary value between whether the sample matches or not. For mismatched samples, the cosine similarity is as small as possible from m to ensure that mismatched samples are clearly distinguished from matched samples.

[0064] The present invention designs an entity extraction system based on a dynamic window attention mechanism. The system mainly includes a data acquisition module, a data preprocessing module, and an automatic annotation module. The specific system architecture diagram is shown in Figure 1 .

[0065] 1. Data acquisition module:

[0066] (1) Multi-channel data integration: Collect multiple data sources such as database logs, application logs, user interaction logs, API call records, etc. to ensure the comprehensiveness and diversity of data collection.

[0067] (2) Sensor data integration: combining database performance indicators, system monitoring data, etc. with external data sources, such as technical forums.

[0068] (3) Real-time data stream processing: Kafka stream data processing technology is used to realize real-time data collection and processing to ensure the timeliness and continuity of data.

[0069] 2. Data preprocessing: Clean, filter and transform the collected data to remove noise and errors, standardize the format, and ensure data quality and consistency.

[0070] (1) Data cleaning: Remove duplicate data, missing values, and outliers through algorithms and rules, and use Spark tools to clean data to improve data quality and reliability.

[0071] (2) Data conversion and standardization: Segment text data and remove stop words to ensure data consistency and availability.

[0072] (3) Dataset division and verification: The processed data is divided into training set, validation set and test set to ensure the adequacy and balance of the data.

[0073] 3. Automatic labeling training module: Based on the dynamic window attention mechanism, by improving the attention weight extraction method of the TMN framework, the model's ability to capture sequence order information and adapt to tasks is enhanced.

[0074] Specifically, the automatic annotation training module includes trigger training, entity recognition model training, and entity recognition model testing. In the trigger training phase, a dynamic window adjustment mechanism is proposed to address the problem that the fixed-size window in the existing entity extraction model is difficult to adapt to different text lengths and complexities. The window size is dynamically adjusted according to the degree of confusion of the hidden state. This mechanism improves the ability to capture contextual information and ensures that important information is not missed, while reducing the introduction of irrelevant noise, thereby improving the accuracy of information extraction and the judgment ability of the model. In addition, the dynamic window size mechanism can better handle complex sentences, provide more accurate contextual understanding, and improve the overall information extraction effect. See the trigger training phase process for details. Figure 2 .

[0075] (1) The specific process of the dynamic window adjustment mechanism is as follows:

[0076] 1) Calculate the hidden state h t The probability distribution of Where P(h t,i ) represents the hidden state h at time step t t The probability of the i-th position in ; d represents the dimension of the hidden state.

[0077] 2) Calculate information entropy Where H(h t ) represents the hidden state h at time step t t Information entropy.

[0078] 3) Normalized information entropy Where max(H(h t )) is the maximum value of information entropy and is used for normalization processing.

[0079] 4) Mapping to window size Where D max is the preset maximum window size, D t is the dynamic window size at time step t.

[0080] (2) In the Local Attention mechanism, at each time step t, the attention function calculates a trigger vector of the input sequence Where z represents the position index in the sequence, p t Indicates the position of the trigger in the sequence, Represents the context vector The attention weights on the trigger vector, is the trigger position p t Left and right D t The context vector representation within each position.

[0081] (3) In the training attention query phase, the trigger vector g is extracted t and sentence vector g s Finally, they are fed into a multi-classifier for entity type prediction and optimized through a joint loss function to obtain the optimal representation.

[0082] Specifically, for trigger classification training, the loss function is as follows:

[0083]

[0084] Among them, Pr(type(e)|g k θ TC ) represents the entity type type(e) predicted by the model given the input vector g k The probability of TC is the model parameter of the trigger classifier; γ is a hyperparameter used to control the strength of the adjustment factor. ε is the smoothing parameter, N represents the total number of categories, and in label smoothing, it is used to evenly distribute ε to all categories.

[0085] The contrast loss function for matching is defined as follows:

[0086]

[0087] Among them, cos(g s ,g t ) represents the sentence vector g s and trigger vector g t The cosine similarity between matched Indicates whether the sample is a label of the matching sample, l matched =1 means the sample is a matching sample, that is, the sentence and the trigger are matched, otherwise l matched = 0 indicates that the sample is a mismatched sample; m is a hyperparameter, which indicates the boundary value between matching and mismatching. For mismatched samples, the cosine similarity is as small as possible from m to ensure that mismatched samples are clearly distinguished from matching samples.

[0088] Example 2

[0089] (1) Calculate dynamic window size

[0090] Assume that the hidden state h t =[1.0,0.5,0.2,0.3],

[0091] Calculate the probability distribution P(h t )=[0.392,0.238,0.176,0.194],

[0092] Calculate the information entropy H(h t)=-(-0.367-0.341-0.305-0.317)=1.33,

[0093] Assume that the maximum information entropy max(H(h t ))=1.5. Therefore, the normalized information entropy H′(h t )=1.33 / 1.5=0.887,

[0094] Assumption D max =5, then the window size at time step t

[0095] (2) Assume that the trigger vector and sentence vector are g respectively. t =[0.39,0.27,0.28,0.2] and g s =[0.5,0.4,0.3,0.2], the entity type probability predicted by the model is 0.7, the total number of categories N is 3, the hyperparameter γ = 2, the smoothing parameter ε = 0.1, then Loss1 = -(1-0.7) 2 ×log((1-0.1)×0.7+0.1 / 3)≈0.037.

[0096] Assume that cos(g s ,g t )=0.8, the boundary value m is 0.5, there are 1 matching sample and 1 mismatching sample, then Loss2=(1-0.8)+max(0,0.5-0.8) 2 =0.2.

[0097] Finally, a window size D is determined at time step t t =4, and by optimizing the loss function, the loss value of the model converges faster and more stably.

[0098] The units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0099] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0100] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0101] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0102] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0103] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for entity extraction based on a dynamic window attention mechanism, the method comprising the following steps: (1) Data collection: collecting database logs, application logs, user interaction logs, and API call records; (2) Data preprocessing: cleaning, filtering and converting the various data collected above; (3) Automatic labeling training; it is characterized by: Step (3) is based on the dynamic window attention mechanism and performs trigger training, entity recognition model training and entity recognition model testing by improving the attention extraction weights of the TMN framework.

2. The entity extraction method based on the dynamic window attention mechanism according to claim 1, characterized in that: When the trigger is trained in step (3), the specific steps are as follows: 1) Calculate dynamic window size a) Calculate the hidden state h t The probability distribution of where h t,i is the hidden state h at time step t t In the i-th dimension, P(h t,i ) is the hidden state h at time step t t The probability distribution of the i-th position in; d is the dimension of the hidden state; b) Calculate information entropy Where H(h t ) is the hidden state h at time step t t Information entropy of c) Normalized information entropy Where max(H(h t )) is the maximum value of information entropy; d) Calculate the mapping to the window size Where D max is the preset maximum window size, D t is the dynamic window size at time step t. 2) Training attention query a) For trigger classification training, calculate the loss function Among them, Pr(type(e)|g k θ TC ) is the entity type type(e) predicted by the model. Given the input vector g k The probability under which e is the entity and θ TC is the model parameter of the trigger classifier; γ is the hyperparameter that controls the strength of the adjustment factor; ε is the smoothing parameter, and N is the total number of categories; b) Calculate the matching contrast loss function Among them, cos(g s ,g t ) is the sentence vector g s and trigger vector g t The cosine similarity between matched =1, the sample is a matching sample; l matched When =0, the sample is a mismatched sample; m is the boundary value between whether the sample matches or not.

3. An entity extraction system based on a dynamic window attention mechanism, which includes a data acquisition module, a data preprocessing module, and an automatic labeling module. Specifically: (1) The data acquisition module includes the following submodules: Multi-channel data integration submodule: This submodule collects database logs, application logs, user interaction logs, and API call records; Sensor data integration submodule: This submodule combines database performance indicators, system monitoring data, etc. with external data sources; Real-time data stream processing submodule: This submodule uses Kafka stream data processing technology to realize real-time data collection and processing. (2) Data preprocessing module: Cleans, filters and converts the collected data, including the following submodules: Data cleaning submodule: This submodule removes duplicate data, missing values, and outliers through algorithms and rules, and uses Spark tools to clean data; Data conversion and standardization submodule: This submodule performs word segmentation and stop word removal on text data; Dataset division and verification submodule: This submodule divides the processed data into training set, verification set and test set; (3) Automatic labeling training module, characterized by: The automatic labeling training module is based on the dynamic window attention mechanism. It performs trigger training, entity recognition model training and entity recognition model testing by improving the attention extraction weights of the TMN framework.

4. The entity extraction system based on dynamic window attention mechanism as described in claim 3, characterized in that: When training triggers in the automatic annotation training module, the specific steps are as follows: 1) Calculate dynamic window size a) Calculate the hidden state h t The probability distribution of where h t,i is the hidden state h at time step t t In the i-th dimension, P(h t,i ) is the hidden state h at time step t t The probability distribution of the i-th position in; d is the dimension of the hidden state; b) Calculate information entropy Where H(h t ) is the hidden state h at time step t t Information entropy of c) Normalized information entropy Where max(H(h t )) is the maximum value of information entropy; d) Calculate the mapping to the window size Where D max is the preset maximum window size, D t is the dynamic window size at time step t. 2) Training attention query a) For trigger classification training, calculate the loss function Among them, Pr(type(e)|g k θ TC ) is the entity type type(e) predicted by the model. Given the input vector g k The probability under which e is the entity and θ TC is the model parameter of the trigger classifier; γ is the hyperparameter that controls the strength of the adjustment factor; ε is the smoothing parameter, and N is the total number of categories; b) Calculate the matching contrast loss function Among them, cos(g s ,g t ) is the sentence vector g s and trigger vector g t The cosine similarity between matched =1, the sample is a matching sample; l matched When =0, the sample is a mismatched sample; m is the boundary value between whether the sample matches or not.

Images