A method for identifying semantic elements of special domain text based on enhanced CRF layer

Abstract

The invention relates to the field of text semantic analysis, and in particular to a method for identifying semantic elements of special-domain text based on an enhanced CRF layer, comprising: performing sample screening analysis on a sample library to obtain a plurality of training samples; for a single training sample, extracting semantic elements from the training sample according to a preset strategy function to obtain a label sequence; calculating a cumulative discounted reward corresponding to the label sequence; using a preset objective function to obtain an expected value of a probability reward corresponding to the label sequence, using a preset gradient formula to perform gradient calculation, and updating model parameters; determining whether to reselect a label according to a preset exploration strategy; and stopping model training and learning under a preset stop condition. The invention improves the ability of a model to effectively capture the complex dependencies between label sequences, thereby improving the accuracy and robustness of the model in semantic element recognition tasks, and improving the ability to process long texts.

Description

Technical Field

[0001] The invention relates to the field of text semantic analysis, and in particular to a method for recognizing semantic elements of special-domain text based on an enhanced CRF layer. Background Art

[0002] BERT is a pre-trained language model that can capture rich language features and contextual information through unsupervised learning on large amounts of text data. Conditional Random Field (CRF) is a statistical model used for labeling tasks of sequence data. It can consider the dependencies between labels and improve the accuracy of sequence labeling. The BERT-CRF model combines the contextual representation ability of BERT with the sequence labeling ability of Conditional Random Field (CRF) and is widely used in Chinese named entity recognition tasks. However, in the prior art, since the BERT-CRF model has not been effectively trained to capture the complex dependencies between label sequences, the model's adaptability and generalization ability in semantic element recognition tasks are not satisfactory.

[0003] Chinese patent publication number CN110083831A discloses a Chinese named entity recognition method based on BERT-BiGRU-CRF. The method includes three stages: the first stage preprocesses massive text corpus and pretrains the BERT language model; the second stage preprocesses the named entity recognition corpus and encodes the named entity recognition corpus using the trained BERT language model; the third stage inputs the encoded corpus into the BiGRU+CRF model for training, and uses the trained model to perform named entity recognition on the sentences to be recognized. It can be seen that the above technical solution has the following problems: the model has not been effectively trained to capture the complex dependencies between label sequences, which makes the model's adaptability and generalization ability in the semantic element recognition task poor. Summary of the invention

[0004] To this end, the present invention provides a method for recognizing semantic elements of special-domain text based on an enhanced CRF layer, so as to overcome the problem of poor adaptability and generalization ability of the model in the prior art in the task of semantic element recognition.

[0005] To achieve the above object, the present invention provides a method for recognizing semantic elements of special domain text based on an enhanced CRF layer, comprising:

[0006] Step S1, performing sample screening analysis on the sample library to obtain a number of training samples;

[0007] Step S2, for a single training sample, extracting semantic elements from the training sample according to a preset strategy function to obtain a label sequence;

[0008] Step S3, calculating the cumulative discount reward corresponding to the tag sequence;

[0009] Step S4, using a preset objective function to obtain the expected value of the probability reward corresponding to the label sequence, using a preset gradient formula to perform gradient calculation, and updating the model parameters;

[0010] Step S5, determining whether to reselect a tag according to a preset exploration strategy;

[0011] Step S6, repeat steps S2 to S5, and stop model training and learning under a preset stop condition.

[0012] Furthermore, the preset strategy function is π(a|s;θ), where s is a given state, a is the probability of selecting the next label, and θ is a model parameter.

[0013] Furthermore, the cumulative discount reward is R, and the calculation formula of R is:

[0014] ,

[0015] Where n is the current time step number, is the discount factor, 0< <1, is the immediate reward corresponding to the t-th time step, t≤n.

[0016] Furthermore, the preset objective function is:

[0017] .

[0018] Furthermore, the preset gradient formula is:

[0019] ;

[0020] Among them, b is the benchmark function.

[0021] Furthermore, the preset exploration strategy includes:

[0022] Detect the current execution probability ε;

[0023] If the execution probability ε is less than the preset execution probability, an unused label is randomly selected and reselected;

[0024] If the execution probability ε is greater than or equal to the preset execution probability, the current label is selected as the best label;

[0025] Among them, the calculation formula of execution probability ε is:

[0026] ;

[0027] Among them, m is the number of iterations, λ is the decay rate, ϵmin is the minimum exploration probability, and ϵmax is the maximum exploration probability.

[0028] Further, under the preset adjustment condition, the uniform state of the model action is detected, and when the uniform state of the model action is in the preset uniform state of the model action, the preset objective function is converted;

[0029] The preset objective function after conversion is:

[0030] ;

[0031] in, , α is the regularization coefficient, .

[0032] Furthermore, the value of the regularization coefficient is determined according to the reward parameter value;

[0033] The regularization coefficient is negatively correlated with the loss fluctuation value.

[0034] Furthermore, the sample screening analysis includes:

[0035] Randomly select a preset number of samples to be selected;

[0036] Test the sample feature richness;

[0037] If the sample feature richness is within the first preset sample feature richness range, increasing the number of training samples;

[0038] If the sample feature richness is within a second preset sample feature richness range, it is determined that the training sample meets the training requirements.

[0039] Furthermore, the sample feature richness is determined according to the special domain text segmentation reference value of each sample to be selected and the number of special domain text characters.

[0040] Compared with the prior art, the beneficial effect of the present invention lies in that the enhanced CRF layer optimization strategy in the technical solution of the present invention can effectively capture the complex dependencies between label sequences, improve the performance of the model in the semantic element recognition task, not only improve the learning ability of the model, but also enhance the adaptability and generalization ability of the model finally learned, optimize and fine-tune the model parameters through training, and achieve the improvement of the recognition effect of the model on the semantic elements of texts in special fields, significantly improve the accuracy and robustness of the recognition of semantic elements of texts in special fields, and especially improve the processing ability for long-length or multimodal content.

[0041] Furthermore, the discount factor in the present invention corresponds to an adjustment method, and the discount factor can be adjusted according to the training sample dependent reference value, so that the value of the discount factor is more in line with the actual working scenario, reducing the recognition error of short-term decision-making, and thus improving the training accuracy of the model of the present invention.

[0042] Furthermore, the preset objective function in the present invention represents the expected value of the product of the logarithmic probability of executing the action under the state and the cumulative reward obtained. The logarithmic probability here is used to ensure the non-negativity of the probability value and make the gradient calculation more stable. The present invention can optimize the strategy parameters to obtain the maximum cumulative reward in long-term game training.

[0043] Furthermore, the value of the regularization coefficient is determined according to the reward parameter value; the regularization coefficient is negatively correlated with the loss fluctuation value, and the value of the regularization coefficient is more in line with the actual scenario, so that the value of the regularization coefficient can meet the needs of model learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a method for recognizing semantic elements of text in a special domain based on an enhanced CRF layer according to an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a preset exploration strategy for an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of sample screening analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0049] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0050] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] See also Figures 1 to 3 As shown, the present invention provides a method for recognizing semantic elements of special domain text based on an enhanced CRF layer, comprising:

[0052] Step S1, performing sample screening analysis on the sample library to obtain a number of training samples;

[0053] Step S2, for a single training sample, extracting semantic elements from the training sample according to a preset strategy function to obtain a label sequence;

[0054] Step S3, calculating the cumulative discount reward corresponding to the tag sequence;

[0055] Step S4, using a preset objective function to obtain the expected value of the probability reward corresponding to the label sequence, using a preset gradient formula to perform gradient calculation, and updating the model parameters;

[0056] Step S5, determining whether to reselect a tag according to a preset exploration strategy;

[0057] Step S6, repeat steps S2 to S5, and stop model training and learning under a preset stop condition.

[0058] The preset stop condition is that the operations from step S2 to step S5 have been performed for each training sample. It can be understood that the label sequence is a core concept of the sequence labeling task in natural language processing. The label sequence assigns a predefined label to each word or token in the analyzed text. For example, the label sequence of "Li Lei and Han Meimei met at Peking University" may be "B-PER I-PER OO B-LOC I-LOC". This is easy to understand for those skilled in the art and will not be elaborated here. In the present invention, the special field text is text material closely related to the industry, discipline or field to be kept confidential, and this type of text usually contains professional terms, industry-specific concepts and expressions.

[0059] Specifically, the present invention also includes pre-preprocessing, which is performed before step S1;

[0060] Pre-processing includes:

[0061] Use some unlabeled text data to pre-train the BERT model;

[0062] Execute the Masked Language Model task and the Next Sentence Prediction task respectively;

[0063] Based on the pre-trained BERT model, fine-tune it using a labeled dataset to adapt to specific named entity recognition tasks;

[0064] Connect the output of the BERT model to the CRF layer to obtain the BERT-CRF model, and perform steps S1 to S6 to optimize for the CRF layer.

[0065] Specifically, the preset strategy function is π(a|s;θ), where s is a given state, a is the probability of selecting the next label, and θ is a model parameter.

[0066] The given state is the sequence of all labels before the current time step. The model parameters include the state transfer matrix and the feature weight vector of the CRF model. An embodiment of the state transfer matrix is ​​provided. If there are 6 state labels, the state transfer matrix will contain 6×6 parameters, and each parameter represents the probability of transferring from one state to another. The feature weight vector contains the weights of all feature functions. The number of feature functions determines the length of the feature weight vector. The feature function can be an artificially designed feature or a feature learned from the data. In the present invention, the feature weight vector is the weight corresponding to each label vector output by the BERT model.

[0067] Among them, the use of preset strategy functions can enable adaptive updating and optimization of model parameters during model training, improve model diversity, and facilitate optimization to improve performance.

[0068] Specifically, the cumulative discount reward is R, and the calculation formula of R is:

[0069] ,

[0070] Where n is the current time step number, is the discount factor, 0< <1, is the immediate reward corresponding to the t-th time step, t≤n.

[0071] The initial value of the discount factor. The user can select the best initial value of the discount factor by testing the model performance under different discount factors on the validation set. The present invention is provided with an adjustment method for the discount factor. The adjustment method for the discount factor is:

[0072] Randomly extract 50 training samples without model learning, detect the dependent reference value of each training sample, calculate the average value of the dependent reference value, and adjust the discount factor according to the average value of the dependent reference value. The average value of the dependent reference value is negatively correlated with the discount factor, thereby reducing the possible errors in short-term decision-making. It is worth noting that the discount factor should be set with a maximum allowable value and a minimum allowable value, the maximum allowable value is 1, and the minimum allowable value is 0. The dependent reference value includes the number of entities / the number of all entities in the training sample that depend on the context information of the entire sentence or paragraph. Entities include but are not limited to names of people, places, and organizations. Whether the recognition of entities depends on the context information of the entire sentence or paragraph can be determined by manual labeling or by a deep learning model.

[0073] Specifically, the preset objective function is:

[0074] ;

[0075] Among them, J(θ) is the expected value of the probability reward.

[0076] Specifically, the preset gradient formula is:

[0077] ;

[0078] Among them, b is the benchmark function.

[0079] Among them, the preset objective function represents the expected value of the product of the logarithmic probability of executing the action under the state and the cumulative reward obtained. The logarithmic probability here is used to ensure the non-negativity of the probability value and make the gradient calculation more stable. The present invention can optimize the strategy parameters to obtain the maximum cumulative reward in long-term game training.

[0080] Specifically, the preset exploration strategy includes:

[0081] Detect the current execution probability ε;

[0082] If the execution probability ε is less than the preset execution probability, an unused label is randomly reselected;

[0083] If the execution probability ε is greater than or equal to the preset execution probability, the current label is selected as the best label;

[0084] Among them, the calculation formula of execution probability ε is:

[0085] ;

[0086] Where m is the number of iterations, λ is the decay rate, εmin is the minimum exploration probability, and εmax is the maximum exploration probability. The minimum exploration probability is the minimum allowed value of ε set by the user as the iteration proceeds, and the maximum exploration probability is the exploration probability at the beginning of the iteration. The value of the decay rate can be set by the user according to the actual application scenario. The user can select the best decay rate by testing the model performance under different decay rates on the validation set, and adjust the decay rate according to the performance of the model on a specific task to achieve the best balance between exploration and utilization.

[0087] Specifically, under the preset adjustment conditions, the model action uniform state is detected, and when the model action uniform state is in the preset model action uniform state, the preset objective function is converted;

[0088] The preset objective function after conversion is:

[0089] ;

[0090] in, , α is the regularization coefficient, .

[0091] Specifically, the value of the regularization coefficient is determined according to the reward parameter value;

[0092] The regularization coefficient is negatively correlated with the loss fluctuation value.

[0093] In the present invention, an allowable value range is set for the regularization coefficient. The value of the regularization coefficient is greater than the minimum value of the allowable value range and less than the maximum value of the allowable value range. The allowable value range is (B1, B2). The values ​​of B1 and B2 can be determined through historical experience. The loss fluctuation value is confirmed by extracting the training errors of the most recent 50 training samples. For a single training sample, the training error is the number of incorrectly recognized characters / the total number of characters, and the loss fluctuation value is (number of error training times / 50).

[0094] Among them, the value of the regularization coefficient is more in line with the actual scenario, so that the value of the regularization coefficient can meet the needs of model learning. The preset adjustment condition is that the absolute value of the difference between H(π) and the preset reference value is less than the preset absolute value. It can be understood that if the value of H(π) is high, it means that the probability distribution of each action taken by the model in this state is relatively uniform, that is, the model is more exploratory in this state. On the contrary, if the value of H(π) is low, it means that the model tends to take certain specific actions in this state, that is, the model is less exploratory in this state. The user can record the log value of the action space as the preset reference value. The closer H(π) is to the log value of the action space, the better the uniformity of the action probability distribution. The value of the preset absolute value can be determined by the user based on historical records.

[0095] Specifically, the sample screening analysis includes:

[0096] Randomly select a preset number of samples to be selected;

[0097] Test the sample feature richness;

[0098] If the sample feature richness is within the first preset sample feature richness range, increasing the number of training samples;

[0099] If the sample feature richness is within a second preset sample feature richness range, it is determined that the training sample meets the training requirements.

[0100] Specifically, the sample feature richness is determined according to the special domain text segmentation reference value of each sample to be selected and the number of special domain text characters.

[0101] Among them, the increase in the number of training samples can be selected by the user according to actual needs, which will not be elaborated here. A method for increasing the number of training samples is provided, and the increase in the number of training samples is negatively correlated with the sample richness. The sample feature richness is used to reflect whether the currently selected samples meet the model training requirements, and then the number of training samples is adaptively adjusted, so that the number of training samples can ensure that the training needs are met while improving the training speed.

[0102] The sample feature richness is the average of the sum of the sub-richnesses of each sample to be selected. The values ​​of the first preset sample feature richness range are all less than the preset sample feature richness, and the values ​​of the second preset richness range are all greater than or equal to the preset sample feature richness. The value of the preset sample feature richness can be set by the user according to the actual application scenario and historical experience. The greater the user's demand for model training accuracy, the greater the value of the preset sample feature richness. No specific restriction is made here.

[0103] Sub-richness = special domain text segmentation reference value × β1 + number of special domain text characters × β2, where the special domain text segmentation reference value is confirmed by extracting the number of spacing characters between adjacent entities in the reading order of each text in the sample to be selected, and recording the average value of the number as the special domain text segmentation reference value of the sample to be selected, and the number of special domain text characters is the number of characters in the sample to be selected.

[0104] In the present invention, the number of training samples obtained in step S1 is set by the user. The higher the user's demand for model accuracy, the larger the number of training samples. A value of the number of training samples obtained in step S1 is provided, and the number is 500. The samples to be selected are randomly selected from the training samples. The preset number is 30% of the number of training samples obtained in step S1, and is randomly selected.

[0105] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A text semantic element recognition method based on enhanced CRF layer, characterized in that: include: Step S1, performing sample screening analysis on the sample library to obtain a number of training samples; Step S2, for a single training sample, extracting semantic elements from the training sample according to a preset strategy function to obtain a label sequence; Step S3, calculating the cumulative discount reward R corresponding to the tag sequence; Step S4, using a preset objective function to obtain the expected value of the probability reward corresponding to the label sequence, using a preset gradient formula to perform gradient calculation, and updating the model parameters; Step S5, determining whether to reselect a tag according to a preset exploration strategy; Step S6, repeating steps S2 to S5, and stopping model training and learning under a preset stop condition; The preset strategy function is π(a|s;θ), where s is a given state, a is the probability of selecting the next label, and θ is a model parameter; The preset objective function is: J(θ)=E[logπ(a|s;θ)*R]; The preset gradient formula is: ∇ θ J(θ)=(R−b)∇ θ logπ(a∣s;θ); Where b is the benchmark function; The preset exploration strategies include: Detect the current execution probability ε; If the execution probability ε is less than the preset execution probability, an unused label is randomly selected and reselected; If the execution probability ε is greater than or equal to the preset execution probability, the current label is selected as the best label; Among them, the calculation formula of execution probability ε is: ε=εmin+(εmax−εmin)×exp(−λ×m); Among them, m is the number of iterations, λ is the decay rate, εmin is the minimum exploration probability, and εmax is the maximum exploration probability; Under the preset adjustment conditions, the model action uniform state is detected, and when the model action uniform state is in the preset model action uniform state, the preset objective function is converted; The preset objective function after conversion is: J(θ)=E[logπ(a|s;θ)* R]+P; Where P = α * H(π), α is the regularization coefficient, and H(π) = −∑aπ(a|s;θ)logπ(a|s;θ).

2. The text semantic element recognition method based on the enhanced CRF layer according to claim 1 is characterized in that: The cumulative discount reward is R, and the calculation formula of R is: Where n is the current time step number, is the discount factor, 0< <1, is the immediate reward corresponding to the t-th time step, t≤n.

3. The text semantic element recognition method based on the enhanced CRF layer according to claim 2 is characterized in that, The value of the regularization coefficient is determined according to the reward parameter value; The regularization coefficient is negatively correlated with the loss fluctuation value.

4. The text semantic element recognition method based on the enhanced CRF layer according to claim 3 is characterized in that: The sample screening analysis includes: Randomly select a preset number of samples to be selected; Test the sample feature richness; If the sample feature richness is within the first preset sample feature richness range, increasing the number of training samples; If the sample feature richness is within a second preset sample feature richness range, it is determined that the training sample meets the training requirements.

5. The text semantic element recognition method based on the enhanced CRF layer according to claim 4 is characterized in that: The sample feature richness is determined based on the text segmentation reference value and the number of text characters of each sample to be selected.

Images