Tax-related legal text-oriented named entity recognition dependent enhancement method

Abstract

The application discloses a kind of tax-related legal text-oriented named entity recognition dependent enhancement method, comprising: tax named entity recognition is regarded as span classification task, a large number of spans are enumerated from input text by sliding window, and the deep representation of each span is generated by feature splicing method;Introduce a contrastive learning loss, and the contrast relationship is mined from highly overlapping span;Scale transformation mechanism is used to realize span interaction, and the geometric information of each candidate span is embedded in native span representation, to encode the interaction dependency between spans.The application converts tax named entity recognition into span classification task, and fully mines the interaction dependency between entities, realizes strong inference relationship, and introduces contrastive learning to improve the discrimination degree between different types of highly overlapping entities, so that the named entity in tax legal text can be more accurately and reasonably recognized, laying a foundation for downstream tasks such as tax incentives.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and tax control technology, and specifically relates to a named entity recognition dependency enhancement method for tax-related legal texts. Background Technology

[0002] Various tax incentives have detailed regulations regarding eligibility criteria, reduction methods, and preferential amounts, leading to low understanding of policies and insufficient benefits for businesses. Therefore, leveraging intelligent technologies to help businesses identify which policies they are eligible for and how much tax benefit they can receive is of great significance. This can better assist taxpayers in enjoying tax incentives and help businesses improve operational efficiency, making it necessary to conduct research on tax incentive matching calculations.

[0003] In the research on tax incentive matching and calculation, determining the beneficiary involves the identification of named entities in tax law, which are proper nouns in tax-related legal texts that contain specific information, such as taxpayer, taxable object, tax type, and tax enforcement type. Named entity identification in tax law is a key technology in tax incentive matching and calculation. Failure to accurately identify named entities in tax-related legal texts will lead to incorrect results in tax incentive matching calculations, resulting in situations where benefits are enjoyed when not due, or benefits are not enjoyed when due. Therefore, how to utilize natural language processing technology to identify named entities representing specific information in tax law texts has become an urgent problem to be solved.

[0004] Currently, no relevant research has proposed a solution for named entity recognition in tax law texts. The publicly available technologies aim to establish methods for named entity recognition in tax-related legal texts; representative works include:

[0005] Reference 1: A Chinese Named Entity Recognition Method and System Based on Deep Neural Networks (202210417839.X)

[0006] Reference 2: A Chinese Named Entity Recognition Method in the Medical Field (202210268640.5)

[0007] Reference 3: A Legal Named Entity Recognition Method and System Based on Cascaded Model and Data Augmentation (202110828255.7)

[0008] Reference 1 designs a Chinese named entity recognition method and system based on deep neural networks. It obtains the dependency relationship by acquiring the bidirectional short word memory model BiLSTM network and calculating the dependency relationship based on the self-attention mechanism, and then performs label prediction through conditional random field.

[0009] Reference 2 designs a Chinese named entity recognition method in the medical field. It adopts the BBCPR model and MCBERT encoder, encodes the fusion embedding through the BiLSTM layer to obtain the final implicit representation of the input sequence, and finally decodes the output of the BiLSTM layer based on the CRF layer to obtain the label sequence and output it.

[0010] Reference 3 designs a legal named entity recognition method based on cascaded models and data augmentation. It constructs a cascaded model through data augmentation, uses a BiLSTM bidirectional long short memory neural network to process the vector representation of fused word features, extracts deep features of the text, and uses two CRF conditional random fields for decoding to obtain entity sequences and attribute sequences, which are then concatenated to obtain the final label representation. The method then judges and outputs the legal entities in the legal documents.

[0011] While the aforementioned traditional methods can solve the problem of named entity recognition in specific domains, it is difficult to directly extend them to the recognition of named entities in tax-related legal texts. This is because named entities in tax-related texts are nested and have strong inferential relationships with each other.

[0012] First, named entities in tax texts are often nested. Nested entities refer to the phenomenon where one entity contains at least one other entity. For example, in the statement "Energy service companies implementing qualified energy performance contracting projects are temporarily exempt from value-added tax," "energy service company" is defined as a "taxpayer" type entity, while "energy service" is defined as an "industry" type entity. These two entities provide different semantic information, indicating different tax-related elements within a tax law provision—that is, what kind of taxpayer the provision applies to and what industry the taxpayer is engaged in. Ignoring either one leads to inaccurate matching of preferential policies. The three documents mentioned above only address conventional named entity recognition methods, which, as the name suggests, do not consider nested entities. When nesting occurs, the inner or outer entity is forcibly deleted, sacrificing some semantic information contained within the entity to reduce the difficulty of recognition. The consequence of this is the loss of some clearly helpful entities and their contained semantic information, compromising the completeness of information extraction, hindering downstream tasks such as preferential policy matching, and even causing errors during preferential policy matching. This is unacceptable in the tax field, where high accuracy is required.

[0013] Secondly, strong inferential relationships exist between nested named entities in tax-related texts. Different entities in the text do not exist independently; that is, there are interactive dependencies between entities, which are more pronounced between nested entities. For example, in the statement "Energy service companies implementing qualified energy performance contracting projects are temporarily exempt from value-added tax," both "energy service company" and "energy service" are named entities. The former is of the "taxpayer" type, while the latter is more likely of the "industry" type, rather than the "tax type" type, which is determined by the semantics of the text. From another perspective, once one entity in a nested structure is identified, it can assist in identifying another entity, meaning there is some kind of interactive dependency between nested entities. None of the three documents mentioned above provide effective technical solutions for the interactive dependencies between entities, which impairs the recognition effectiveness of named entities in tax-related legal texts.

[0014] Third, as mentioned earlier, entities in tax-related texts are often nested, leading to category confusion. In a nested structure (containing several nested entities and their corresponding text paragraphs), different types of entities share the same context. Even multiple entities of different types can have very similar characteristics, making it difficult for the model to identify them. A direct result is that "energy service company" might be identified as an "industry type," while "energy service" might be considered a "taxpayer" type. Similarly, the aforementioned literature does not provide an effective technical solution to this problem.

[0015] In summary, named entities in tax-related legal texts are characterized by nested structures, strong inferential relationships between them, and low distinguishability among nested entities of the same type. These issues have become urgent problems to be solved in entity recognition of tax-related legal texts. Summary of the Invention

[0016] This invention aims to provide a dependency enhancement method for named entity recognition in tax-related legal texts, namely a tax named entity recognition method based on span interaction and representation comparison. The method models tax named entity recognition as span classification, proposes a scaling transformation mechanism to mine the interaction dependencies between entities, and introduces a supervised span comparison learning method to mine the comparison relationships from highly overlapping spans, thereby constructing a tax named entity recognition model.

[0017] The present invention is achieved using the following technical solution:

[0018] A dependency enhancement method for named entity recognition in tax-related legal texts includes:

[0019] First, span enumeration generation and entity detection are performed. Tax named entity recognition is treated as a span classification task. A large number of spans are enumerated by sampling on the input text through a sliding window, and a deep representation of each span is generated through feature concatenation. Second, contrastive relationship mining of highly overlapping spans is performed. A contrastive learning loss is introduced to mine contrastive relationships from highly overlapping spans, enhancing the model's ability to distinguish between different types of highly overlapping entities. Third, interaction dependency mining between spans is performed. A scaling mechanism is used to realize span interaction. Interaction dependency features are mined from the geometric information of candidate spans and embedded into the native span representation, enhancing the native span representation. Finally, a joint loss function is constructed based on entity detection loss, entity classification loss, and contrastive learning loss, and the model is jointly trained end-to-end.

[0020] A further improvement of the present invention is that the method specifically includes the following implementation steps:

[0021] 1) Span Enumeration Generation and Entity Detection

[0022] This method deals with tax-related legal texts. First, it needs to perform text representation, converting the tax-related legal texts described in natural language into low-dimensional dense real-valued vectors. Then, it performs sliding window enumeration on the text to obtain each span as an entity candidate, and converts each entity candidate into a vector representation. After obtaining the entity candidates, it performs entity detection to determine whether a candidate span is mentioned as an entity. If an entity is mentioned, it can be considered as a text fragment of an entity, but its type is not yet known. The identification of the entity type is carried out in subsequent steps.

[0023] 2) Mining of highly overlapping span comparison relationships

[0024] For highly overlapping spans, a span-level supervised contrastive learning objective is introduced; under the guidance of entity detection labels, contrastive learning loss is used to find commonalities among similar samples, bringing the semantic distance between these similar samples closer and making the semantic distance between these samples and samples of different classes increasingly farther, thereby improving the ability to identify highly overlapping but different entities.

[0025] 3) Mining cross-span interaction dependencies

[0026] The study explores the interactive dependencies between spans and uses these dependencies to enhance the native span representation. Specifically, it scales the discrete geometric features of the span to convert them into relative displacement and width ratio, embeds this transformation into a high-dimensional space using a scaling mechanism, and fuses it with the native span representation through a self-attention mechanism to enhance the native span representation.

[0027] 4) End-to-end joint training

[0028] Based on the forward propagation results obtained from the aforementioned steps, a multi-classifier for tax named entity recognition is trained using joint loss to achieve the recognition of tax named entities.

[0029] A further improvement of this invention is that, in step 1), the span enumeration generation and entity detection specifically include the following steps:

[0030] Step 1. Jointly embed text representation

[0031] Text representation refers to the vectorization of tax-related legal texts, transforming natural language texts into low-dimensional dense real-valued vectors so that they can be input into the model for computation.

[0032] Step 2. Candidate Span Generation

[0033] First, based on a predefined length L, enumerate all spans in the text whose length is less than or equal to L; all candidate spans form a set: s set ={s1,s2,…,s m}, where s i = (start, ebn) represents the i-th candidate span; secondly, by calculating the intersection-over-union (IoU) ratio between candidate spans and real entities, spans with an IoU ratio satisfying a predefined threshold are considered training samples; IoU is used to measure the degree of overlap between spans, defined as... Here, A and B are two spans; if the IoU between the real entity and multiple spans meets a predefined threshold, then the span with the largest IoU among these spans is selected as the training sample; for such a training sample, it is assigned the same label as the associated real entity. And calculate the boundary offset between them. Other spans are discarded and not used in training;

[0034] Step 3. Entity Inspection

[0035] First, the span is represented, and then converted into a distributed representation for a span sample s. i = (start, end), where start and end refer to the indices of the start and end characters of the span. The boundary character representation is concatenated with the max-pooled representation of the characters within the span to obtain the span representation [S1, S2, ..., S]. m Then, span detection is performed based on a multilayer perceptron (MLP):

[0036]

[0037] Inner pool =MaxPooling(h start ,…,h end )

[0038]

[0039] in Indicates the splicing operation, MLP det This represents an MLP classifier that performs span detection.

[0040] A further improvement of the present invention is that, in Step 1 of step 1), the specific implementation steps are as follows: First, the BERT language model is used as a text encoder to obtain the context embedding; in order to enhance the context embedding, word embedding, part-of-speech embedding and character-level embedding are concatenated to the context embedding to obtain the combined embedding; finally, the combined embedding is input into Bi-LSTM to obtain the final text representation; word embedding, part-of-speech embedding and character-level embedding are all obtained from pre-released third-party persistent vector files.

[0041] A further improvement of this invention is that, in step 2), the mining of high-overlapping span comparison relationships specifically includes the following steps:

[0042] Step 1: Construct positive and negative sample pairs

[0043] Supervised contrastive learning loss (SCL) is introduced on the span-level representation to enhance the model's ability to identify highly overlapping spans. Specifically, guided by the span detection label, the SCL loss seeks commonalities among positive samples and widens the distance between them and negative samples. Positive samples refer to spans that are detected as mentioned by entities. To achieve this, positive and negative sample pairs are constructed.

[0044] Step 2. N-paris supervised contrastive learning

[0045] Similar to unsupervised learning, contrastive loss requires constructing pairs of positive and negative samples:

[0046]

[0047] Among them, s i and Treat each other as positive / negative samples;

[0048] Calculated using the mini-batch N-pairs loss method and Loss of contrastive learning under supervised conditions:

[0049]

[0050] Here, τ represents the temperature coefficient, used to adjust the degree of attention given to different samples. It is a distributed representation of the positive and negative comparison pairs constructed in Step 1; sim(h1,h2) is the similarity function, defined as:

[0051] A further improvement of this invention is that, in Step 1 of step 2), after step 1, it is already known whether a span is predicted to be an entity mention, i.e., each span will be assigned a label: True or False; for spans s detected as entity mentions... i and its extended equivalents They will treat each other as positive samples, and together they will be constructed as a positive sample pair. SCL makes their semantic distance closer; it generates augmented peers through Dropout, i.e. When executing Dropu, h i It is s i Distributed representation, output It is a dense vector with the same dimension as the input; for negative sample pairs, an s i Detected as an entity, another span If detected as a non-entity, these two variables will be considered as negative samples to each other and together they will be constructed as a negative sample pair. Under the effect of contrastive learning loss, the semantic distance between two samples in a negative sample pair will become increasingly greater.

[0052] A further improvement of this invention is that, in step 3), the inter-span interaction dependency mining specifically includes the following steps:

[0053] Step 1. Span Scale Transformation

[0054] Given a span representing a set S r =[S′1,S′2,…S′ i …,S′ m and span boundary set S b =[(s1,e1),(s2,e2),…(s i ,e i )...,(s m ,e m ]], perform a scaling operation:

[0055]

[0056] Where 0≤i,j≤m,x i S′ i The central position, w i S represents i width, This indicates a series operation; st is the result of scaling the center position and width of the span, resulting in two correlation matrices to describe the relative displacement and overlap between spans, which imply the interactive dependency between spans.

[0057] Step 2. Scale Transformation Embedding

[0058] The scaling transformation *st* is embedded into the high-dimensional representation using sine and cosine functions:

[0059] st 2i =sin(st / 10000) 2i / d )

[0060] st 2i+1 =cos(st / 10000) 2i / d )

[0061] Where d represents the dimension of the embedding vector;

[0062] The embedded representation is then subjected to a linear transformation and activated by ReLU:

[0063]

[0064] Step 3. Enhance native span representation

[0065] Interactions between bridging joints are based not only on their geometric information but also on their semantic information; therefore, span representation computes semantic interactions through a classic self-attention mechanism.

[0066]

[0067] Calculate the impact of other spans:

[0068]

[0069] Finally, the final enhanced span representation is obtained by weighting the span representations. The enhanced representation of the i-th span is:

[0070]

[0071] After obtaining the augmented representation, entity classification is performed based on a multilayer perceptron (MLP) to determine the type of the entity.

[0072]

[0073] MLP ent This represents an MLP classifier that performs entity classification.

[0074] A further improvement of this invention is that, in step 4), the end-to-end joint training specifically includes the following steps:

[0075] First, to differentiate the contributions of different spans in the loss calculation, the loss is weighted according to their intersection-union (IoU) scores; for the i-th span b i Weight w i The calculation is as follows:

[0076]

[0077] Where α∈{α1,α2} represents the IoU threshold, e i b i The relevant real entities, η is a focusing parameter used to adjust the rate at which certain sample weights smoothly decrease;

[0078] In span-based methods, span enumeration generates a large number of negative samples, far exceeding the number of positive samples, leading to sample imbalance. Therefore, Focal Loss is used in span detection, and this loss function has been proven to effectively alleviate the sample imbalance problem.

[0079]

[0080] Among them, w i λ represents the weighting of the i-th span, and λ represents the focus parameter of Focal Loss;

[0081] For entity classifiers, only cross-entropy loss is used:

[0082]

[0083]

[0084] Where w i This represents the weighting of the i-th span, while It is an MLP that performs entity classification in step 3. ent The output;

[0085] Finally, the span-level contrastive learning loss, span detection loss, and entity classification loss are jointly trained; the total loss is calculated as follows:

[0086]

[0087] Where λ1, λ2, and λ3 are the respective weights of the loss.

[0088] The present invention has at least the following beneficial technical effects:

[0089] This invention addresses the problem of named entity recognition in tax-related legal texts, proposing a dependency enhancement method for named entity recognition in tax-related legal texts. Compared with existing technologies, the advantages of this invention are:

[0090] (1) This invention can solve the problem of nested named entity recognition in tax-related legal texts. Existing technologies generally only target conventional named entity recognition methods, which have the drawback of losing some semantic information contained in the entities. This invention models named entity recognition in tax-related legal texts as a span classification problem, classifying different text fragments independently, and can identify nested named entities.

[0091] (2) This invention mines the interactive dependencies between nested entities. These interactive dependencies can assist in inferring the type of entities. Existing technologies do not provide effective technical solutions for the interactive dependencies between entities, which can impair the recognition effect of named entities in tax-related legal texts. This invention enhances the original entity representation by mining the dependency interactions from the geometric features of entities.

[0092] (3) This invention addresses the problem of low discriminative power for highly overlapping samples in nested entities. In a nested structure (containing several nested entities and the text paragraphs in which they appear), different types of entities share the same context. Even multiple entities of different types can have very similar features, making it difficult for the model to identify them. Existing technologies do not provide effective solutions to this problem. This invention constructs contrastive samples at the span level and introduces supervised contrastive learning to effectively mine the contrastive relationships of the same type in nested structures, thereby enhancing the model's discriminative power for highly overlapping samples in nested structures. Attached Figure Description

[0093] Figure 1 This is a flowchart of the overall framework.

[0094] Figure 2 The flowchart for span enumeration generation and entity detection is provided.

[0095] Figure 3 Flowchart for mining high-overlapping span comparison relationships.

[0096] Figure 4 A flowchart for mining interaction dependencies across spans.

[0097] Figure 5 This is a diagram of the Transformer model based on the scaling mechanism.

[0098] Figure 6 The diagram shows a tax-related named entity recognition algorithm model based on span interaction and representation comparison. Figure 6 (A) is the backbone network. Figure 6 (B) represents Span-level representation contrastive learning. Figure 6 (C) represents a Transformer based on scaling. Detailed Implementation

[0099] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanyingdrawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can berealized in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0100] In this embodiment, all texts related to tax reduction and fee reduction policies issued since January 1, 2022 are selected from the tax policy catalog on the official website of the State Taxation Administration and saved as local text files. The entity types of tax-related legal texts are predefined, specifically including: taxpayers, goods or services, tax-related actions, tax types and industries, locations, times, amounts, purchasers, tax rates, etc. These eight categories are used as the data for the embodiments of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0101] As Figure 1 shown, in the specific implementation of the present invention, a method for enhancing named entity recognition for tax-related legal texts specifically includes the following steps:

[0102] Step 1. Span enumeration generation and entity detection

[0103] As Figure 2 shown, convert the text described in natural language into a distributed numerical representation, generate spans in an enumeration manner, and perform entity detection, specifically including the following steps:

[0104] S101. Joint embedding text representation

[0105] Delete illegal characters such as special symbols, paragraph separators, policy numbers, and specific abbreviations in the legal policy text. In this embodiment, the legal provisions are segmented into sentences according to full stops or semicolons. Taking a preferential policy as an example, such as "Energy conservation service companies implementing eligible contract energy management projects are temporarily exempt from value-added tax", segment it into a single-word sequence at the single-character level, and the single-word sequence {节, 能, 服, 务, 公, 司, 实, 施, 符, 合, 条, 件, 的, 合, 同, 能, 源, 管, 理, 项, 目, 暂, 免, 征, 收, 增, 值, 税} can be obtained. Among them, the single character is the smallest segmentation unit, and the entities included are: {"节能服务": "行业", "节能服务公司": "纳税人", "增值税": "税种", "合同能源管理项目": "商品或服务"}.

[0106] This embodiment uses the BERT language model as the text encoder to obtain the context embedding vector for each word, with a dimension of 1024. To enhance the context embedding, the word embedding (100-dimensional), part-of-speech (POS) embedding (50-dimensional), and character-level embedding (50-dimensional) for each word are concatenated with the context embedding to obtain a combined embedding with a dimension of 1224. Finally, the combined embedding is input into a Bi-LSTM, and the output of the Bi-LSTM serves as the final text representation. The word embedding, POS embedding, and character-level embedding are all obtained from pre-released third-party persistent vector files.

[0107] S102. Candidate Span Generation

[0108] First, based on a predefined length L, this embodiment enumerates all spans in the text whose length is less than or equal to L. In this embodiment, the value of L is set to 15, which covers almost all lengths of entities. All candidate spans form a set: s set =[s1,s2,…,s m ], where s i = (start, end) represents the i-th candidate span. In this embodiment, the obtained span set is: {"Save", "Energy Saving", "Energy Saving Service", "Energy Saving Service"...}. Next, the intersection-over-union (IoU) ratio between the candidate spans and the real entities is calculated. Spans with an IoU that satisfy a predefined threshold are considered training samples. In this embodiment, the IoU threshold is set to 0.75. If the IoU between the real entity and multiple spans all satisfy the predefined threshold, the span with the largest IoU among these spans is selected as the training sample. For such a training sample, it is assigned the same label as the associated real entity. And calculate the boundary offset between them. In this embodiment, entities are used as known information to provide supervisory information for model training. Given the entity {"Energy Saving Service Company": "Taxpayer"}, a span set is generated through enumeration. The IoU is calculated by traversing this span set. For example, if the intersection-union ratio (IoU) of {"Energy Saving Service Company"} and {"Energy Saving Service Company": "Taxpayer"} is 1, which is greater than 0.75, then the label for the span {"Energy Saving Service Company"} is "Taxpayer", and the offset is 0. Other spans are discarded and do not participate in training.

[0109] S103. Entity Inspection

[0110] The purpose of entity detection is to determine whether a candidate span is an entity mention. An entity mention refers to an entity whose type is currently unknown; we only know that a text fragment (i.e., a span) is an entity, but we don't yet know its type. First, the span is represented by converting it into a distributed representation. For a span sample s...i =(start, end), where start and end refer to the indices of the start and end characters of the span. The splicing boundary characters of the present invention represent the maximum pooling representation of the characters inside the span to obtain the span representation [S1, S2, …, S m . In this embodiment, taking the span {"energy conservation service company"} as an example, the representation of this span is the representation of the single character "jie", spliced with the maximum pooling result of the representation of "neng fu wu gong", and then spliced with the representation of the single character "si". The dimension of the span is 3 times the joint embedding representation in S101, that is, 3672, so a span representation with a dimension of 3672 is obtained.

[0111] Then, the 3672-dimensional span representation is input into a multi-layer perceptron (MLP) for entity detection. The multi-layer perceptron is composed of multiple fully connected layers. In this embodiment, the multi-layer perceptron includes three fully connected layers, and the number of nodes is (3672, 1000, 100, 2) in sequence. This multi-layer perceptron constitutes a binary classifier, and the output dimension is 2. The recognition index is determined through Argmax.

[0112] Step 2. Mining the contrastive relationship of highly overlapping spans

[0113] As Figure 3 shown, a supervised contrastive learning loss (SCL) is introduced in the span-level representation to enhance the model's ability to identify highly overlapping spans. Specifically, the SCL loss can, under the guidance of the span detection label (obtained in Step 1), find the commonalities between positive samples and pull the distance between them and negative samples farther apart. The specific steps are as follows:

[0114] S201. Constructing positive and negative sample pairs

[0115] After Step 1, it is already known whether a span is predicted as an entity mention, that is, each span will be assigned a label: True or False. For example, {"energy conservation service company"} is True, while {"neng fu wu gong"} is False. For the span {"energy conservation service company"} detected as an entity mention and its augmented equivalent {"energy conservation service company"} + , they will regard each other as positive samples, and the two are jointly constructed into a positive sample pair {({"energy conservation service company"}, {"energy conservation service company"} + )}. SCL will make their semantic distances closer.

[0116] In the embodiment, an augmented equivalent of a span sample is generated by randomly discarding some features through Dropout, that is When performing Droput, h i is the span s iThe distributed representation, generated by S103 in step 1, outputs... It is an augmented dense vector with the same dimension as the input. For negative sample pairs, an s i Detected as an entity, another span If detected as a non-entity, these two variables will be considered as negative samples to each other and together they will be constructed as a negative sample pair. Under the effect of contrastive learning loss, the semantic distance between two samples in a negative sample pair will become increasingly greater.

[0117] S202. N-paris Supervised Comparative Learning

[0118] Similar to unsupervised learning, contrastive loss requires constructing pairs of positive and negative samples:

[0119]

[0120] Among them, s i and Treating each other as positive / negative samples, this embodiment only includes spans where the IoU meets the threshold.

[0121] Calculated using the mini-batch N-pairs loss method and Loss of contrastive learning under supervised conditions:

[0122]

[0123] Here, τ represents the temperature coefficient, used to adjust the degree of attention given to different samples. It is a distributed representation of the positive and negative comparison pairs constructed in Step 1; sim(h1,h2) is the similarity function, defined as:

[0124] Step 3. Mining Inter-span Dependency Relationships

[0125] like Figure 4 As shown, we perform cross-span interaction dependency mining, extracting interaction relationships from the geometric features of the span and embedding them into a high-dimensional space. These relationships are then fused into the native span representation using a self-attention mechanism to enhance the span representation. Specifically, this includes the following steps:

[0126] S301. Span Scale Transformation

[0127] In this embodiment, step 1 performs span enumeration to generate a candidate span set, and step S103 performs span detection to obtain the detected span set, denoted as S. r =[S′1,S′2,…S′ i …,S′m ], and the span boundary set S b =[(s1,e1),(s2,e2),…(s i ,e i )...,(s m ,e m Perform a scaling operation on the above set:

[0128]

[0129] Where 0≤i,j≤m,x i S′ i The central position, w i S′ i width, This indicates a serial operation.

[0130] In this embodiment, for example, two spans, "Energy Saving Service" and "Energy Saving Service Company," are scaled by their respective center index positions and widths to obtain st, which indicates the scale transformation results of the span center position and width. This yields two correlation matrices: a width correlation matrix and a center offset correlation matrix. Figure 5 As shown, this is used to describe the relative displacement and overlap between spans, which implies the interactive dependency between spans.

[0131] S302. Scale Transform Embedding

[0132] like Figure 5 As shown, after obtaining the correlation matrix through scaling transformation in S301, the scaling transformation st is then embedded into the high-dimensional representation using sine and cosine functions:

[0133] st 2i =sin(st / 10000) 2i / d )

[0134] st 2i+1 =cos(st / 10000) 2i / d )

[0135] Where d represents the dimension of the embedding vector, which is 100 in this embodiment.

[0136] The embedded representation is then subjected to a linear transformation and activated by ReLU:

[0137]

[0138] In this embodiment The dimension is 1024, representing the interaction features between the i-th span and the j-th span.

[0139] S303. Enhance native span representation

[0140] Interactions between spans are based not only on the geometric information of the spans (e.g., position and width) but also on semantic information. Therefore, span representation computes semantic interactions through a classic self-attention mechanism:

[0141]

[0142] Now, let's calculate the impact of other spans:

[0143]

[0144] Finally, the final enhanced span representation can be obtained by weighting the span representations. The enhanced representation of the i-th span is:

[0145]

[0146] After obtaining the enhanced representation, an MLP classification device is input to perform entity classification and determine the type of the entity. This device is composed of multiple fully connected layers. In this embodiment, the multilayer perceptron includes three fully connected layers with the number of nodes being (3672, 1000, 100, 8) in sequence. This multilayer perceptron forms a multi-classifier with an output dimension of 8. The recognition index is confirmed by Argmax, and the final recognition result is confirmed by Argmax compared with the category preset table.

[0147] Step 4. End-to-end joint training

[0148] The networks from steps 1, 2, and 3 are concatenated to form a training device. The network parameters of the training device are initialized using a standard random initialization method to ensure that the network parameters follow a Gaussian distribution. End-to-end training is performed based on a joint loss. The entire training process is implemented using the Python + PyTorch deep learning platform and includes the following steps:

[0149] First, to differentiate the contributions of different spans in the loss calculation, similar to existing work, this embodiment weights the loss based on their intersection-over-union (IoU) scores. A weight is calculated for each span to weight the loss. In this embodiment, the specific calculation method for the weighting is as follows:

[0150] For the i-th span b i Weight w i The calculation is as follows:

[0151]

[0152] Where α∈{α1,α2} represents the IoU threshold, e i bi The relevant real entities, η is a focusing parameter used to adjust the rate at which certain sample weights smoothly decrease.

[0153] In span-based methods, span enumeration generates a large number of negative samples, far exceeding the number of positive samples, leading to sample imbalance. Therefore, this embodiment uses Focal Loss in span detection to reduce the proportion of a large number of negative samples, smooth out the model bias caused by sample imbalance, and prevent the model from "going astray" during training. This loss function has been proven to effectively alleviate the sample imbalance problem.

[0154]

[0155] In this embodiment, the forward propagation probability of entity detection is After receiving the loss from entity detection, Focal Loss reassigns different weights to reduce the proportion of negative samples and alleviate sample imbalance.

[0156] In this embodiment, cross-entropy loss is used as the entity classification loss, and the input of this loss is the MLP that performs entity classification in Step 3. ent The output.

[0157] Finally, the span-level contrastive learning loss, span detection loss, and entity classification loss are jointly trained. The total loss is calculated as follows:

[0158]

[0159] Where λ1, λ2, and λ3 are the respective weights of the loss. In step 2, S202 calculates that, in this embodiment, the sum of λ1, λ2, and λ3 is 1, and is set to 0.3, 0.3, and 0.4 respectively. The parameters are updated using a mini-batch gradient descent algorithm, and parameter optimization uses the Adam optimizer, which is easy to use, has a fast descent speed, and is easy to deploy in a real production environment.

[0160] Those skilled in the art will readily understand that the above description is merely an embodiment of the method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dependency enhancement method for named entity recognition in tax-related legal texts, characterized in that, include: First, span enumeration and entity detection are performed. Tax named entity recognition is treated as a span classification task. A large number of spans are enumerated by sampling on the input text through a sliding window, and a deep representation of each span is generated through feature concatenation. Second, high-overlapping span contrast relationship mining is performed. A contrastive learning loss is introduced to mine contrast relationships from highly overlapping spans, enhancing the model's ability to distinguish between different types of highly overlapping entities. Third, inter-span interaction dependency mining is performed. A scaling mechanism is used to realize span interaction. Interaction dependency features are mined from the geometric information of candidate spans and embedded into the native span representation, enhancing the native span representation. Finally, a joint loss function is constructed based on entity detection loss, entity classification loss, and contrastive learning loss, and the model is jointly trained end-to-end. The specific implementation steps of this method are as follows: 1) Span Enumeration Generation and Entity Detection This method deals with tax-related legal texts. First, it needs to perform text representation, converting the tax-related legal texts described in natural language into low-dimensional dense real-valued vectors. Then, it performs sliding window enumeration on the text to obtain each span as an entity candidate, and converts each entity candidate into a vector representation. After obtaining the entity candidates, it performs entity detection to determine whether a candidate span is mentioned as an entity. If an entity is mentioned, it can be considered as a text fragment of an entity, but its type is not yet known. The identification of the entity type is carried out in subsequent steps. 2) Mining of highly overlapping span comparison relationships For highly overlapping spans, a span-level supervised contrastive learning objective is introduced. Guided by entity detection labels, contrastive learning loss is used to find commonalities among similar samples, narrowing the semantic distance between these similar samples and increasing the semantic distance between these samples and samples of different classes, thereby improving the ability to identify highly overlapping but different entities. The high-overlapping span contrastive relationship mining specifically includes the following steps: Step 1. Construct positive and negative sample pairs Supervised contrastive learning loss (SCL) is introduced on the span-level representation to enhance the model's ability to identify highly overlapping spans. Specifically, guided by the span detection label, the SCL loss seeks commonalities among positive samples and widens the distance between them and negative samples. Positive samples refer to spans that are detected as mentioned by entities. To achieve this, positive and negative sample pairs are constructed. Step 2. N-paris supervised contrastive learning; 3) Mining cross-span interaction dependencies The study explores the interactive dependencies between spans and uses these dependencies to enhance the native span representation. Specifically, it scales the discrete geometric features of the span to convert them into relative displacement and width ratio, embeds this transformation into a high-dimensional space using a scaling mechanism, and fuses it with the native span representation through a self-attention mechanism to enhance the native span representation. 4) End-to-end joint training Based on the forward propagation results obtained from the aforementioned steps, a multi-classifier for tax named entity recognition is trained using joint loss to achieve the recognition of tax named entities.

2. The named entity recognition dependency enhancement method for tax-related legal texts according to claim 1, characterized in that, Step 1), span enumeration generation and entity detection specifically include the following steps: Step 1. Jointly embed text representation Text representation refers to the vectorization of tax-related legal texts, transforming natural language texts into low-dimensional dense real-valued vectors so that they can be input into the model for computation. Step 2. Candidate Span Generation First, based on a predefined length L, enumerate all spans in the text whose length is less than or equal to L; all candidate spans form a set: ,in Let represent the i-th candidate span; secondly, by calculating the Intersection over Union (IoU) between candidate spans and real entities, spans with an IoU satisfying a predefined threshold are considered training samples; IoU is used to measure the degree of overlap between spans, defined as... Here, A and B are two spans; if the IoU between the real entity and multiple spans meets a predefined threshold, then the span with the largest IoU among these spans is selected as the training sample; for such a training sample, it is assigned the same label as the associated real entity. And calculate the boundary offset between them. Other spans are discarded and not included in the training. Step 3. Entity Inspection First, the span is represented by converting it into a distributed representation for span samples. Here, start and end refer to the indices of the start and end characters of the span. The span representation is obtained by concatenating the boundary character representation with the max-pooled representation of the characters within the span. Then, span detection is performed based on a multilayer perceptron (MLP): in This indicates a splicing operation. This represents an MLP classifier that performs span detection.

3. The named entity recognition dependency enhancement method for tax-related legal texts according to claim 2, characterized in that, In Step 1 of Step 1), the specific implementation steps are as follows: First, use the BERT language model as a text encoder to obtain the context embedding; in order to enhance the context embedding, concatenate the word embedding, part-of-speech embedding and character-level embedding after the context embedding to obtain the combined embedding; finally, input the combined embedding into Bi-LSTM to obtain the final text representation; the word embedding, part-of-speech embedding and character-level embedding are all obtained from the pre-released third-party persistent vector files.

4. The named entity recognition dependency enhancement method for tax-related legal texts according to claim 2, characterized in that, In step 2), Step 2. N-paris involves supervised contrastive learning, including: Similar to unsupervised learning, contrastive loss requires constructing pairs of positive and negative samples: in, and / Treat each other as positive / negative samples; Calculated using the mini-batch N-pairs loss method and Loss of contrastive learning under supervised conditions: in, This represents a temperature coefficient, used to adjust the level of attention given to different samples. It is a distributed representation of the positive and negative comparison pairs constructed in Step 1; It is a similarity function, defined as: .

5. The named entity recognition dependency enhancement method for tax-related legal texts according to claim 4, characterized in that, In Step 1 of Step 2), after Step 1, it is already known whether a span is predicted to be an entity mention; that is, each span will be assigned a label: True or False. For spans detected as entity mentions... and its extended equivalents They will treat each other as positive samples, and together they will be constructed as a positive sample pair. SCL makes their semantic distance closer; it generates augmented peers through Dropout, i.e. When executing Dropu, yes Distributed representation, output It is a dense vector with the same dimension as the input; for negative sample pairs, one Detected as an entity, another If detected as a non-entity, these two variables will be considered as negative samples to each other and together they will be constructed as a negative sample pair. Under the effect of contrastive learning loss, the semantic distance between two samples in a negative sample pair will become increasingly greater.

6. The named entity recognition dependency enhancement method for tax-related legal texts according to claim 4, characterized in that, Step 3), the mining of cross-span interaction dependencies, specifically includes the following steps: Step 1. Scale transformation between spans Given a span, represent a set and span boundary set Perform a scaling operation: in , express The central location, express width, Indicates a serial operation; It is the result of the scale transformation of the center position and width of the span, from which two correlation matrices are obtained to describe the relative displacement and overlap between the spans, which contain the interactive dependency between the spans. Step 2. Scale Transformation Embedding Scaling transformation using sine and cosine functions Embedded into high-dimensional representation: in The dimension of the embedding vector; Then, the embedded representation undergoes a linear transformation and is derived from... ReLU activation: Step 3. Enhance native span representation Interactions between bridging joints are based not only on their geometric information but also on their semantic information; therefore, span representation computes semantic interactions through a classic self-attention mechanism. Calculate the impact of other spans: Finally, the final enhanced span representation is obtained by weighting the span representations. The enhanced representation of the i-th span is: After obtaining the augmented representation, entity classification is performed based on a multilayer perceptron (MLP) to determine the type of the entity. in Indicates the entity classification to be performed. MLP Classifier.

7. The named entity recognition dependency enhancement method for tax-related legal texts according to claim 6, characterized in that, Step 4) involves end-to-end joint training, specifically including the following steps: First, to differentiate the contributions of different spans in the loss calculation, the loss is weighted according to their Intersection over Union (IoU) scores; for the i-th span... Weight The calculation is as follows: in express IoU Threshold, express The relevant real entities, It is a focusing parameter used to adjust the rate at which the weights of certain samples decrease smoothly; In span-based methods, span enumeration generates a large number of negative samples, far exceeding the number of positive samples, leading to sample imbalance. Therefore, span detection utilizes... Focal Loss This loss function has been proven to effectively alleviate the problem of imbalanced samples: in, Indicates the first i Weighting of each span, express Focal Loss The focusing parameters; For entity classifiers, only cross-entropy loss is used: in This represents the weighting of the i-th span, while Entity classification is performed in step 3. The output; Finally, the span-level contrastive learning loss, span detection loss, and entity classification loss are jointly trained; the total loss is calculated as follows: in , and These are the respective weights of the losses.

Images