Sensitive information-oriented data enhancement method, system and equipment and medium
Mask prediction of named entities through E-MLM model solves the problem of existing data augmentation methods destroying semantic relationships in named entity recognition, improves the generalization ability and semantic understanding of the model, and achieves high-quality data augmentation effect .
Patent Information
- Application Number
- CN202411010677.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
Smart Images

Figure CN118940762A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing, and more specifically to a data enhancement method, system, device and medium for sensitive information. Background Art
[0002] Data augmentation plays a vital role in the field of natural language processing, especially in named entity recognition tasks. Its core purpose is to increase the diversity and generalization ability of model training by generating additional training samples, thereby improving the performance of the model on unseen data. Although existing data augmentation methods such as word-level modification and back-translation have enriched the training dataset to a certain extent, they have also exposed related problems, limiting the effectiveness and practicality of the technology.
[0003] Currently, named entity recognition is highly dependent on the precise matching of entities and their contexts. Traditional methods such as simple word replacement or addition may destroy the original semantic relationship, resulting in the generated samples being grammatically correct but not semantically consistent with the original sentence. For example, replacing a specific food entity may change the actual meaning of the entire sentence, and the model may learn the wrong entity-context correspondence, reducing recognition accuracy.
[0004] Some data augmentation methods, such as back-translation, rely on external word alignment tools for translation and alignment operations. However, these tools may not always be able to accurately identify and align named entities, which may cause the entity names or tags of the generated new samples to be inconsistent with the original samples, affecting model training. For example, "Apple is my favorite fruit" will generate "My favorite fruit is apple" after back-translation from Chinese to English. The positions of the corresponding fruit entity "apple" in the two sentences are not the same, and the entity label positions cannot be directly applied.
[0005] In addition, the lack of in-depth understanding of entity contexts leads to the neglect of the meaning and role of entities in specific contexts during the enhancement process. For example, simple character-level operations may not consider whether the replacement or modification conforms to the contextual logic of the original text, affecting the quality of the enhanced data.
[0006] While maintaining data diversity, avoiding excessive enhancement leading to increased noise and potential overfitting risks is a challenge faced by existing technologies. In particular, when the enhancement strategy is too aggressive, a large number of samples that deviate greatly from the original domain may be generated, and the model may learn the wrong pattern.
[0007] Therefore, how to design a data enhancement method for sensitive information to enhance the accuracy of entity recognition without compromising the authenticity and semantic coherence of the data is an urgent problem that technical personnel in this field need to solve. Summary of the invention
[0008] In view of this, the present invention provides a data enhancement method for sensitive information, which achieves the goal of generating diverse and rich training samples by masking named entities in original text and predicting their contents using the E-MLM model.
[0009] In order to achieve the above object, the present invention adopts the following technical solution:
[0010] In a first aspect, the present invention provides a data enhancement method for sensitive information, comprising:
[0011] S1. Construct a sensitive information dataset for named entity recognition, perform label sequence linearization processing on the sensitive information dataset, and obtain the sensitive information dataset after label sequence linearization processing; the label sequence linearization processing is: inserting entity labels into the original text sequence in the sensitive information dataset, and constructing a linear sequence in which entity labels and named entities coexist;
[0012] S2. Based on the sensitive information dataset after linearization of the label sequence, the RoBERTa-WWM model is used as a pre-training model, and a named entity-guided masking strategy is adopted to perform iterative training to obtain an E-MLM model;
[0013] S3. Input the data set to be processed into the E-MLM model to obtain the corresponding enhanced data set.
[0014] In S1, a sensitive information dataset for named entity recognition is constructed, including:
[0015] S11. Acquire sensitive information data; the sensitive information data is Internet platform data;
[0016] S12, preprocessing the sensitive information data to obtain preprocessed sensitive information data;
[0017] S13, segmenting the long text data in the pre-processed sensitive information data according to the punctuation mark priority to obtain segmented sensitive information data;
[0018] S14. Based on the segmented sensitive information data, define entity categories, use the BIOES annotation method to perform classification annotation, and obtain a sensitive information data set for named entity recognition.
[0019] Further, in S12, the preprocessing includes:
[0020] Invalid symbols are removed; paragraph tags, hyperlink tags and URL information in the sensitive information data are removed through parsing strategies; and text content that is not related to the preset sensitive information category is removed using regular expressions.
[0021] Furthermore, in S1, entity tags are inserted into the original text sequence in the sensitive information dataset to construct a linear sequence in which entity tags and named entities coexist, including:
[0022] Based on the original text sequence X=(x1, x2, ..., x n ), in x i Add labels indicating the head entity B- and the tail entity E- before and after, and add entity labels S- before and after each single-word entity; for non-entity markers O and entity internal markers I-, do not perform label sequence linearization processing;
[0023] Get a linear sequence of entity tags and named entities
[0024] Furthermore, in S2, the RoBERTa-WWM model is obtained by stacking multiple Transformer encoder layers; the Transformer encoder layer includes: a self-attention mechanism and a feedforward neural network;
[0025] In the training phase, the RoBERTa-WWM model adopts the WWM strategy to select the entire word for masking operation; the masking operation is dynamic masking; the dynamic masking is: before each iterative training, the linear sequence is masked based on the mask ratio η. Randomly select different entity words for mask replacement to obtain the sequence The sequence As input, iterative training:
[0026]
[0027] Among them, θ represents the RoBERTa-WWM model parameters, X represents the original text sequence, and x j Represents the named entity in the original text sequence X; if x j If masked, α i =1, otherwise α i =0.
[0028] Furthermore, the iterative training adopts the Adam optimizer, wherein the iteration period is set to 20, the batch size is set to 30, and the learning rate is set to 1e-5.
[0029] Furthermore, the S3 includes:
[0030] S31, randomly selecting some named entities in the text sequence in the data set to be processed to perform masking processing, and constructing a masked text sequence;
[0031] S32, based on the masked text sequence, generating a probability distribution including k candidate entities, and forming a vocabulary V;
[0032] S33, selecting the first m entities with the highest probability in the vocabulary V to construct a candidate entity set C; wherein m≤k;
[0033] S34, randomly selecting an entity from the candidate entity set C, replacing the mask position in the text sequence to generate a new text sequence;
[0034] S35: remove the labels used for training annotation in the new text sequence to obtain an enhanced sequence in text form.
[0035] In a second aspect, the present invention provides a data enhancement system for sensitive information, comprising:
[0036] Dataset construction and processing module: used to construct a sensitive information dataset for named entity recognition, perform label sequence linearization processing on the sensitive information dataset, and obtain the sensitive information dataset after label sequence linearization processing; the label sequence linearization processing is: inserting entity labels into the original text sequence in the sensitive information dataset, and constructing a linear sequence in which entity labels and named entities coexist;
[0037] E-MLM model training module: used to use the RoBERTa-WWM model as a pre-training model based on the sensitive information dataset after linearization of the label sequence, and adopt a named entity-guided masking strategy to perform iterative training to obtain an E-MLM model;
[0038] Enhanced data set generation module: used to input the data set to be processed into the E-MLM model to obtain the corresponding enhanced data set.
[0039] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data enhancement method when executing the computer program.
[0040] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the above-mentioned data enhancement method when executed by a processor.
[0041] The descriptions of the second to fourth aspects of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second to fourth aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.
[0042] It can be seen from the above technical solutions that, compared with the prior art, the data enhancement method provided by the present invention has the following beneficial effects:
[0043] The E-MLM model is a new model for data enhancement in named entity recognition tasks. By masking the named entities in the original text and predicting their content, the E-MLM model achieves the goal of generating diverse and rich training samples for named entity recognition tasks. By increasing the diversity of training data sets, E-MLM not only helps improve the generalization ability and robustness of named entity recognition models, but also helps the model better adapt to various scenarios and contexts in the real world.
[0044] Specifically, it includes but is not limited to the following aspects:
[0045] 1. Data enhancement and expansion: The E-MLM model effectively expands the training data set by automatically generating new samples with corresponding entity labels. This helps solve the problem of data scarcity or imbalance and improves the performance of natural language processing models in various scenarios.
[0046] 2. Improved generalization ability: E-MLM enhances the model’s understanding of high-level semantics and context by masking and predicting entities, alleviates the problem of vocabulary sparsity, and improves the semantic consistency of generated text, thereby improving the generalization ability of the named entity recognition model.
[0047] 3. Enhanced model robustness: The diverse entities generated by E-MLM can help the model better cope with noise, mislabeling, and other anomalies. By training the model to adapt to different data changes, E-MLM improves the robustness of the named entity recognition model to the input data, making it more stable and reliable. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0049] Figure 1 A flow chart of a data enhancement method for sensitive information provided by an embodiment of the present invention;
[0050] Figure 2A schematic diagram of a process for constructing a sensitive information dataset for named entity recognition provided by an embodiment of the present invention;
[0051] Figure 3 A schematic diagram of a linear sequence provided by an embodiment of the present invention;
[0052] Figure 4 A schematic diagram of the structure of the RoBERTa-WWM model provided in an embodiment of the present invention;
[0053] Figure 5 A schematic diagram of a data enhancement process based on an E-MLM model provided in an embodiment of the present invention;
[0054] Figure 6 A schematic diagram of the structure of a data enhancement system for sensitive information provided by an embodiment of the present invention;
[0055] Figure 7 A schematic diagram of the structure of an electronic device is provided for an embodiment of the present invention. DETAILED DESCRIPTION
[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] The data enhancement method provided in the embodiment of the present application can be applied to a data enhancement server. The above-mentioned data enhancement server can be hardware or software. When the data enhancement server is hardware, it can be implemented as a distributed server cluster that provides data enhancement services, or it can be implemented as a single server. When the data enhancement server is software, it can be installed in the servers listed above. It can be implemented as multiple software or software modules, or it can be implemented as a single software or software module, which is not specifically limited here.
[0058] Embodiment 1;
[0059] like Figure 1 As shown, this embodiment provides a data enhancement method for sensitive information, which is applied to named entity recognition, including:
[0060] S1. Construct a sensitive information dataset for named entity recognition, perform label sequence linearization processing on the sensitive information dataset, and obtain the sensitive information dataset after label sequence linearization processing; the label sequence linearization processing is: inserting entity labels into the original text sequence in the sensitive information dataset, and constructing a linear sequence in which entity labels and named entities coexist;
[0061] S2. Based on the sensitive information dataset after linearization of the label sequence, the RoBERTa-WWM model is used as a pre-training model, and a named entity-guided masking strategy is adopted to perform iterative training to obtain an E-MLM model;
[0062] S3. Input the data set to be processed into the E-MLM model to obtain the corresponding enhanced data set.
[0063] In this scheme, the named entity is used as the mask center based on the E-MLM model and combined with the whole word masking mechanism, which effectively enhances the diversity and quality of named entity recognition training data and improves the entity recognition accuracy of the natural language processing model under limited sample conditions.
[0064] The following is a further detailed description of the various steps and related features of the above data enhancement method:
[0065] like Figure 2 As shown, in this embodiment S1, a sensitive information dataset for named entity recognition is constructed, including:
[0066] S11. Obtain sensitive information data; the sensitive information data is Internet platform data; collecting data directly from the Internet platform ensures the reality and diversity of the data, covers a wide range of real-world scenarios, and is conducive to the model learning rich contextual features and entity expressions.
[0067] S12, preprocessing the sensitive information data to obtain preprocessed sensitive information data;
[0068] Preprocessing includes: removing invalid symbols; removing paragraph tags, hyperlink tags and URL information in the sensitive information data through parsing strategies; and using regular expressions to remove text content that is not related to the preset sensitive information category.
[0069] By removing invalid symbols, reducing noise interference, and improving data quality; removing paragraph tags, hyperlink tags, and URL information, ensuring that the model focuses on the substantive content of the text rather than format elements, and enhancing the model's understanding of text semantics; applying a regular expression screening mechanism, systematically excluding text that does not match the preset sensitive information type, making the data set highly concentrated on the target entity type, and optimizing the focus and efficiency of model training. This laid an accurate and efficient data foundation for subsequent named entity recognition tasks, and significantly enhanced the recognition accuracy and generalization ability of the model.
[0070] S13. Through punctuation priority, the long text data in the pre-processed sensitive information data is segmented to obtain the sensitive information data after segmentation; long text segmentation is performed according to punctuation priority to ensure the semantic integrity of the text fragments, while adapting to the text processing needs of different lengths and structures, so that the model can better understand and learn sentence structure and enhance the ability to understand complex texts.
[0071] S14. Based on the sensitive information data after the segmentation process, define entity categories, classify and annotate using the BIOES annotation method, and obtain a sensitive information dataset for named entity recognition. Among them, entity types are divided into eight categories, including: nationality (CONT), educational background (EDU), place name (LOC), name (NAME), organization name (ORG), profession (PRO), ethnicity (RACE), and title (TITLE).
[0072] Compared with the classic BIO (Begin, Inside, Outside) annotation method, the above BIOES annotation method adds clear identification of entity ends and single entities, which can handle entity boundary issues more flexibly, especially when entities overlap or nest. The meanings of each BIOES tag include: B (Begin): indicates the starting position of an entity, I (Inside): indicates the non-first and last parts of an entity, O (Outside): indicates a word that does not belong to any entity, E (End): indicates the end position of an entity, S (Single): indicates that a single word is a complete entity in itself without internal structure.
[0073] Through BIOES annotation, each word is given a label to indicate its role in entity recognition, thus helping the model understand and distinguish entity information in the text. It is suitable for processing complex entity structures and improving recognition accuracy.
[0074] In this embodiment S1, entity tags are inserted into the original text sequence in the sensitive information dataset to construct a linear sequence in which entity tags and named entities coexist, including:
[0075] Based on the original text sequence X=(x1, x2, ..., x n ), in x i Add labels indicating the head entity B- and the tail entity E- before and after, and add entity labels S- before and after each single-word entity; for non-entity markers O and entity internal markers I-, do not perform label sequence linearization processing;
[0076] Get a linear sequence of entity tags and named entities
[0077]
[0078] Among them, t i , t' i Indicates the entity character x i .
[0079] like Figure 3 As shown, taking "Peng has made great contributions to the company" as an example, based on the above method, a linear sequence in which entity labels and named entities coexist is constructed to obtain a linear sequence.
[0080] In this embodiment S2, Figure 4 As shown in the figure, the RoBERTa-WWM model is based on the RoBERTa architecture and is fine-tuned to adapt to the named entity recognition task. It achieves deep context-aware representation learning by stacking multiple Transformer encoder layers. Each Transformer encoder layer consists of two parts: a self-attention mechanism and a feedforward neural network; the self-attention mechanism is responsible for capturing the dependencies between words at each position in the input sequence, while the feedforward neural network (FFN) further refines this contextual information and increases the expressiveness of the model.
[0081] The WWM (WholeWordMasking) strategy is an improvement over BERT's original WordPiece Masking, which ensures that the entire word rather than a single subword is masked, thereby better preserving lexical semantic integrity.
[0082] In the training phase, the RoBERTa-WWM model adopts the WWM strategy to select the entire word for masking operation; the masking operation is dynamic masking; the dynamic masking is: before each iterative training, the linear sequence is masked based on the mask ratio η. Randomly select different entity words for mask replacement to obtain the sequence The sequence As input, iterative training:
[0083]
[0084] Among them, θ represents the RoBERTa-WWM model parameters, X represents the original text sequence, and x j Represents the named entity in the original text sequence X; if x j If masked, α i =1, otherwise α i =0.
[0085] The iterative training adopts the Adam optimizer, wherein the iteration cycle is set to 20, the batch size is set to 30, and the learning rate is set to 1e-5.
[0086] The combination of dynamic masking and WWM strategy in this step improves the model's ability to understand unseen entities and their contexts, especially in the case of few samples. And through reasonable iteration cycles, batch sizes, and fine-tuned learning rates, the training process is ensured to be both efficient and stable, reducing training time and computational costs.
[0087] In this embodiment S3, Figure 5 As shown, including:
[0088] S31, randomly selecting some named entities in the text sequence in the data set to be processed to perform masking processing, and constructing a masked text sequence;
[0089] S32, based on the masked text sequence, generating a probability distribution including k candidate entities, and forming a vocabulary V;
[0090] S33, selecting the first m entities with the highest probability in the vocabulary V to construct a candidate entity set C; wherein m≤k;
[0091] S34, randomly selecting an entity from the candidate entity set C, replacing the mask position in the text sequence to generate a new text sequence;
[0092] S35: remove the labels used for training annotation in the new text sequence to obtain an enhanced sequence in text form.
[0093] In this step, the original text is augmented using the E-MLM model. Especially when facing masked entities, not only is a replacement consistent with the original entity generated to maintain the authenticity of the data, but also diversity is effectively introduced by considering the first m most likely candidate entities in the probability distribution and randomly sampling the strategy to generate new and meaningful augmented sentences. It breaks the limitations of augmented data caused by highly consistent replacements and increases the robustness of natural language processing models when encountering unseen entities or contexts in entity recognition tasks.
[0094] And removing the label marks in the generated sequence ensures that the enhanced data can be used for training without labeled data, further improving the learning efficiency and generalization performance of the model on a small amount of labeled data.
[0095] Embodiment 2;
[0096] like Figure 6 As shown, this embodiment provides a data enhancement system for sensitive information, including:
[0097] Dataset construction and processing module: used to construct a sensitive information dataset for named entity recognition, perform label sequence linearization processing on the sensitive information dataset, and obtain the sensitive information dataset after label sequence linearization processing; the label sequence linearization processing is: inserting entity labels into the original text sequence in the sensitive information dataset, and constructing a linear sequence in which entity labels and named entities coexist;
[0098] E-MLM model training module: used to use the RoBERTa-WWM model as a pre-training model based on the sensitive information dataset after linearization of the label sequence, and adopt a named entity-guided masking strategy to perform iterative training to obtain an E-MLM model;
[0099] Enhanced data set generation module: used to input the data set to be processed into the E-MLM model to obtain the corresponding enhanced data set.
[0100] Embodiment 3;
[0101] like Figure 7 As shown, this embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the data enhancement method of the above embodiment when executing the computer program.
[0102] Embodiment 4;
[0103] This embodiment provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the data enhancement method of the above embodiment is implemented.
[0104] The above technical solution proposes a data enhancement method, system, device and medium for sensitive information, which is mainly used to improve the accuracy and generalization ability of named entity recognition. In the above method, a sensitive information dataset is first constructed, and entity labels are inserted through label sequence linearization processing to form rich training samples. Then, the RoBERTa-WWM model is combined with a named entity-oriented dynamic masking strategy for iterative training to obtain an E-MLM model dedicated to enhancement. After that, the model is applied to the dataset to be processed, and an enhanced dataset is generated through masking and replacement operations, which effectively expands the diversity of training data. This solution integrates data preprocessing, model training and data enhancement processes, aiming to optimize the recognition performance of natural language models when processing sensitive information, especially in the scenario of protecting privacy and information security, showing significant technical advantages and application value.
[0105] In the above embodiments provided in the present application, it should be understood that the disclosed methods, systems, devices and media can be implemented in other ways. The above described methods, systems, devices and media embodiments are only schematic. For example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation. Each functional unit can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0106] The units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0107] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc.
[0108] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data enhancement method for sensitive information, applied to named entity recognition, characterized in that: include: S1. Construct a sensitive information dataset for named entity recognition, perform label sequence linearization processing on the sensitive information dataset, and obtain the sensitive information dataset after label sequence linearization processing; the label sequence linearization processing is: inserting entity labels into the original text sequence in the sensitive information dataset, and constructing a linear sequence in which entity labels and named entities coexist; S2. Based on the sensitive information dataset after linearization of the label sequence, the RoBERTa-WWM model is used as a pre-training model, and a named entity-guided masking strategy is adopted to perform iterative training to obtain an E-MLM model; S3. Input the data set to be processed into the E-MLM model to obtain the corresponding enhanced data set.
2. A data enhancement method for sensitive information according to claim 1, characterized in that: In S1, a sensitive information dataset for named entity recognition is constructed, including: S11. Acquire sensitive information data; the sensitive information data is Internet platform data; S12, preprocessing the sensitive information data to obtain preprocessed sensitive information data; S13, segmenting the long text data in the pre-processed sensitive information data according to the punctuation mark priority to obtain segmented sensitive information data; S14. Based on the segmented sensitive information data, define entity categories, use the BIOES annotation method to perform classification annotation, and obtain a sensitive information data set for named entity recognition.
3. A data enhancement method for sensitive information according to claim 2, characterized in that: In S12, the preprocessing includes: Invalid symbols are removed; paragraph tags, hyperlink tags and URL information in the sensitive information data are removed through parsing strategies; and text content that is not related to the preset sensitive information category is removed using regular expressions.
4. The data enhancement method for sensitive information according to claim 1, characterized in that: In S1, entity tags are inserted into the original text sequence in the sensitive information dataset to construct a linear sequence in which entity tags and named entities coexist, including: Based on the original text sequence X=(x1, x2, ..., x n ), in x i Add labels indicating the head entity B- and the tail entity E- before and after, and add entity labels S- before and after each single-word entity; for non-entity markers O and entity internal markers I-, do not perform label sequence linearization processing; Get a linear sequence of entity tags and named entities 5. The data enhancement method for sensitive information according to claim 1, characterized in that: In S2, the RoBERTa-WWM model is obtained by stacking multiple Transformer encoder layers; the Transformer encoder layer includes: a self-attention mechanism and a feedforward neural network; In the training phase, the RoBERTa-WWM model adopts the WWM strategy to select the entire word for masking operation; the masking operation is dynamic masking; the dynamic masking is: before each iterative training, the linear sequence is masked based on the mask ratio η. Randomly select different entity words for mask replacement to obtain the sequence The sequence As input, iterative training: Among them, θ represents the RoBERTa-WWM model parameters, X represents the original text sequence, and x j Represents the named entity in the original text sequence X; if x j If masked, α i =1, otherwise α i =0.
6. A data enhancement method for sensitive information according to claim 5, characterized in that: The iterative training adopts the Adam optimizer, wherein the iteration cycle is set to 20, the batch size is set to 30, and the learning rate is set to 1e-5.
7. The method for data enhancement for sensitive information according to claim 1, characterized in that: The S3 includes: S31, randomly selecting some named entities in the text sequence in the data set to be processed to perform masking processing, and constructing a text sequence after masking processing; S32, based on the masked text sequence, generating a probability distribution including k candidate entities, and forming a vocabulary V; S33, selecting the first m entities with the highest probability in the vocabulary V to construct a candidate entity set C; wherein m≤k; S34, randomly selecting an entity from the candidate entity set C, replacing the mask position in the text sequence to generate a new text sequence; S35: remove the labels used for training annotation in the new text sequence to obtain an enhanced sequence in text form.
8. A data enhancement system for sensitive information, characterized in that: include: Dataset construction and processing module: used to construct a sensitive information dataset for named entity recognition, perform label sequence linearization processing on the sensitive information dataset, and obtain the sensitive information dataset after label sequence linearization processing; the label sequence linearization processing is: inserting entity labels into the original text sequence in the sensitive information dataset, and constructing a linear sequence in which entity labels and named entities coexist; E-MLM model training module: used to use the RoBERTa-WWM model as a pre-training model based on the sensitive information dataset after linearization of the label sequence, and adopt a named entity-guided masking strategy to perform iterative training to obtain an E-MLM model; Enhanced data set generation module: used to input the data set to be processed into the E-MLM model to obtain the corresponding enhanced data set.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data enhancement method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data enhancement method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Crop disease and insect pest named entity recognition method fusing RoBERTa-wwm and adversarial training
CN116629259A
Chinese microblog named entity recognition method based on tag semantic enhancement
CN117521657A
Nested named entity recognition-oriented data enhancement method and system and storage medium
CN117973388A
