A de-identification method, apparatus and equipment

Abstract

This invention discloses a de-identification method, apparatus, and device. The method involves inputting a dataset to be processed into a target model to obtain a dataset labeled with target entities; replacing the target entities with pre-defined alternative text to obtain a de-identified dataset; and training the target model in the following manner: performing first entity labeling based on the IOB labeling method to obtain a first labeled dataset; performing second entity labeling based on regular expressions and the first entity labels to obtain a second labeled dataset, which serves as the training set; training multiple pre-trained sub-models based on the training set until the accuracy of the trained sub-models is greater than or equal to an accuracy threshold, thus determining that the sub-models have passed training, and using the model composed of the trained sub-models as the target model. This invention achieves high accuracy and efficiency in target entity recognition by grouping and training multiple pre-trained sub-models, thereby achieving high accuracy and efficiency in de-identification.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a de-identification method, apparatus, and device. Background Technology

[0002] In healthcare, electronic health records (EHRs) are commonly used to update patient health information in real time, digitally track patient health status, and share patient information across different institutions, ensuring continuity of treatment. Therefore, structured data is widely used in EHRs. However, unstructured data (such as medical history, physical examination records, and pathology reports) contains even richer patient information. To effectively utilize this unstructured data in cutting-edge research, there is a pressing need for scalable de-identification technologies.

[0003] However, existing de-identification technologies identify entities by using predefined rules in pattern matching systems and then replace them. Since the rules of pattern matching systems are fixed, if the text format does not fully conform to the predefined rules, the system may produce false alarms or detection errors. Therefore, how to achieve high accuracy and high efficiency in de-identification technology has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a de-identification method, apparatus, and device to solve the problem of low accuracy in existing de-identification technologies.

[0005] Firstly, this application provides a de-identification method, the method comprising:

[0006] After inputting the obtained dataset to be processed into the target model, a dataset labeled with the target entities is obtained;

[0007] The target entity is replaced based on the preset alternative text to obtain a de-identified dataset;

[0008] The target model is trained in the following manner:

[0009] The first entity is labeled in the target dataset using the IOB labeling method, resulting in the first labeled dataset.

[0010] Based on regular expressions and the first entity tags, the first tag dataset is subjected to second entity tags to obtain a second tag dataset, and the second tag dataset is used as the training set.

[0011] Multiple pre-trained sub-models are trained based on the training set. If the accuracy of the trained sub-model is less than the accuracy threshold, the process returns to the step of performing second entity labeling on the first labeled dataset based on regular expressions and the first entity label, until the accuracy of the trained sub-model is greater than or equal to the accuracy threshold. If the accuracy of the trained sub-model is greater than or equal to the accuracy threshold, the sub-model is determined to have passed training, and the model composed of the multiple trained sub-models is taken as the target model.

[0012] In one possible implementation, the step of performing first entity labeling on the target dataset based on the IOB labeling method to obtain a first labeled dataset includes:

[0013] The target dataset is segmented into words to obtain multiple words from the target dataset;

[0014] Based on the IOB tagging method, for each word, if the word is determined to be an entity, the beginning part of the entity of the word is marked as the first tag, and the internal part of the entity of the word is marked as the second tag; if the word is determined to be a non-entity, the word is marked as the third tag.

[0015] The labeled target dataset is used as the first labeled dataset.

[0016] In one possible implementation, before performing second entity tagging on the tag dataset based on regular expressions and the first entity tag, the method further includes:

[0017] Calculate the repetition rate of the first target text in the first labeled dataset;

[0018] If the repetition rate of the first target text is determined to be greater than or equal to the repetition rate threshold, the first target text is added to the target set.

[0019] Delete the first target text from the first labeled dataset based on the target set.

[0020] In one possible implementation, the regular expression is determined by the following method:

[0021] The target words of the first labeled dataset are determined based on the pattern matching method, and the first entity label of the target words is updated to the first replacement word based on the preset first correspondence relationship;

[0022] Based on the first entity label in the first labeled dataset, determine the entity category corresponding to the target word;

[0023] Based on the entity category and the preset second correspondence, the first replacement word is updated to the second replacement word, and the first entity tag of the target word and the third correspondence between the second replacement word are used as the regular expression.

[0024] In one possible implementation, training multiple pre-trained sub-models based on the training set includes:

[0025] Assign a pre-trained sub-model to each pre-trained entity category in the training set;

[0026] Determine the target entity category in the training set of the pre-trained sub-model, wherein the target entity category is the entity category corresponding to the highest confidence during the training process;

[0027] If the target entity category matches the pre-trained entity category, the pre-trained sub-model is considered to have completed training; if the target entity category does not match the pre-trained entity category, the pre-trained sub-model continues to be trained.

[0028] In one possible implementation, before the step of returning to the second entity labeling of the labeled dataset based on the regular expression and the first entity label when the accuracy of the trained sub-model is less than the accuracy threshold, the method further includes:

[0029] If a second target text is determined to be absent from the second tag dataset, then the second target text is added to the target set.

[0030] Delete the second target text from the labeled dataset based on the target set.

[0031] In one possible implementation, replacing the target entity based on preset alternative text to obtain a de-identified dataset includes:

[0032] Determine the entity category and location of the target entity;

[0033] Target entities with the same entity category and the same entity instance are labeled with a preset tag;

[0034] Based on the fourth correspondence between the preset label and the alternative text, the target entity is replaced with the alternative text to obtain a de-identified dataset, wherein the fourth correspondence is preset.

[0035] Secondly, embodiments of this application also provide a de-identification device, the device comprising:

[0036] The labeling unit is used to input the acquired dataset to be processed into the target model to obtain a dataset with labeled target entities;

[0037] The replacement unit is used to replace the target entity based on a preset alternative text to obtain a de-identified dataset;

[0038] The training unit is used to perform first entity labeling on the target dataset based on the IOB labeling method to obtain a first labeled dataset; to perform second entity labeling on the labeled dataset based on regular expressions and the first entity labels to obtain a second labeled dataset, and to use the second labeled dataset as a training set; to train multiple pre-trained sub-models based on the training set; if the accuracy of the trained sub-models is less than an accuracy threshold, the unit returns to the step of performing second entity labeling on the labeled dataset based on regular expressions and the first entity labels, until the accuracy of the trained sub-models is greater than or equal to the accuracy threshold; if the accuracy of the trained sub-models is greater than or equal to the accuracy threshold, the sub-model is determined to have passed training, and the model composed of the multiple trained sub-models is used as the target model.

[0039] In one possible implementation, the training unit is specifically used for:

[0040] The target dataset is segmented into words to obtain multiple words from the target dataset;

[0041] Based on the IOB tagging method, for each word, if the word is determined to be an entity, the beginning part of the entity of the word is marked as the first tag, and the internal part of the entity of the word is marked as the second tag; if the word is determined to be a non-entity, the word is marked as the third tag.

[0042] The labeled target dataset is used as the first labeled dataset.

[0043] In one possible implementation, the training unit is specifically used for:

[0044] Calculate the repetition rate of the first target text in the first labeled dataset;

[0045] If the repetition rate of the first target text is determined to be greater than or equal to the repetition rate threshold, the first target text is added to the target set.

[0046] Delete the first target text from the first labeled dataset based on the target set.

[0047] In one possible implementation, the training unit is specifically used for:

[0048] The target words of the first labeled dataset are determined based on the pattern matching method, and the first entity label of the target words is updated to the first replacement word based on the preset first correspondence relationship;

[0049] Based on the first entity label in the first labeled dataset, determine the entity category corresponding to the target word;

[0050] Based on the entity category and the preset second correspondence, the first replacement word is updated to the second replacement word, and the first entity tag of the target word and the third correspondence between the second replacement word are used as the regular expression.

[0051] In one possible implementation, the training unit is specifically used for:

[0052] Assign a pre-trained sub-model to each pre-trained entity category in the training set;

[0053] Determine the target entity category in the training set of the pre-trained sub-model, wherein the target entity category is the entity category corresponding to the highest confidence during the training process;

[0054] If the target entity category matches the pre-trained entity category, the pre-trained sub-model is considered to have completed training; if the target entity category does not match the pre-trained entity category, the pre-trained sub-model continues to be trained.

[0055] In one possible implementation, the training unit is further configured to:

[0056] If a second target text is determined to be absent from the second tag dataset, then the second target text is added to the target set.

[0057] Delete the second target text from the labeled dataset based on the target set.

[0058] In one possible implementation, the replacement unit is specifically used for:

[0059] Determine the entity category and location of the target entity;

[0060] Target entities with the same entity category and the same entity instance are labeled with a preset tag;

[0061] Based on the fourth correspondence between the preset label and the alternative text, the target entity is replaced with the alternative text to obtain a de-identified dataset, wherein the fourth correspondence is preset.

[0062] Thirdly, embodiments of this application also provide a de-identification device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in any one of the first aspects.

[0063] The beneficial effects of this invention are as follows:

[0064] This application provides a de-identification method, apparatus, and device. The method involves inputting an acquired dataset to be processed into a target model to obtain a dataset labeled with target entities. The target entities are then replaced with pre-defined alternative text to obtain a de-identified dataset. The target model is trained as follows: the target dataset is labeled with first entities using the IOB labeling method to obtain a first labeled dataset; the labeled dataset is labeled with second entities using regular expressions and the first entity labels to obtain a second labeled dataset, which is then used as a training set; multiple pre-trained sub-models are trained based on the training set; if the accuracy of the trained sub-models is less than an accuracy threshold, the process returns to the step of labeling the labeled dataset with second entities using regular expressions and the first entity labels, until the accuracy of the trained sub-models is greater than or equal to the accuracy threshold; if the accuracy of the trained sub-models is greater than or equal to the accuracy threshold, the sub-model is considered to have passed training, and the model composed of the trained sub-models is used as the target model. This application achieves high accuracy and efficiency in target entity recognition and thus high accuracy and efficiency in de-identification by training multiple pre-trained sub-models in groups. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A flowchart illustrating a de-identification method provided in an embodiment of this application;

[0067] Figure 2(a) is a schematic diagram of a de-identification method provided in an embodiment of this application;

[0068] Figure 2(b) is a schematic diagram of another de-identification method provided in the embodiments of this application;

[0069] Figure 3A flowchart illustrating a training method for a target model provided in an embodiment of this application;

[0070] Figure 4 A flowchart illustrating another method for training a target model provided in an embodiment of this application;

[0071] Figure 5 A flowchart illustrating another method for training a target model provided in an embodiment of this application;

[0072] Figure 6(a) is a schematic diagram of another de-identification method provided in the embodiments of this application;

[0073] Figure 6(b) is a schematic diagram of another de-identification method provided in the embodiments of this application;

[0074] Figure 7 A flowchart illustrating another method for training a target model provided in an embodiment of this application;

[0075] Figure 8 A flowchart illustrating another method for training a target model provided in an embodiment of this application;

[0076] Figure 9 This is a schematic diagram of a de-identification device provided in an embodiment of this application;

[0077] Figure 10 This is a schematic diagram of a de-identification device provided in an embodiment of this application. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0079] The following explains some of the technical terms used in this application.

[0080] (1) Entity: refers to an object or thing that has a unique identifier and can be distinguished from other objects. It usually refers to a specific unit of information in text, such as a person's name, place name, organization name, etc. In other words, an entity is a part of the text that has a specific meaning.

[0081] (2) De-identification: In fields such as healthcare, finance, and scientific research, in order to reduce the leakage of personal information, information that may reveal personal identity is removed or modified from the data, so that the personal identity cannot be directly traced when the data is used or shared.

[0082] (3) Electronic Health Record (EHR): It is a digital patient health information recording system that includes detailed health information, treatment history, diagnosis, examination results, medication records, etc. Compared with traditional paper medical records, EHR enables information to be shared between different medical institutions and doctors.

[0083] It should be noted that the terms "comprising" and "having" and their variations used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0084] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as implying relative importance or implicitly indicating the number of indicated technical features. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0085] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.

[0086] The application scenarios of the technical solutions in the embodiments of this application are described below.

[0087] In healthcare, Electronic Health Records (EHRs) are commonly used to update patient health information in real time, digitally track patient health status, and share patient information across different institutions, ensuring continuity of patient treatment. Therefore, structured data is widely used in EHRs. However, unstructured data (such as medical history, physical examination records, and pathology reports) contains richer patient information. Traditional manual de-identification methods are prone to oversight or errors when processing large amounts of data. To effectively utilize this unstructured data in cutting-edge research, there is an urgent need for scalable de-identification techniques.

[0088] However, the de-identification technology implemented by pattern matching systems in related technologies identifies entities by predefined rules and then replaces them. Since the rules of the pattern matching system are fixed, if the text format in the dataset does not fully conform to the predefined rules, the pattern matching system may produce false alarms or detection errors.

[0089] The machine learning system de-identification technology in related technologies uses feature vector transformation to convert the input dataset into a numerical vector form, which makes it easier for computers to recognize and process. However, when processing large amounts of data, it is time-consuming and inefficient.

[0090] Deep learning systems in related technologies rely on large amounts of labeled datasets. For some specialized fields, the application of deep learning systems is limited because labeled datasets are difficult to obtain. Furthermore, deep learning systems have low accuracy when processing complex sentences.

[0091] As can be seen from the above, there is no de-identification technology with high accuracy and efficiency among the relevant technologies. Therefore, how to achieve a de-identification technology with high accuracy and efficiency has become an urgent technical problem to be solved.

[0092] To address this issue, embodiments of this application provide a de-identification method, apparatus, and device. A target model is obtained by training multiple pre-trained sub-models in groups. The dataset to be processed is then input into the target model to obtain a dataset labeled with target entities. The target entities are then replaced based on preset alternative text to obtain a de-identified dataset. This achieves high accuracy and efficiency in target entity recognition, thereby achieving high accuracy and efficiency in de-identification.

[0093] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0094] Figure 1 An exemplary flowchart of a de-identification method provided in an embodiment of this application is shown, such as... Figure 1 The image shows a de-identification method provided in an embodiment of this application, which specifically includes the following process:

[0095] S101. After inputting the obtained dataset to be processed into the target model, a dataset labeled with the target entities is obtained;

[0096] As an example, after the dataset to be processed is input into the target model, the target model labels the position of the category of the target entity in the dataset to be processed, so that the target entity can be de-identified based on the position of the category of the target entity.

[0097] S102. Replace the target entity with a pre-defined alternative text to obtain a de-identified dataset;

[0098] As an example, this application first determines the entity category and location of the target entity; then, it labels the target entities that have the same entity category and the same entity instance with preset labels; finally, based on the fourth correspondence between the preset labels and the alternative text, it replaces the target entity with the alternative text to obtain the de-identified dataset, wherein the fourth correspondence is preset.

[0099] For example, as shown in Figures 2(a) and 2(b), a schematic diagram of a de-identification method provided in an embodiment of this application is provided, wherein Figure 2(a) represents the dataset of the target entities that are labeled, and Figure 2(b) represents the de-identified dataset.

[0100] As shown in Figure 2(a), the dataset labeled with target entities includes labels indicating the location of the target entity's category. Specifically, the target entity categories for "2020-02-01", "15Jan 2020", and "1Feb 2020" are dates; "123456" and "92" are numbers; "John" and "Doe" are names; and "New York" is a place name. To reduce personal information leakage and avoid tracing individual identities during data use, it is necessary to modify or replace information in the data that may reveal personal identities.

[0101] This application first determines the entity category and location of the target entity based on the dataset of labeled target entities, and then labels the target entities with the same entity category and the same entity instance with preset labels. For example, the "Doe" label that appears twice in Figure 2(a) can be the same preset label A1, "John" can be labeled with preset label A2, "2020-02-01" can be labeled with preset label A3, "15Jan 2020" can be labeled with preset label A4, "1Feb 2020" can be labeled with preset label A5, "123456" can be labeled with preset label A6, "92" can be labeled with preset label A7, and "New York" can be labeled with preset label A8.

[0102] It should be noted that the preset labels A1, A2, A3, A4, A5, A6, A7 and A8 can be strings composed of English characters, strings composed of numbers, strings composed of both English characters and numbers, or other forms, without any restrictions here.

[0103] Based on the pre-defined fourth correspondence between preset labels and alternative texts, the target entity is replaced with the alternative text. For example, in this application, the pre-defined fourth correspondence includes the following: preset label A1 corresponds to the alternative text "Martin", preset label A2 corresponds to the alternative text "Tom", preset label A3 corresponds to the alternative text "2020-03-04", preset label A4 corresponds to the alternative text "13Jan2020", preset label A5 corresponds to the alternative text "18Feb 2020", preset label A6 corresponds to the alternative text "HKLMU9B", preset label A7 corresponds to the alternative text "90+", and preset label A8 corresponds to the alternative text "London". This process replaces the target entity with the alternative text, resulting in a de-identified dataset.

[0104] In this embodiment, the dataset to be processed is input into the target model to obtain a dataset labeled with target entities. The target entities are then replaced based on preset alternative text to obtain a de-identified dataset. This achieves high accuracy and efficiency in target entity recognition, thereby achieving high accuracy and efficiency in de-identification.

[0105] Specifically, the target model is trained in the following ways, such as Figure 3 The diagram shown is a flowchart illustrating a training method for a target model provided in an embodiment of this application, including the following steps:

[0106] S301. The target dataset is labeled with the first entity based on the IOB labeling method to obtain the first labeled dataset;

[0107] It should be noted that before performing the first entity labeling on the target dataset based on the IOB labeling method, the target dataset needs to be cleaned to remove invalid, erroneous, and incomplete data.

[0108] As an example, such as Figure 4 The diagram shown is a flowchart of another target model training method provided in this application embodiment, in which the first labeled dataset is obtained in the following manner.

[0109] S401. Perform word segmentation on the target dataset to obtain multiple words from the target dataset;

[0110] In one possible implementation, word segmentation is used to convert the text in the target dataset into smaller text units, which can be words or characters. For example, if the text in the target dataset is "John Smith livesin New York.", after word segmentation, we get "John", "Smith", "lives", "in", "New", "York", and ".".

[0111] S402. Based on the IOB tagging method, for each word, if the word is determined to be an entity, the beginning part of the entity of the word is tagged as the first tag, and the internal part of the entity of the word is tagged as the second tag; if the word is determined to be a non-entity, the word is tagged as the third tag.

[0112] For example, the IOB tagging method labels the beginning part of a word as the first tag B, the internal part of the word as the second tag I, and non-entity words as the third tag O. Furthermore, since entities include various categories, such as person names (PER) and location names (LOC), the beginning part of an entity categorized as a person can be labeled as the first tag B-PER, and the internal part of the entity categorized as a person can be labeled as the second tag I-PER. Similarly, the beginning part of an entity categorized as a location can be labeled as the first tag B-LOC, and the internal part of the entity categorized as a location can be labeled as the second tag I-LOC.

[0113] For example, consider the following words in the target dataset obtained from word segmentation: “John”, “Smith”, “lives”, “in”, “New”, “York”, and “.”. “John Smith” is an entity categorized as a person's name, where “John” is the beginning part of the person's name entity and is labeled with the first label B-PER, and “Smith” is the inner part of the person's name entity and is labeled with the second label I-PER. “New York” is an entity categorized as a place name, where “New” is the beginning part of the place name entity and is labeled with the first label B-LOC, and “York” is the inner part of the place name entity and is labeled with the second label I-LOC. “lives”, “in”, and “.” are non-entity words and are labeled with the third label O.

[0114] S403. Use the labeled target dataset as the first labeled dataset;

[0115] For example, taking the text "John Smith lives in New York." in the target dataset as an example, the dataset with the first label B-PER and B-LOC, the second label I-PER and I-LOC, and the third label O is used as the first labeled dataset.

[0116] It should be noted that the entity category can be a person's name, place name, organization name, date, time, quantity, etc., and is not limited thereto. It should be understood that the preferred embodiments described herein are for illustration and explanation only, and are not intended to limit this application. Furthermore, the embodiments and features described herein can be combined with each other without conflict.

[0117] As an example, such as Figure 5 The diagram shown illustrates a training method for another target model provided in this application. After obtaining the first labeled dataset and before performing second entity labeling on the first labeled dataset, the method further includes:

[0118] S501. Calculate the repetition rate of the first target text in the first labeled dataset;

[0119] S502. If it is determined that the repetition rate of the first target text is greater than or equal to the repetition rate threshold, the first target text is added to the target set.

[0120] S503. Delete the first target text in the first labeled dataset based on the target set.

[0121] For example, the first target text in the target set is used to exclude repetitive and meaningless text, reducing noise in subsequent model training, such as chapter titles in clinical records, repetitive use of doctors' habits, etc.

[0122] For example, such as Figure 6(a) and 6(b) The diagram illustrates another de-identification method provided in this application embodiment. Figure 6(a) represents an exemplary text in the first labeled dataset, and Figure 6(b) represents an exemplary text in the first labeled dataset after deleting the first target text. Since "DATE" and "NUMBER" appear multiple times in the first labeled dataset, if the repetition rate of "DATE" and "NUMBER" is determined to be greater than or equal to a repetition rate threshold, "DATE" and "NUMBER" are added to the target set as the first target text. Based on the target set, the first target texts "DATE" and "NUMBER" are deleted from the first labeled dataset. The gray-marked portion in Figure 6(b) represents the deletion of the first target text from the first labeled data.

[0123] S302. Based on regular expressions and the first entity tags, perform second entity tags on the first tag dataset to obtain the second tag dataset, and use the second tag dataset as the training set;

[0124] As an example, such as Figure 7 The diagram shown is a flowchart illustrating another training method for a target model provided in this application embodiment. The regular expression is determined using the following method:

[0125] S701. Determine the target words of the first labeled dataset based on the pattern matching method, and update the first entity label of the target words to the first replacement word based on the preset first correspondence relationship;

[0126] S702. Based on the first entity label in the first label dataset, determine the entity category corresponding to the target word;

[0127] S703. Based on the entity category and the preset second correspondence, update the first replacement word to the second replacement word, and use the first entity tag of the target word and the third correspondence of the second replacement word as a regular expression.

[0128] For example, this application updates the first entity tag of the target word to the first replacement word based on the pattern matching method and the preset first correspondence. The first correspondence is predetermined based on the Named Entity Recognition (NER) template. For example, when the entity is composed of numbers, the first entity tag is updated to N; when the entity is composed of letters, the first entity tag is updated to L; and when the entity is composed of both numbers and letters, the first entity tag is updated to M.

[0129] The pre-defined second correspondence is the correspondence between the first and second replacement words. The first replacement word "L" corresponds to the second replacement word "[A-Za-z]+", indicating that the entity marked with L corresponds to one or more letters. The first replacement word "N" corresponds to the second replacement word "\d+", indicating that the entity marked with N corresponds to one or more numbers. The first replacement word "M" corresponds to the second replacement word "\w", indicating that the entity marked with M corresponds to a letter or a number.

[0130] For example, taking the date "12 / 25 / 2024" appearing in the first labeled dataset as an example, the first entity label of the entity "12 / 25 / 2024" is updated to the first replacement word N / N / N based on the preset first correspondence. Based on the entity category and the preset second correspondence, the first replacement word N / N / N is updated to the second replacement word \d+ / \d+ / \d+. Thus, based on the regular expression, the first entity label of the entity that satisfies the date format 12 / 25 / 2024 is updated to d+ / \d+ / \d+, thereby realizing the second entity label of the labeled dataset and obtaining the second labeled dataset.

[0131] For example, taking the time "3:45PM" appearing in the first labeled dataset as an example, the first entity label of the entity "3:45PM" is updated to the first replacement word N:NM based on the preset first correspondence. Based on the entity category and the preset second correspondence, the first replacement word N / N / N is updated to the second replacement word \d+:\d+\w. Thus, based on the regular expression, the first entity label of the entity that satisfies the 3:45PM time format is updated to \d+:\d+\w, thereby realizing the second entity label of the labeled dataset and obtaining the second labeled dataset.

[0132] This application embodiment uses regular expressions and first entity tags to perform second entity tagging on a first tag dataset, resulting in a second tag dataset. Regular expressions are used to strengthen the entity tags in the first tag dataset, thereby enhancing the accuracy of the target model.

[0133] S303. Train multiple pre-trained sub-models based on the training set. If the accuracy of the trained sub-model is less than the accuracy threshold, return to the step of performing second entity labeling on the labeled dataset based on regular expressions and the first entity label, until the accuracy of the trained sub-model is greater than or equal to the accuracy threshold. If the accuracy of the trained sub-model is greater than or equal to the accuracy threshold, determine that the sub-model has passed training, and use the model composed of multiple trained sub-models as the target model.

[0134] As an example, such as Figure 8 The diagram shown is a flowchart illustrating another method for training a target model provided in an embodiment of this application.

[0135] S801. Assign a pre-trained sub-model to each pre-trained entity category in the training set;

[0136] S802. Determine the target entity category in the training set of the pre-trained sub-model, where the target entity category is the entity category corresponding to the highest confidence during the training process;

[0137] S803. If the target entity category is consistent with the pre-trained entity category, determine that the pre-trained sub-model has completed training; if the target entity category is inconsistent with the pre-trained entity category, continue training the pre-trained sub-model.

[0138] For example, a pre-trained sub-model is assigned to each pre-trained entity category in the training set. For instance, if the pre-trained entity categories in the training set are person names, place names, and dates, then a first pre-trained sub-model is assigned to train entities in the training set whose entity category is person names, a second pre-trained sub-model is assigned to train entities in the training set whose entity category is place names, and a third pre-trained sub-model is assigned to train entities in the training set whose entity category is dates.

[0139] Taking the assignment of the first pre-training sub-model to train entities in the training set whose entity category is human name as an example, during the training process, if the target entity category determined by the first pre-training sub-model is consistent with the entity category of the first pre-training sub-model, that is, if the target entity category is human name, the first pre-training sub-model is determined to have completed training. If the target entity category determined by the first pre-training sub-model is inconsistent, that is, if the target entity category is not human name, the first pre-training sub-model continues to be trained.

[0140] This application employs a group training method to train multiple pre-trained sub-models, achieving high accuracy and efficiency in target entity recognition, thereby achieving high accuracy and efficiency in de-identification. Furthermore, this application reduces reliance on large labeled datasets through model training methods, improving the model's generalization ability.

[0141] It should be noted that each pre-trained sub-model includes a case-sensitive XL-NET model and a case-insensitive XL-NET model. Training both the case-sensitive and case-insensitive XL-NET models simultaneously during the pre-trained sub-model training process can improve the flexibility and accuracy of the model in recognizing entities.

[0142] As an example, if the accuracy of the trained sub-model is less than the accuracy threshold, before returning to the step of performing second entity labeling on the labeled dataset based on regular expressions and the first entity label, the following steps are also included:

[0143] The system identifies second target text that does not exist in the second labeled dataset and adds it to the target set. Then, it removes second target text from the labeled dataset based on the target set. In other words, during the iterative fine-tuning of the sub-model, the accuracy of the sub-model is improved by adding second target text to the target set and removing it from the training set.

[0144] For example, during the iterative fine-tuning of the sub-model, public datasets, such as datasets including cities and addresses, can be added to improve the training effect of the sub-model and thus improve its accuracy.

[0145] It should be noted that after training the pre-trained sub-model based on the training set, the accuracy of the trained sub-model is determined based on the test set to verify its performance. If the accuracy of the trained sub-model is less than the accuracy threshold, the process returns to the step of performing second entity labeling on the first labeled dataset based on regular expressions and the first entity label, and continues to train the sub-model until the accuracy of the trained sub-model is greater than or equal to the accuracy threshold.

[0146] If the accuracy of a trained sub-model is greater than or equal to an accuracy threshold, the sub-model is deemed to have passed training, and the model composed of multiple trained sub-models is used as the target model. For example, taking three pre-trained sub-models as an example, if the accuracy of each of the three pre-trained sub-models is greater than or equal to the accuracy threshold, the model composed of the three trained pre-trained sub-models is used as the target model.

[0147] Based on the same inventive concept, this application also provides a de-identification device, which is similar in principle to the above-described de-identification method. Details that are repeated will not be repeated here. Figure 9 The diagram shown is a schematic representation of a de-identification device provided in an embodiment of this application, comprising:

[0148] The labeling unit 901 is used to input the acquired dataset to be processed into the target model to obtain a dataset with labeled target entities;

[0149] Replacement unit 902 is used to replace the target entity based on the preset replacement text to obtain the de-identified dataset;

[0150] Training unit 903 is used to perform first entity labeling on the target dataset based on the IOB labeling method to obtain a first labeled dataset; to perform second entity labeling on the first labeled dataset based on regular expressions and the first entity labels to obtain a second labeled dataset, and to use the second labeled dataset as the training set; to train multiple pre-trained sub-models based on the training set; if the accuracy of the trained sub-model is less than the accuracy threshold, the process returns to the step of performing second entity labeling on the first labeled dataset based on regular expressions and the first entity labels, until the accuracy of the trained sub-model is greater than or equal to the accuracy threshold; if the accuracy of the trained sub-model is greater than or equal to the accuracy threshold, the sub-model is determined to have passed training, and the model composed of the multiple trained sub-models is used as the target model.

[0151] This application provides a de-identification method and apparatus. The method involves inputting an acquired dataset to be processed into a target model to obtain a dataset labeled with target entities. The target entities are then replaced with pre-defined alternative text to obtain a de-identified dataset. The target model is trained as follows: the target dataset is labeled with first entities using the IOB labeling method to obtain a first labeled dataset; the labeled dataset is labeled with second entities using regular expressions and the first entity labels to obtain a second labeled dataset, which is then used as a training set; multiple pre-trained sub-models are trained based on the training set; if the accuracy of the trained sub-models is less than an accuracy threshold, the process returns to the step of labeling the labeled dataset with second entities using regular expressions and the first entity labels, until the accuracy of the trained sub-models is greater than or equal to the accuracy threshold; if the accuracy of the trained sub-models is greater than or equal to the accuracy threshold, the sub-model is considered to have passed training, and the model composed of the trained sub-models is used as the target model. This application achieves high accuracy and efficiency in target entity recognition by training multiple pre-trained sub-models in groups, thereby achieving high accuracy and efficiency in de-identification.

[0152] In one possible implementation, the training unit 903 is specifically used for:

[0153] The target dataset is segmented into words to obtain multiple words from the target dataset;

[0154] Based on the IOB tagging method, for each word, if the word is determined to be an entity, the beginning part of the entity is tagged as the first tag, and the internal part of the entity is tagged as the second tag; if the word is determined to be a non-entity, the word is tagged as the third tag.

[0155] The labeled target dataset is used as the first labeled dataset.

[0156] In one possible implementation, the training unit 903 is specifically used for:

[0157] Calculate the repetition rate of the first target text in the first labeled dataset;

[0158] If the repetition rate of the first target text is determined to be greater than or equal to the repetition rate threshold, the first target text is added to the target set.

[0159] Delete the first target text from the first labeled dataset based on the target set.

[0160] In one possible implementation, the training unit 903 is specifically used for:

[0161] The target words of the first labeled dataset are determined based on the pattern matching method, and the first entity label of the target words is updated to the first replacement word based on the preset first correspondence.

[0162] Based on the first entity label in the first labeled dataset, determine the entity category corresponding to the target word;

[0163] Based on entity categories and a pre-defined second correspondence, the first replacement word is updated to the second replacement word, and the third correspondence between the first entity tag of the target word and the second replacement word is used as a regular expression.

[0164] In one possible implementation, the training unit 903 is specifically used for:

[0165] Assign a pre-trained sub-model to each pre-trained entity category in the training set;

[0166] Determine the target entity category in the training set of the pre-trained sub-model, where the target entity category is the entity category corresponding to the highest confidence during training;

[0167] If the target entity category matches the pre-trained entity category, the pre-trained sub-model is completed; if the target entity category does not match the pre-trained entity category, the pre-trained sub-model continues to be trained.

[0168] In one possible implementation, the training unit 903 is further configured to:

[0169] If a second target text is found that does not have a second entity tag in the second tag dataset, the second target text is added to the target set.

[0170] The second target text in the labeled dataset is removed based on the target set.

[0171] In one possible implementation, the replacement unit 902 is specifically used for:

[0172] Determine the entity category and location of the target entity;

[0173] Target entities with the same entity category and the same entity instance are labeled with a preset tag;

[0174] Based on the fourth correspondence between the preset labels and the alternative text, the target entity is replaced with the alternative text to obtain the de-identified dataset, where the fourth correspondence is preset.

[0175] Based on the same inventive concept, this application also provides a de-identification device, which is similar in principle to the above-described de-identification method and apparatus; repeated details will not be elaborated further. Figure 10The diagram shown is a schematic of a de-identification device provided in an embodiment of this application, including at least one processor 1001; and a memory 1002 communicatively connected to at least one processor 1001; wherein the memory 1002 stores instructions that can be executed by at least one processor 1001, and the instructions are executed by at least one processor 1001 to enable at least one processor 1001 to perform the method of any one of the first aspects.

[0176] This application provides a de-identification method, apparatus, and device. The method involves inputting an acquired dataset to be processed into a target model to obtain a dataset labeled with target entities. The target entities are then replaced with pre-defined alternative text to obtain a de-identified dataset. The target model is trained as follows: the target dataset is labeled with first entities using the IOB labeling method to obtain a first labeled dataset; the labeled dataset is labeled with second entities using regular expressions and the first entity labels to obtain a second labeled dataset, which is then used as a training set; multiple pre-trained sub-models are trained based on the training set; if the accuracy of the trained sub-models is less than an accuracy threshold, the process returns to the step of labeling the labeled dataset with second entities using regular expressions and the first entity labels, until the accuracy of the trained sub-models is greater than or equal to the accuracy threshold; if the accuracy of the trained sub-models is greater than or equal to the accuracy threshold, the sub-model is considered to have passed training, and the model composed of the trained sub-models is used as the target model. This application achieves high accuracy and efficiency in target entity recognition and thus high accuracy and efficiency in de-identification by training multiple pre-trained sub-models in groups.

[0177] The present application has been described above with reference to block diagrams and / or flowcharts illustrating methods, apparatus (systems), and / or computer program products according to embodiments of the present application. It should be understood that a block of a block diagram and / or flowchart, as well as combinations of blocks of block diagrams and / or flowcharts, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, and / or other programmable data processing means to produce a machine, such that the instructions, executable via the computer processor and / or other programmable data processing means, create methods for implementing the functions / actions specified in the blocks of the block diagrams and / or flowcharts.

[0178] Accordingly, this application can also be implemented using hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, this application can take the form of a computer program product on a computer-usable or computer-readable storage medium, having computer-usable or computer-readable program code implemented in the medium for use by or in conjunction with an instruction execution system. In the context of this application, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, transmit, or deliver a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0179] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A de-identification method, characterized in that, The method includes: After inputting the obtained dataset to be processed into the target model, a dataset labeled with the target entities is obtained; The target entity is replaced based on the preset alternative text to obtain a de-identified dataset; The target model is trained in the following manner: The first entity is labeled in the target dataset using the IOB labeling method, resulting in the first labeled dataset. Based on regular expressions and the first entity tags, the first tag dataset is subjected to second entity tags to obtain a second tag dataset, and the second tag dataset is used as the training set. Multiple pre-trained sub-models are trained based on the training set. If the accuracy of the trained sub-model is less than the accuracy threshold, the process returns to the step of performing second entity labeling on the first labeled dataset based on regular expressions and the first entity label, until the accuracy of the trained sub-model is greater than or equal to the accuracy threshold. If the accuracy of the trained sub-model is greater than or equal to the accuracy threshold, the sub-model is determined to have passed training, and the model composed of the multiple trained sub-models is taken as the target model.

2. The method as described in claim 1, characterized in that, The first entity labeling of the target dataset based on the IOB labeling method yields the first labeled dataset, which includes: The target dataset is segmented into words to obtain multiple words from the target dataset; Based on the IOB tagging method, for each word, if the word is determined to be an entity, the beginning part of the entity of the word is marked as the first tag, and the internal part of the entity of the word is marked as the second tag; if the word is determined to be a non-entity, the word is marked as the third tag. The labeled target dataset is used as the first labeled dataset.

3. The method as described in claim 1, characterized in that, Before performing second entity tagging on the first tag dataset based on regular expressions and the first entity tag, the method further includes: Calculate the repetition rate of the first target text in the first labeled dataset; If the repetition rate of the first target text is determined to be greater than or equal to the repetition rate threshold, the first target text is added to the target set. Delete the first target text from the first labeled dataset based on the target set.

4. The method as described in claim 1, characterized in that, The regular expression is determined by the following method: The target words of the first labeled dataset are determined based on the pattern matching method, and the first entity label of the target words is updated to the first replacement word based on the preset first correspondence relationship; Based on the first entity label in the first labeled dataset, determine the entity category corresponding to the target word; Based on the entity category and the preset second correspondence, the first replacement word is updated to the second replacement word, and the first entity tag of the target word and the third correspondence between the second replacement word are used as the regular expression.

5. The method as described in claim 1, characterized in that, The training of multiple pre-trained sub-models based on the training set includes: Assign a pre-trained sub-model to each pre-trained entity category in the training set; Determine the target entity category in the training set of the pre-trained sub-model, wherein the target entity category is the entity category corresponding to the highest confidence during the training process; If the target entity category matches the pre-trained entity category, the pre-trained sub-model is considered to have completed training; if the target entity category does not match the pre-trained entity category, the pre-trained sub-model continues to be trained.

6. The method as described in claim 3, characterized in that, Before the step of returning to the second entity labeling of the first labeled dataset based on the regular expression and the first entity label when the accuracy of the trained sub-model is less than the accuracy threshold, the method further includes: If a second target text is determined to be absent from the second tag dataset, then the second target text is added to the target set. Delete the second target text from the labeled dataset based on the target set.

7. The method according to any one of claims 1 to 6, characterized in that, The process of replacing the target entity with a preset alternative text to obtain a de-identified dataset includes: Determine the entity category and location of the target entity; Target entities with the same entity category and the same entity instance are labeled with a preset tag; Based on the fourth correspondence between the preset label and the alternative text, the target entity is replaced with the alternative text to obtain a de-identified dataset, wherein the fourth correspondence is preset.

8. A de-identification device, characterized in that, The device includes: The labeling unit is used to input the acquired dataset to be processed into the target model to obtain a dataset with labeled target entities; The replacement unit is used to replace the target entity based on a preset alternative text to obtain a de-identified dataset; The training unit is used to perform first entity labeling on the target dataset based on the IOB labeling method to obtain a first labeled dataset; to perform second entity labeling on the first labeled dataset based on regular expressions and the first entity labels to obtain a second labeled dataset, and to use the second labeled dataset as a training set; to train multiple pre-trained sub-models based on the training set; if the accuracy of the trained sub-model is less than an accuracy threshold, the unit returns to the step of performing second entity labeling on the first labeled dataset based on regular expressions and the first entity labels, until the accuracy of the trained sub-model is greater than or equal to the accuracy threshold; if the accuracy of the trained sub-model is greater than or equal to the accuracy threshold, the sub-model is determined to have passed training, and the model composed of the multiple trained sub-models is used as the target model.

9. The apparatus as claimed in claim 8, characterized in that, The training unit is specifically used for: The target dataset is segmented into words to obtain multiple words from the target dataset; Based on the IOB tagging method, for each word, if the word is determined to be an entity, the beginning part of the entity of the word is tagged as the first tag, and the internal part of the entity of the word is tagged as the second tag. If the term is determined to be a non-entity, the term is marked as a third tag; The labeled target dataset is used as the first labeled dataset.

10. A de-identification device, characterized in that, The method includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

Images