An apparatus for automatic identification and dynamic masking of sensitive text data

Abstract

The application discloses an automatic identification and dynamic masking processing device for sensitive text data, and belongs to the technical field of automatic identification and text desensitization processing of sensitive information. The device is configured to perform the following operations: performing source-aware preprocessing on the text to be processed to output a pure text stream with uniform format; detecting potential sensitive segments in the pure text stream to output a candidate segment list containing position information and a preliminary type hypothesis; performing deep semantic analysis on each candidate segment to output an entity type probability distribution and a boundary confidence score corresponding to each candidate segment; ruling on each candidate segment according to a preset arbitration strategy to output a confirmed target entity set and corresponding entity type label, boundary positioning information and comprehensive confidence score; and mapping the target entity to a corresponding differential masking strategy based on a credibility level to perform dynamic masking processing on sensitive entities in the pure text stream.

Description

Technical Field

[0001] This application relates to the field of automatic identification of sensitive information and text desensitization processing technology, and more specifically, to an automatic identification and dynamic masking processing device for sensitive text data. Background Technology

[0002] In the information age, text data is increasingly used in finance, healthcare, government affairs, education, and other fields, involving a large amount of sensitive information, including personal identification information, account information, contract terms, and trade secrets. This sensitive information is easily obtained or misused during data transmission, storage, and sharing, posing serious risks to data security and privacy protection. Traditional manual identification methods are inefficient, susceptible to subjective factors, and unable to meet the demands of massive text processing. Existing automated processing technologies mostly rely on fixed rules or simple keyword matching, making it difficult to accurately identify complex semantic relationships and lacking dynamic masking and adaptive adjustment capabilities, easily leading to false masking or missed masking, and making it difficult to balance security and data availability.

[0003] In conclusion, how to achieve efficient and accurate identification of sensitive text and dynamic adaptive masking processing has become an urgent technical problem to be solved. Summary of the Invention

[0004] To overcome a series of shortcomings in existing technologies, the purpose of this application is to provide an automated identification and dynamic masking device for sensitive text data, comprising a preprocessing module, a candidate segment detection module, a character-level language model module, an arbitration fusion module, a legality verification module, and a dynamic masking module. The device is configured to perform the following operations: S1, through the preprocessing module, performs source-aware preprocessing on the text to be processed, and outputs a plain text stream with a uniform format; S2, through the candidate segment detection module, detects potential sensitive segments in the plain text stream and outputs a list of candidate segments containing position information and preliminary type assumptions; S3 performs deep semantic analysis on each candidate segment through the character-level language model module, and outputs the entity type probability distribution and boundary confidence score corresponding to each candidate segment. S4, based on the candidate fragment list, entity type probability distribution and boundary confidence score, constructs a two-dimensional scoring matrix through the arbitration fusion module, and adjudicates each candidate fragment according to the preset arbitration strategy, outputting the confirmed target entity set and the corresponding entity type label, boundary location information and comprehensive confidence score; S5 performs multi-dimensional legality verification on the target entity set through the legality verification module, and generates an entity recognition report containing the credibility level; S6, based on the credibility level, maps the target entity to the corresponding differentiated masking strategy through the dynamic masking module, and performs dynamic masking processing on sensitive entities in the plain text stream.

[0005] In some embodiments, S1 includes the following steps: Obtain the text source information of the text to be processed. The text source information includes at least the transmission protocol type, the original encoding format identifier, the document structure type label, and the business scenario classification label. Based on the document structure type tag, perform structure expansion and redundant node stripping on the text to be processed, and convert text of different structure types into continuous text content. Based on the original encoding format identifier, perform a unified encoding conversion on the text after structural expansion processing to convert it into a preset standard encoding format; During the encoding conversion process, placeholder replacement is performed on illegal byte sequences, and a mapping relationship is established between the placeholder positions and the original byte content; Based on the business scenario classification tags, load the corresponding domain thesaurus and stop word list, perform word segmentation and noise filtering on the text, and the noise includes at least template-like repetitive sentences, format delimiter sequences and characters without semantic control. Output a plain text stream without embedded annotations and construct an independent source metadata structure. Establish the correspondence between the source metadata structure and the plain text stream through character offset index.

[0006] In some embodiments, S2 includes the following steps: Based on a pre-defined sensitive information rule base, rule matching processing is performed on the plain text stream to obtain the first candidate segment set; Based on a pre-built sensitive dictionary matching model, multi-pattern matching processing is performed on the plain text stream, and initial confidence weights are assigned to the matching results according to the sensitivity level to obtain a second candidate segment set; A sliding window scan is performed on the plain text stream based on character-level statistical features to identify regions with dense abnormal information and obtain a third set of candidate segments. The first, second, and third candidate segment sets are merged to eliminate overlaps or inclusion relationships, forming a unified candidate segment set. For candidate segments in the unified candidate segment set that have conflicting type assumptions, the target type label is determined by fusion processing based on the preset weights of each detection method. Generate an information structure for each candidate segment, including start position, end position, type label, and detection source marker, to form a candidate segment list.

[0007] In some embodiments, S3 includes the following steps: Expand the candidate fragments at the character level to obtain the corresponding character representation sequence; Convert a character sequence into a character embedding vector sequence; The character embedding vector sequence is input into a bidirectional long short-term memory network to extract character-level sequence dependency features in both the forward and reverse directions. The hidden state vectors in the two directions are then concatenated to obtain the first feature representation sequence. The first feature representation sequence is input into a multi-head self-attention network to calculate the association weights between character positions, thereby obtaining a second feature representation sequence containing long-range dependency information. This second feature representation sequence is then mapped to a unified semantic representation space to obtain the semantic representation of the candidate segment. Based on the semantic representation, a multi-task prediction branch is constructed, wherein: in the entity type prediction branch, a probability distribution vector covering a preset set of entity types is output; in the boundary location prediction branch, the boundary label probability is output for each character position within the candidate segment, and the boundary confidence score of the candidate segment is determined accordingly.

[0008] In some embodiments, S4 includes the following steps: A two-dimensional scoring matrix is ​​constructed using the unique ID of the candidate fragment in the unified candidate fragment set as the row index and the entity type in the preset entity type set as the column index. Based on the detection results and reliability weights of the candidate fragment detection module, the initial hit weights for the corresponding entity types of each candidate fragment are determined. The probability distribution of entity types is calibrated, and the probability values ​​of the corresponding entity types are extracted as semantic probability components. The overall score of the candidate segment is discounted based on the boundary confidence score to obtain the boundary penalty component; Based on the preset weighting coefficients, the initial hit weight, semantic probability component and boundary penalty component are weighted and summed, and the weighted result is written into the corresponding cell of the two-dimensional scoring matrix. Perform a column-wise maximum value selection operation on each row of the two-dimensional scoring matrix to determine the entity type of each candidate fragment; The maximum score is normalized to generate a comprehensive confidence score, and the entity type label, comprehensive confidence score, and boundary location information are output.

[0009] In some embodiments, the preset arbitration strategy includes a conflict resolution sub-strategy, a low-confidence filtering sub-strategy, and a cross-fragment consistency correction sub-strategy, wherein: The conflict resolution sub-strategy targets multiple overlapping candidate segments within the same text interval. It uses a greedy non-maximum suppression algorithm to resolve conflicts based on the comprehensive confidence score of each candidate segment. It prioritizes retaining candidate segments with higher comprehensive confidence scores and performs suppression and deletion operations on other candidate segments whose character-level overlap ratio exceeds a preset overlap ratio threshold. The character-level overlap ratio is defined as the ratio of the number of overlapping characters between two candidate segments to the number of characters in the shorter candidate segment. The low confidence filtering sub-strategy sets a threshold for the overall confidence score. Candidate segments with an overall confidence score lower than the threshold are directly marked as negative examples and removed from the candidate segment list. The threshold is set separately for different entity types to adapt to the differences in recognition difficulty of different types of entities. The cross-fragment consistency correction sub-strategy performs cross-instance consistency checks on candidate fragments with the same text content within the same document. If the recognition results of the same text content in different locations show inconsistent types, the recognition result with the highest comprehensive confidence score is taken as the standard, and the entity type labeling and comprehensive confidence score of the other inconsistent fragments are uniformly corrected. This ensures that the recognition conclusions of the same content within the same document maintain logical consistency and reduces the risk of omissions caused by inconsistent types in subsequent masking processing.

[0010] In some embodiments, a multi-dimensional validity check is performed on each candidate fragment by a validity check module, including the following steps: Based on the format specifications of the entity type corresponding to the candidate fragment, the character composition, length range, and check bits of the candidate fragment are verified, and those that fail are marked with a format anomaly. The candidate fragments that pass the format validation are subjected to semantic rationality validation to determine whether their values ​​or structures meet the semantic constraints of the corresponding entity type. If they do not meet the constraints, a semantic anomaly marker is generated. Extract the context window of a preset length before and after the candidate fragment, input the candidate fragment and context into the lightweight distillation language model to calculate the false perplexity score, and compare it with the preset threshold. If it exceeds the threshold, generate a suspected false alarm label. When logically related entities exist in the same document, cross-field consistency verification is performed according to predefined logical constraint rules. If the constraints are not met, the overall confidence level is reduced and a suspected false alarm is generated. Based on the results of format compliance verification, semantic rationality verification, contextual relevance verification, and cross-field cross-validation, a multi-dimensional legality verification result for candidate fragments is generated.

[0011] In some embodiments, S6 includes the following steps: Obtain the entity recognition information output in the entity recognition report. The entity recognition information includes the entity text content, entity type, confidence level, and the entity's position information in the original text. For each identified entity, a combined index key is constructed based on its entity type and confidence level to be used for strategy positioning in the differentiated masking strategy library; Based on the composite index key, retrieve the corresponding masking strategy record in the differentiated masking strategy library with a dual-dimensional index structure built according to entity type and credibility level. The masking strategy record includes the masking method field and related strategy parameter fields. Read the masking method field and the parameter fields corresponding to the masking method from the masking strategy record, including the masking character set field, the preserved prefix length field, the preserved suffix length field, or the format placeholder template field; Based on the value of the masking method field, the corresponding masking processing is performed on the identified entity, including any one of full character replacement, partial character replacement, type placeholder replacement, noise perturbation replacement, or irreversible hash digest replacement; The replacement result obtained after the masking process is filled back into the corresponding position of the original text according to the position information of the identified entities. After all identified entities have completed the masking process, the masked target text content is generated and output.

[0012] In some embodiments, the device is further configured to: S7, during the dynamic masking process, monitor in real time whether the current masking operation meets the preset masking integrity standard; S8. If the current occlusion process does not meet the occlusion integrity standard, then maintain the dynamic occlusion process and adaptively adjust the occlusion range or occlusion strategy in conjunction with the entity recognition report until the occlusion integrity standard is met. S9. When the occlusion integrity standard is met, update the status of the entity recognition report, supplement the final occlusion strategy information, the actual occlusion range information, and the occlusion integrity verification result to form an updated entity recognition report. S10: End the dynamic masking process for the current text, and output the desensitized text and the updated entity recognition report. In some embodiments, S8 includes the following steps: Obtain the target text after the current occlusion processing and the corresponding entity recognition report, and calculate the coverage index, false occlusion index and format consistency index based on the target text; The coverage rate, false blockage rate and format consistency rate are compared with their respective preset standards to determine whether there are any integrity indicators that do not meet the preset standards. When there are integrity indicators that do not meet the preset standards, determine the type of the non-compliant indicator; When the non-compliance indicator is the coverage indicator, retrieve the high-confidence level entities and medium-confidence level entities that have not yet completed the masking process from the entity identification report, and perform supplementary masking processing on the entities. The supplementary masking processing adopts the same differentiated masking strategy as the original masking processing. When the non-compliance indicator is the false alarm rate, filter the entities marked as suspected false alarms from the list of obscured entities; When the non-compliant indicator is the format consistency indicator, for the same type of entities with the same text content but different masking forms, the masking form used in the first masking process is used as the benchmark, and the masking form of the remaining entities is uniformly corrected. After completing this round of adaptive adjustment, update the masked target text and related statistical data, and record the adjustment operation and corresponding integrity index changes in the audit log. Recalculate the coverage rate, false occlusion rate, and format consistency metrics, and determine whether they all meet the preset standards. If they do not all meet the standards and the maximum adjustment round limit has not been reached, proceed to the next round of adaptive adjustment. If they all meet the standards or the maximum adjustment round limit has been reached, terminate the adaptive adjustment process.

[0013] Compared with the prior art, this application has the following beneficial effects: This application achieves high-precision automatic identification and adaptive dynamic masking processing of sensitive text data through an arbitration fusion mechanism that combines multi-source candidate fragment detection with character-level deep semantic analysis. Attached Figure Description

[0014] Figure 1 This is a schematic diagram illustrating the working principle of an automated identification and dynamic masking device for sensitive text data disclosed in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some embodiments of this invention, but not all embodiments.

[0016] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The embodiments and directional terms described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0018] like Figure 1 As shown, an automated identification and dynamic masking device for sensitive text data includes a preprocessing module, a candidate segment detection module, a character-level language model module, an arbitration fusion module, a legality verification module, and a dynamic masking module. The device is configured to perform the following operations: S1, through the preprocessing module, performs source-aware preprocessing on the text to be processed, and outputs a plain text stream with a uniform format; S2, through the candidate segment detection module, detects potential sensitive segments in the plain text stream and outputs a list of candidate segments containing position information and preliminary type assumptions; S3 performs deep semantic analysis on each candidate segment through the character-level language model module, and outputs the entity type probability distribution and boundary confidence score corresponding to each candidate segment. S4, based on the candidate fragment list, entity type probability distribution and boundary confidence score, constructs a two-dimensional scoring matrix through the arbitration fusion module, and adjudicates each candidate fragment according to the preset arbitration strategy, outputting the confirmed target entity set and the corresponding entity type label, boundary location information and comprehensive confidence score; S5 performs multi-dimensional legality verification on the target entity set through the legality verification module, and generates an entity recognition report containing the credibility level; S6, based on the credibility level, maps the target entity to the corresponding differentiated masking strategy through the dynamic masking module, and performs dynamic masking processing on sensitive entities in the plain text stream; S7, during the dynamic masking process, monitor in real time whether the current masking operation meets the preset masking integrity standard; S8. If the current occlusion process does not meet the occlusion integrity standard, then maintain the dynamic occlusion process and adaptively adjust the occlusion range or occlusion strategy in conjunction with the entity recognition report until the occlusion integrity standard is met. S9. When the occlusion integrity standard is met, update the status of the entity recognition report, supplement the final occlusion strategy information, the actual occlusion range information, and the occlusion integrity verification result to form an updated entity recognition report. S10: End the dynamic masking process for the current text, and output the desensitized text and the updated entity recognition report.

[0019] The automated identification and dynamic masking device for sensitive text data described in this application performs source-aware preprocessing on the text to be processed by a preprocessing module and outputs a unified plain text stream; a candidate segment detection module identifies potential sensitive segments and generates a candidate list with location information and type assumptions; a character-level language model module performs semantic analysis on the candidate segments to obtain entity type probability distribution and boundary confidence; based on this, an arbitration fusion module constructs a scoring matrix and makes a ruling, outputting a target entity set and comprehensive confidence information, and a legality verification module performs multi-dimensional verification to generate an entity identification report; subsequently, a dynamic masking module implements differentiated masking strategies according to the confidence level, and detects the masking integrity in real time during the masking process, adaptively adjusting the masking range or strategy until the standard is met if the integrity is not met; finally, the entity identification report is updated and desensitized text is output, thereby realizing automated, accurate, and traceable management of sensitive information identification and masking.

[0020] In some embodiments, S1 includes the following steps: Obtain the text source information of the text to be processed. The text source information includes at least the transmission protocol type, the original encoding format identifier, the document structure type label, and the business scenario classification label. Based on the document structure type tag, perform structure expansion and redundant node stripping on the text to be processed, and convert text of different structure types into continuous text content. Based on the original encoding format identifier, perform a unified encoding conversion on the text after structural expansion processing to convert it into a preset standard encoding format; During the encoding conversion process, placeholder replacement is performed on illegal byte sequences, and a mapping relationship is established between the placeholder positions and the original byte content; Based on the business scenario classification tags, load the corresponding domain thesaurus and stop word list, perform word segmentation and noise filtering on the text, and the noise includes at least template-like repetitive sentences, format delimiter sequences and characters without semantic control. Output a plain text stream without embedded annotations and construct an independent source metadata structure. Establish the correspondence between the source metadata structure and the plain text stream through character offset index.

[0021] In step S1 of this application, source-aware preprocessing is performed on the text by obtaining the source information of the text to be processed, including the transmission protocol type, original encoding format identifier, document structure type label, and business scenario classification label. Based on the document structure type label, structural expansion and redundant node removal are performed to convert texts with different structures into continuous text content. A unified encoding conversion is then performed based on the original encoding format identifier, replacing illegal byte sequences with placeholders and establishing mapping relationships during the conversion process. Subsequently, a domain lexicon and stop word list are loaded based on the business scenario classification label, and the text is segmented and noise filtered. Finally, a plain text stream in a unified format is output, and a source metadata structure is constructed using character offset indexes to establish corresponding relationships, thereby achieving standardized and traceable preprocessing of multi-source text data.

[0022] In some embodiments, S2 includes the following steps: Based on a pre-defined sensitive information rule base, rule matching processing is performed on the plain text stream to obtain the first candidate segment set; Based on a pre-built sensitive dictionary matching model, multi-pattern matching processing is performed on the plain text stream, and initial confidence weights are assigned to the matching results according to the sensitivity level to obtain a second candidate segment set; A sliding window scan is performed on the plain text stream based on character-level statistical features to identify regions with dense abnormal information and obtain a third set of candidate segments. The first, second, and third candidate segment sets are merged to eliminate overlaps or inclusion relationships, forming a unified candidate segment set. For candidate segments in the unified candidate segment set that have conflicting type assumptions, the target type label is determined by fusion processing based on the preset weights of each detection method. Generate an information structure for each candidate segment, including start position, end position, type label, and detection source marker, to form a candidate segment list.

[0023] In step S2 of this application, multi-strategy candidate segment detection is performed on the plain text stream. A first candidate segment set is obtained by rule matching based on a sensitive information rule base. A second candidate segment set is obtained by performing multi-pattern matching based on a sensitive dictionary matching model and assigning initial confidence. A third candidate segment set is obtained by scanning and identifying densely populated areas of abnormal information using character-level statistical feature sliding window scanning. Subsequently, interval merging and conflict fusion processing are performed on each candidate segment set to eliminate overlapping relationships and determine a unified type label. Finally, an information structure containing location range, type label, and detection source mark is generated for each candidate segment to form a candidate segment list, thereby achieving high coverage and structured detection of potential sensitive segments.

[0024] In some embodiments, S3 includes the following steps: Expand the candidate fragments at the character level to obtain the corresponding character representation sequence; Convert a character sequence into a character embedding vector sequence; The character embedding vector sequence is input into a bidirectional long short-term memory network to extract character-level sequence dependency features in both the forward and reverse directions. The hidden state vectors in the two directions are then concatenated to obtain the first feature representation sequence. The first feature representation sequence is input into a multi-head self-attention network to calculate the association weights between character positions, thereby obtaining a second feature representation sequence containing long-range dependency information. This second feature representation sequence is then mapped to a unified semantic representation space to obtain the semantic representation of the candidate segment. A multi-task prediction branch is constructed based on the semantic representation, wherein: In the entity type prediction branch, the output is a probability distribution vector that covers the preset set of entity types; In the boundary localization prediction branch, the boundary label probability is output for each character position within the candidate segment, and the boundary confidence score of the candidate segment is determined accordingly.

[0025] In step S3 of this application, the candidate fragments are expanded into character sequences and converted into character embedding vector sequences by performing character-level deep semantic analysis. These sequences are then input into a bidirectional long short-term memory network to extract forward and reverse sequence dependency features to form a first feature representation sequence. Furthermore, a multi-head self-attention network is used to obtain long-range dependency information and map it to a unified semantic representation space to obtain the semantic representation of the candidate fragments. Based on this, a multi-task prediction branch is constructed to output the entity type probability distribution and character boundary label probability, and the boundary confidence score is calculated, thereby achieving refined analysis of entity type recognition and boundary localization of candidate fragments.

[0026] In some embodiments, S4 includes the following steps: A two-dimensional scoring matrix is ​​constructed using the unique ID of the candidate fragment in the unified candidate fragment set as the row index and the entity type in the preset entity type set as the column index. Based on the detection results and reliability weights of the candidate fragment detection module, the initial hit weights for the corresponding entity types of each candidate fragment are determined. The probability distribution of entity types is calibrated, and the probability values ​​of the corresponding entity types are extracted as semantic probability components. The overall score of the candidate segment is discounted based on the boundary confidence score to obtain the boundary penalty component; Based on the preset weighting coefficients, the initial hit weight, semantic probability component and boundary penalty component are weighted and summed, and the weighted result is written into the corresponding cell of the two-dimensional scoring matrix. Perform a column-wise maximum value selection operation on each row of the two-dimensional scoring matrix to determine the entity type of each candidate fragment; The maximum score is normalized to generate a comprehensive confidence score, and the entity type label, comprehensive confidence score, and boundary location information are output.

[0027] In some embodiments, the preset arbitration strategy includes a conflict resolution sub-strategy, a low-confidence filtering sub-strategy, and a cross-fragment consistency correction sub-strategy, wherein: The conflict resolution sub-strategy targets multiple overlapping candidate segments within the same text interval. It uses a greedy non-maximum suppression algorithm to resolve conflicts based on the comprehensive confidence score of each candidate segment. It prioritizes retaining candidate segments with higher comprehensive confidence scores and performs suppression and deletion operations on other candidate segments whose character-level overlap ratio exceeds a preset overlap ratio threshold. The character-level overlap ratio is defined as the ratio of the number of overlapping characters between two candidate segments to the number of characters in the shorter candidate segment. The low confidence filtering sub-strategy sets a threshold for the overall confidence score. Candidate segments with an overall confidence score lower than the threshold are directly marked as negative examples and removed from the candidate segment list. The threshold is set separately for different entity types to adapt to the differences in recognition difficulty of different types of entities. The cross-fragment consistency correction sub-strategy performs cross-instance consistency checks on candidate fragments with the same text content within the same document. If the recognition results of the same text content in different locations show inconsistent types, the recognition result with the highest comprehensive confidence score is taken as the standard, and the entity type labeling and comprehensive confidence score of the other inconsistent fragments are uniformly corrected. This ensures that the recognition conclusions of the same content within the same document maintain logical consistency and reduces the risk of omissions caused by inconsistent types in subsequent masking processing.

[0028] In step S4 of this application, a two-dimensional scoring matrix is ​​constructed with candidate segments as row indices and entity types as column indices. The initial hit weights, semantic probability components, and boundary penalty components of the candidate segment detection results are weighted and calculated to determine the entity type of each candidate segment and generate a comprehensive confidence score and boundary location information. Based on this, the recognition results are adjudicated according to a preset arbitration strategy. Overlapping segments are eliminated through a conflict resolution sub-strategy, low-confidence results are removed through a low-confidence filtering sub-strategy, and the recognition conclusions of the same text content are unified through a cross-segment consistency correction sub-strategy, thereby obtaining accurate and consistent target entity recognition results.

[0029] In some embodiments, a multi-dimensional validity check is performed on each candidate fragment by a validity check module, including the following steps: Based on the format specifications of the entity type corresponding to the candidate fragment, the character composition, length range, and check bits of the candidate fragment are verified; among them, the ID card number type is verified using ISO 7064, the bank card number type is verified using the Luhn algorithm, and the telephone number type is verified for the country code and number segment validity; those that fail are marked with a format anomaly. The candidate fragments that pass the format validation are subjected to semantic rationality validation to determine whether their values ​​or structures meet the semantic constraints of the corresponding entity type; those that do not meet the constraints are marked with semantic anomalies. Extract the context window of a preset length before and after the candidate fragment, input the candidate fragment and context into the lightweight distillation language model to calculate the false perplexity score, and compare it with the preset threshold; those exceeding the threshold are marked as suspected false alarms. When logically related entities exist in the same document, cross-field consistency verification is performed according to predefined logical constraint rules; entities that do not meet the constraints have their overall confidence level reduced and suspected false alarms are marked. Based on the results of format compliance verification, semantic rationality verification, contextual relevance verification, and cross-field cross-validation, a multi-dimensional legality verification result for candidate fragments is generated.

[0030] The multi-dimensional legality verification method described in this application performs format compliance, semantic rationality, contextual relevance, and cross-field consistency verification on candidate fragments through a legality verification module. Specifically, it verifies character composition, length range, and check bits according to entity type specifications, marking those that fail as format anomalies. For fragments that pass format verification, semantic constraints are further assessed, marking those that do not meet them as semantic anomalies. False perplexity is calculated using a context window and compared with a threshold; anomalies are marked as suspected false alarms. Cross-field consistency verification is performed when logically related entities exist, reducing the confidence level of those that do not meet constraints. Finally, the verification results are integrated to generate a multi-dimensional legality verification conclusion for candidate fragments, thereby improving the accuracy and reliability of entity recognition results.

[0031] In some embodiments, the confidence level is divided as follows: the comprehensive confidence score after multi-dimensional legality verification is mapped to a discrete set of confidence levels, which includes four levels: high confidence level, medium confidence level, low confidence level, and suspected false alarm level. Candidate segments with a comprehensive confidence score higher than a first preset threshold and passing all verification dimensions are classified as high confidence level, triggering the strictest masking strategy. Candidate segments with a comprehensive confidence score between the first and second preset thresholds or passing only some verification dimensions are classified as medium confidence level, triggering the standard masking strategy. Candidate segments with a comprehensive confidence score between the second and third preset thresholds are classified as low confidence level, triggering a conservative masking strategy, and a manual review suggestion mark is added to the entity recognition report. Candidate segments with a comprehensive confidence score lower than the third preset threshold and marked as suspected false alarm are classified as suspected false alarm level, without triggering masking processing, but their recognition information is fully retained in the entity recognition report for audit review.

[0032] The aforementioned credibility grading method in this application maps the comprehensive confidence score obtained from multi-dimensional legitimacy verification to four levels: high credibility, medium credibility, low credibility, and suspected false alarm. Among them, those above the first threshold and passing all verifications are classified as high credibility, triggering the most stringent masking; those between the first and second thresholds or passing only some verifications are classified as medium credibility, triggering standard masking; those between the second and third thresholds are classified as low credibility, triggering conservative masking and adding a manual review mark; those below the third threshold and marked as suspected false alarms are classified as suspected false alarms, without masking but retaining complete information for auditing, thereby achieving graded handling and controllable risk management.

[0033] In some embodiments, S6 includes the following steps: Obtain the entity recognition information output in the entity recognition report. The entity recognition information includes the entity text content, entity type, confidence level, and the entity's position information in the original text. For each identified entity, a combined index key is constructed based on its entity type and confidence level to be used for strategy positioning in the differentiated masking strategy library; Based on the composite index key, retrieve the corresponding masking strategy record in the differentiated masking strategy library with a dual-dimensional index structure built according to entity type and credibility level. The masking strategy record includes the masking method field and related strategy parameter fields. Read the masking method field and the parameter fields corresponding to the masking method from the masking strategy record, including the masking character set field, the preserved prefix length field, the preserved suffix length field, or the format placeholder template field; Based on the value of the masking method field, the corresponding masking processing is performed on the identified entity, including any one of full character replacement, partial character replacement, type placeholder replacement, noise perturbation replacement, or irreversible hash digest replacement; The replacement result obtained after the masking process is filled back into the corresponding position of the original text according to the position information of the identified entities. After all identified entities have completed the masking process, the masked target text content is generated and output.

[0034] In step S6 of this application, the entity text, type, credibility level, and location information from the entity recognition report are obtained. An index key is constructed based on the entity type and credibility level. The corresponding masking strategy is retrieved from the dual-dimensional differentiated masking strategy library. After reading the masking method and parameter fields in the strategy, the entity is processed by full character replacement, partial replacement, type placeholder replacement, noise perturbation, or irreversible hash digest processing. The processing result is backfilled into the original text according to the entity location information. Finally, the masked target text is generated and output, realizing refined and reliable dynamic sensitive information protection.

[0035] In some embodiments, the preset occlusion integrity standard includes three quantitative standards: coverage rate, false occlusion rate, and format consistency. The coverage rate metric is defined as the ratio of the number of identified entities processed by the current masking operation to the total number of all high-confidence and medium-confidence entities in the entity identification report. The false alarm rate is defined as the ratio of the number of entities that were marked as suspected false alarms but still had their masking operations performed in the current masking operation to the total number of masked entities. The format consistency index requires that entities of the same type and with the same original text content within the same document should present a consistent masking form after masking processing. In other words, there should be no situation where the same sensitive content has an inconsistent masking form in different locations.

[0036] The masking integrity standard described in this application achieves precise evaluation and control of dynamic masking effects through three quantitative indicators: coverage rate, false masking rate, and format consistency. The coverage rate measures the proportion of processed high-confidence and medium-confidence entities, the false masking rate measures the ratio of suspected false alarm entities that are falsely masked, and the format consistency indicator requires that entities of the same type and with the same original text content within the same document present a uniform masking form. Based on this, the integrity, accuracy, and consistency management of sensitive entity masking operations are achieved, thereby improving the security and reliability of desensitized text.

[0037] In some embodiments, S8 includes the following steps: Obtain the target text after the current occlusion processing and the corresponding entity recognition report, and calculate the coverage index, false occlusion index and format consistency index based on the target text; The coverage rate, false blockage rate and format consistency rate are compared with their respective preset standards to determine whether there are any integrity indicators that do not meet the preset standards. When there are integrity indicators that do not meet the preset standards, determine the type of the non-compliant indicator; When the non-compliance indicator is the coverage indicator, retrieve the high-confidence level entities and medium-confidence level entities that have not yet completed the masking process from the entity identification report, and perform supplementary masking processing on the entities. The supplementary masking processing adopts the same differentiated masking strategy as the original masking processing. When the non-compliance metric is the false masking rate, entities marked as suspected false alarms are filtered from the list of masked entities. For entities processed using reversible masking, the masking undoing operation is performed based on the mapping record between the original text content and the masking result stored in the protected memory buffer. For entities processed using irreversible hash digest replacement, no undoing operation is performed, and an irreversible undoing prompt is added to the entity identification report. At the same time, the entity is excluded from the statistical scope of the false masking rate metric. When the non-compliant indicator is the format consistency indicator, for the same type of entities with the same text content but different masking forms, the masking form used in the first masking process is used as the benchmark, and the masking form of the remaining entities is uniformly corrected. After completing this round of adaptive adjustment, update the masked target text and related statistical data, and record the adjustment operation and corresponding integrity index changes in the audit log. Recalculate the coverage rate, false occlusion rate, and format consistency metrics, and determine whether they all meet the preset standards. If they do not all meet the standards and the maximum adjustment round limit has not been reached, proceed to the next round of adaptive adjustment. If they all meet the standards or the maximum adjustment round limit has been reached, terminate the adaptive adjustment process.

[0038] Step S8 of this application, through real-time monitoring and comparison of coverage rate, false masking rate, and format consistency indicators, enables targeted adjustments to items that fail to meet the masking integrity standards. When the coverage rate fails to meet the standards, high-confidence and medium-confidence entities that have not yet been processed are masked. When the false masking rate fails to meet the standards, reversible masking entities are undoed, and irreversible entities are given an irreversible warning and excluded from statistics. When the format consistency fails to meet the standards, the masking form of the same text content is uniformly corrected. After each round of adjustment, the target text and statistical data are updated and recorded in the audit log. This process is iterated until all integrity indicators meet the preset standards or the maximum number of adjustment rounds is reached, thereby ensuring the integrity, accuracy, and consistency of sensitive text masking.

[0039] In some embodiments, the content structure of the entity recognition report is organized according to a unified reporting model. This model includes three primary nodes: a text-level summary section, an entity details section, and a processing record section. The text-level summary section includes a unique identifier for the processing task, the hash digest value of the original text, the text source category, the processing completion timestamp, a total count of recognized entities, and the distribution of entity numbers grouped by confidence level. The entity details section records, in sub-record form, the entity number, entity type label, starting character offset, ending character offset, hash digest of the original text content, overall confidence score, confidence level, whether a suspected false alarm marker is attached, the name of the actual masking strategy used, the masked text content, and the results of each dimension of legality verification. The process log records the processing latency of each module in the device in chronological order, the total number of candidate segments output by the candidate segment detection module, the number of candidate segments resolved by the arbitration fusion module, the number of candidate segments whose confidence is reduced by the legality verification module, and the number of rounds of adaptive adjustment performed by the dynamic masking module and the operation summary of each round of adjustment; the entity recognition report is serialized in JSON format and supports conversion to HTML format report according to the preset report template; the entity recognition report is performed on integrity signature before output. The integrity signature and the audit log module use the same HMAC-SHA256 signature mechanism based on the hardware security module to ensure that the report is not tampered with during transmission and storage, and the signature can be verified by the recipient through the public signature verification interface.

[0040] The entity recognition report in this application organizes text-level summaries, entity details, and processing records through a unified reporting mode, achieving complete recording and traceable management of identified entity information. The text-level summary includes task identifier, original hash, source category, completion time, entity statistics, and confidence distribution. The entity details record the number, type label, character offset, original hash, comprehensive confidence level, confidence level, suspected false alarm marker, actual masking strategy, masked text, and legality verification status for each identified entity. The processing record records the module processing delay, number of candidate segments, number of conflict resolutions, number of confidence adjustments, and adaptive adjustment rounds and summaries in chronological order. The report is serialized in JSON format and supports HTML conversion. Before output, an HMAC-SHA256 integrity signature is performed to ensure secure transmission and storage and the feasibility of signature verification.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated identification and dynamic masking device for sensitive text data, comprising a preprocessing module, a candidate segment detection module, a character-level language model module, an arbitration fusion module, a legality verification module, and a dynamic masking module, characterized in that, The device is configured to perform the following operations: S1, through the preprocessing module, performs source-aware preprocessing on the text to be processed, and outputs a plain text stream with a uniform format; S2, through the candidate segment detection module, detects potential sensitive segments in the plain text stream and outputs a list of candidate segments containing position information and preliminary type assumptions; S3 performs deep semantic analysis on each candidate segment through the character-level language model module, and outputs the entity type probability distribution and boundary confidence score corresponding to each candidate segment. S4, based on the candidate fragment list, entity type probability distribution and boundary confidence score, constructs a two-dimensional scoring matrix through the arbitration fusion module, and adjudicates each candidate fragment according to the preset arbitration strategy, outputting the confirmed target entity set and the corresponding entity type label, boundary location information and comprehensive confidence score; S5 performs multi-dimensional legality verification on the target entity set through the legality verification module, and generates an entity recognition report containing the credibility level; S6, based on the credibility level, maps the target entity to the corresponding differentiated masking strategy through the dynamic masking module, and performs dynamic masking processing on sensitive entities in the plain text stream.

2. The automated identification and dynamic masking processing device according to claim 1, characterized in that, S1 includes the following steps: Obtain the text source information of the text to be processed. The text source information includes at least the transmission protocol type, the original encoding format identifier, the document structure type label, and the business scenario classification label. Based on the document structure type tag, perform structure expansion and redundant node stripping on the text to be processed, and convert text of different structure types into continuous text content. Based on the original encoding format identifier, perform a unified encoding conversion on the text after structural expansion processing to convert it into a preset standard encoding format; During the encoding conversion process, placeholder replacement is performed on illegal byte sequences, and a mapping relationship is established between the placeholder positions and the original byte content; Based on the business scenario classification tags, load the corresponding domain thesaurus and stop word list, perform word segmentation and noise filtering on the text, and the noise includes at least template-like repetitive sentences, format delimiter sequences and characters without semantic control. Output a plain text stream without embedded annotations and construct an independent source metadata structure. Establish the correspondence between the source metadata structure and the plain text stream through character offset index.

3. The automated identification and dynamic masking processing device according to claim 1, characterized in that, S2 includes the following steps: Based on a pre-defined sensitive information rule base, rule matching processing is performed on the plain text stream to obtain the first candidate segment set; Based on a pre-built sensitive dictionary matching model, multi-pattern matching processing is performed on the plain text stream, and initial confidence weights are assigned to the matching results according to the sensitivity level to obtain a second candidate segment set; A sliding window scan is performed on the plain text stream based on character-level statistical features to identify regions with dense abnormal information and obtain a third set of candidate segments. The first, second, and third candidate segment sets are merged to eliminate overlaps or inclusion relationships, forming a unified candidate segment set. For candidate segments in the unified candidate segment set that have conflicting type assumptions, the target type label is determined by fusion processing based on the preset weights of each detection method. Generate an information structure for each candidate segment, including start position, end position, type label, and detection source marker, to form a candidate segment list.

4. The automated identification and dynamic masking processing device according to claim 1, characterized in that, S3 includes the following steps: Expand the candidate fragments at the character level to obtain the corresponding character representation sequence; Convert a character sequence into a character embedding vector sequence; The character embedding vector sequence is input into a bidirectional long short-term memory network to extract character-level sequence dependency features in both the forward and reverse directions. The hidden state vectors in the two directions are then concatenated to obtain the first feature representation sequence. The first feature representation sequence is input into a multi-head self-attention network to calculate the association weights between character positions, thereby obtaining a second feature representation sequence containing long-range dependency information. This second feature representation sequence is then mapped to a unified semantic representation space to obtain the semantic representation of the candidate segment. Based on the semantic representation, a multi-task prediction branch is constructed, wherein: in the entity type prediction branch, a probability distribution vector covering a preset set of entity types is output; in the boundary location prediction branch, the boundary label probability is output for each character position within the candidate segment, and the boundary confidence score of the candidate segment is determined accordingly.

5. The automated identification and dynamic masking processing device according to claim 1, characterized in that, S4 includes the following steps: A two-dimensional scoring matrix is ​​constructed using the unique ID of the candidate fragment in the unified candidate fragment set as the row index and the entity type in the preset entity type set as the column index. Based on the detection results and reliability weights of the candidate fragment detection module, the initial hit weights for the corresponding entity types of each candidate fragment are determined. The probability distribution of entity types is calibrated, and the probability values ​​of the corresponding entity types are extracted as semantic probability components. The overall score of the candidate segment is discounted based on the boundary confidence score to obtain the boundary penalty component; Based on the preset weighting coefficients, the initial hit weight, semantic probability component and boundary penalty component are weighted and summed, and the weighted result is written into the corresponding cell of the two-dimensional scoring matrix. Perform a column-wise maximum value selection operation on each row of the two-dimensional scoring matrix to determine the entity type of each candidate fragment; The maximum score is normalized to generate a comprehensive confidence score, and the entity type label, comprehensive confidence score, and boundary location information are output.

6. The automated identification and dynamic masking processing device according to claim 1, characterized in that, The pre-defined arbitration strategy includes a conflict resolution sub-strategy, a low-confidence filtering sub-strategy, and a cross-segment consistency correction sub-strategy, among which: The conflict resolution sub-strategy targets multiple overlapping candidate segments within the same text interval. It uses a greedy non-maximum suppression algorithm to resolve conflicts based on the comprehensive confidence score of each candidate segment. It prioritizes retaining candidate segments with higher comprehensive confidence scores and performs suppression and deletion operations on other candidate segments whose character-level overlap ratio exceeds a preset overlap ratio threshold. The character-level overlap ratio is defined as the ratio of the number of overlapping characters between two candidate segments to the number of characters in the shorter candidate segment. The low confidence filtering sub-strategy sets a threshold for the overall confidence score. Candidate segments with an overall confidence score lower than the threshold are directly marked as negative examples and removed from the candidate segment list. The threshold is set separately for different entity types to adapt to the differences in recognition difficulty of different types of entities. The cross-fragment consistency correction sub-strategy performs cross-instance consistency checks on candidate fragments with the same text content within the same document. If the recognition results of the same text content in different locations show inconsistent types, the recognition result with the highest comprehensive confidence score is taken as the standard, and the entity type labeling and comprehensive confidence score of the other inconsistent fragments are uniformly corrected. This ensures that the recognition conclusions of the same content within the same document maintain logical consistency and reduces the risk of omissions caused by inconsistent types in subsequent masking processing.

7. The automated identification and dynamic masking processing device according to claim 1, characterized in that, The validity verification module performs multi-dimensional validity verification on each candidate fragment, including the following steps: Based on the format specifications of the entity type corresponding to the candidate fragment, the character composition, length range, and check bits of the candidate fragment are verified, and those that fail are marked with a format anomaly. The candidate fragments that pass the format validation are subjected to semantic rationality validation to determine whether their values ​​or structures meet the semantic constraints of the corresponding entity type. If they do not meet the constraints, a semantic anomaly marker is generated. Extract the context window of a preset length before and after the candidate fragment, input the candidate fragment and context into the lightweight distillation language model to calculate the false perplexity score, and compare it with the preset threshold. If it exceeds the threshold, generate a suspected false alarm label. When logically related entities exist in the same document, cross-field consistency verification is performed according to predefined logical constraint rules. If the constraints are not met, the overall confidence level is reduced and a suspected false alarm is generated. Based on the results of format compliance verification, semantic rationality verification, contextual relevance verification, and cross-field cross-validation, a multi-dimensional legality verification result for candidate fragments is generated.

8. The automated identification and dynamic masking processing device according to claim 1, characterized in that, S6 includes the following steps: Obtain the entity recognition information output in the entity recognition report. The entity recognition information includes the entity text content, entity type, confidence level, and the entity's position information in the original text. For each identified entity, a combined index key is constructed based on its entity type and confidence level to be used for strategy positioning in the differentiated masking strategy library; Based on the composite index key, retrieve the corresponding masking strategy record in the differentiated masking strategy library with a dual-dimensional index structure built according to entity type and credibility level. The masking strategy record includes the masking method field and related strategy parameter fields. Read the masking method field and the parameter fields corresponding to the masking method from the masking strategy record, including the masking character set field, the preserved prefix length field, the preserved suffix length field, or the format placeholder template field; Based on the value of the masking method field, the corresponding masking processing is performed on the identified entity, including any one of full character replacement, partial character replacement, type placeholder replacement, noise perturbation replacement, or irreversible hash digest replacement; The replacement result obtained after the masking process is filled back into the corresponding position of the original text according to the position information of the identified entities. After all identified entities have completed the masking process, the masked target text content is generated and output.

9. The automated identification and dynamic masking processing device according to any one of claims 1-8, characterized in that, The device is also configured to: S7, during the dynamic masking process, monitor in real time whether the current masking operation meets the preset masking integrity standard; S8. If the current occlusion process does not meet the occlusion integrity standard, then maintain the dynamic occlusion process and adaptively adjust the occlusion range or occlusion strategy in conjunction with the entity recognition report until the occlusion integrity standard is met. S9. When the occlusion integrity standard is met, the entity recognition report is updated to include the final occlusion strategy information, the actual occlusion range information, and the occlusion integrity verification result, thus forming an updated entity recognition report. S10: End the dynamic masking process for the current text, and output the desensitized text and the updated entity recognition report.

10. The automated identification and dynamic masking processing device according to claim 9, characterized in that, S8 includes the following steps: Obtain the target text after the current occlusion processing and the corresponding entity recognition report, and calculate the coverage index, false occlusion index and format consistency index based on the target text; The coverage rate, false blockage rate and format consistency rate are compared with their respective preset standards to determine whether there are any integrity indicators that do not meet the preset standards. When there are integrity indicators that do not meet the preset standards, determine the type of the non-compliant indicator; When the non-compliance indicator is the coverage indicator, retrieve high-confidence level entities and medium-confidence level entities that have not yet completed the masking process from the entity identification report, and perform supplementary masking processing on the entities. The supplementary masking processing adopts the same differentiated masking strategy as the original masking processing. When the non-compliance indicator is the false alarm rate, filter the entities marked as suspected false alarms from the list of obscured entities; When the non-compliant indicator is the format consistency indicator, for entities of the same type with the same text content but inconsistent masking forms, the masking form used in the first masking process is used as the benchmark, and the masking form of the remaining entities is uniformly corrected. After completing this round of adaptive adjustment, update the masked target text and related statistical data, and record the adjustment operation and corresponding integrity index changes in the audit log. Recalculate the coverage rate, false occlusion rate, and format consistency indicators, and determine whether they all meet the preset standards. If not all conditions are met and the maximum adjustment round limit has not been reached, proceed to the next round of adaptive adjustment; if all conditions are met or the maximum adjustment round limit has been reached, terminate the adaptive adjustment process.

Images