Diversified text sensitive data synthesis method for desensitization effect evaluation

Abstract

The invention provides a diversified text sensitive data synthesis method for desensitization effect evaluation, which comprises the following steps: constructing a sensitive entity system meeting desensitization evaluation requirements, the sensitive entity system comprises a general field, a medical field and a financial field, and each field comprises a plurality of entity types; obtaining an original data set, counting entity distribution on the original data set, constructing a target distribution model, and designing a diversified strategy based on entity types and sentence patterns; guiding the large language model to generate candidate corpora according to the target distribution model and the diversification strategy, performing character-level alignment labeling and consistency verification on the candidate corpora, and generating a synthetic data set based on the candidate corpora; and performing multi-dimensional quality verification on the synthetic data set from the data layer, the entity layer and the semantic layer to obtain an evaluation result, and feeding back the evaluation result to a closed-loop controller to adjust a quota and a generation parameter so as to generate a final synthetic data set. The method can be applied to validity evaluation of a data desensitization tool, and the problems of single evaluation dimension, data sparsity and the like in text desensitization evaluation are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text data set generation of large language models, and in particular to a diversified text sensitive data synthesis method for desensitization effect evaluation. BACKGROUND

[0002] Text data desensitization processing is an important problem in the fields of privacy protection and natural language processing. With the wide application of large-scale language models in search, dialogue and intelligent assistants and other scenarios, sensitive information such as names, identity cards, financial and medical data is often carried in user input. If it is directly put into the model training or reasoning link without processing, it may not only lead to privacy leakage, but also bring task bias and compliance risk. Therefore, the text desensitization task has become a technical problem of common concern in academia and industry.

[0003] In recent years, there are still deficiencies in the annotation and construction of desensitization evaluation data sets in research work. Such a situation directly leads to the difficulty of fine-grained and systematic evaluation of existing desensitization methods, so it is urgent to construct a comprehensive and fine-labeled special evaluation data set. In existing research, academia and industry often use two methods to evaluate the desensitization effect. The first type is to directly use existing large-scale data sets for specific task scenarios, for example, using multiple large model evaluation data sets to test the macro performance of desensitization technology in complex dialogue and instruction scenarios, and selecting the NLP benchmark data set ACE2005 to observe the performance of the privacy protection framework in the event extraction task. This method is close to the real application background, but since these data sets are not designed for text desensitization, the coverage and annotation of sensitive information are not systematic, making it difficult to conduct fine-grained privacy protection effect analysis. The second type of method is to use large language models to synthesize customized data sets. GPT-4 Turbo generates a prompt set for specific sensitive entities and sensitive topics to evaluate detection and recognition capabilities. This method can flexibly construct samples according to task requirements, but has the problems of high generation cost and difference from real distribution.

[0004] To further improve the quality of evaluation and training data, more and more researches have begun to focus on the optimization of synthetic data generation technology. Some works improve the quality and diversity of data through different strategies, and realize self-correction by the introspection ability of LLM itself, so as to improve the rationality of the generation result; A more powerful external model is introduced as a judge to screen the candidate data to ensure the accuracy and consistency of the final corpus. At the same time, some researchers start from scalability and explore the automatic generation process of synthetic data, which reduces human intervention and improves generation efficiency by letting the model generate or optimize instructions automatically. The integrated data generation framework integrates generation, quality evaluation, screening and iterative optimization into a pipeline system to continuously improve the generation strategy through feedback loops.

[0005] In general, the existing research still has shortcomings. On the one hand, the desensitization evaluation dataset lacks a unified sensitive entity system and multi-domain coverage, making it difficult to meet the fine-grained analysis requirements. On the other hand, although synthetic data has flexibility and scalability, it still lacks balance, diversity and quality control. Therefore, a method that takes into account data system completeness, data balance and data diversity is needed to better support the research and application of text desensitization technology. SUMMARY

[0006] The purpose of the present application is to overcome the above technical deficiencies and provide a diversified text sensitive data synthesis method for desensitization effect evaluation, which solves the technical problems that the desensitization evaluation dataset in the prior art lacks a unified sensitive entity system and multi-domain coverage, making it difficult to meet the fine-grained analysis requirements. Although synthetic data has flexibility and scalability, it still lacks balance, diversity and quality control.

[0007] To achieve the above technical purpose, the technical scheme of the present application provides a diversified text sensitive data synthesis method for desensitization effect evaluation, comprising the steps of: A sensitive entity system that meets the desensitization evaluation requirements is constructed, including general domain, medical domain, and financial domain, each domain including multiple entity types; An original dataset is obtained, the entity distribution is counted on the original dataset, a target distribution model is constructed, and a diversification strategy is designed based on entity types and sentence patterns; A large language model is guided to generate candidate corpus according to the target distribution model and the diversification strategy, and character-level alignment annotation and consistency verification are performed on the candidate corpus, and a synthetic dataset is generated based on the candidate corpus; The synthetic dataset is verified from the data layer, entity layer and semantic layer to obtain evaluation results, and the evaluation results are fed back to the closed-loop controller to adjust the quota and generation parameters to generate the final synthetic dataset.

[0008] Compared with the prior art, the beneficial effects of the present application include:

[0009] The present application provides a text sensitive data synthesis method based on diversification strategy, which can be applied to the effectiveness evaluation of data desensitization tool, and solves the problems of single evaluation dimension and sparse data in text desensitization evaluation. The method introduces target distribution and diversity strategy on the original data set, designs entity and sentence generation mechanism, constructs multi-dimensional index system, and improves data quality through "calibration-generation-evaluation" closed loop iteration, generates data with balanced distribution and diversity, and is suitable for entity recognition and desensitization tasks in multiple sensitive scenarios.

[0010] According to some embodiments of the present application, the entity types in the general field include: name, age, gender, nationality, occupation, nationality, education background, marital status, political inclination, family members, geographical location, travel information; The entity types in the medical field include: disease, drug, clinical manifestation, medical procedure, allergy information, fertility information; The entity types in the financial field include: salary amount, investment product, tax record, credit record, entity asset, transaction information.

[0011] According to some embodiments of the present application, the entity distribution on the original data set is counted and the target distribution model is constructed, including the steps of: The entity categories in the original data set are counted to obtain the statistical results, and the frequency distribution of each entity type is obtained; On the basis of the statistical results, the target distribution is constructed in combination with the task requirements, the "total target number" parameter is set in the generation configuration file, the target number is specified for all entity types, and the individual target is set for specific entity types through the "individual target number" field.

[0012] According to some embodiments of the present application, the entity distribution on the original data set is counted and the target distribution model is constructed, including the steps of: The upper and lower limit tolerance interval is set for each entity type through the tolerance parameter.

[0013] According to some embodiments of the present application, the diversification strategy includes: sentence diversification generation and entity diversification generation.

[0014] According to some embodiments of the present application, the sentence diversification generation includes the steps of: Generate diversified sentence attributes: first call the API interface of the large language model, ask the model in the form of "meta question" to get the diversified sentence attributes that the sentence of the sensitive entity should pay attention to, and the diversified sentence attributes include: text content, sensitive granularity, language style; Generating diverse sentence types: For the generated diverse sentence attributes, further call the API interface of the large language model to generate multiple different types: The attribute "text content" is extended to: medical cases, bank transactions, job resumes, news reports, legal documents, financial contracts, social media updates, online customer service chat records; The attribute "sensitive granularity" is extended to: precise value, range value; The attribute "language style" is extended to: objective report, oral dialogue, formal report, command instruction.

[0015] According to some embodiments of the present application, the entity diversification generation includes: Vanilla entity generation and latent entity generation, Vanilla entity generation provides real and high-frequency basic entities, and latent entity combines sentence diversification strategy to generate rare or contextualized extended entities.

[0016] According to some embodiments of the present application, the large language model generates candidate corpus according to the target distribution model and the diversification strategy, including the steps of: Generating sentences: loading the target distribution file model, the sentence diversification configuration file and the diversified entity pool, and in the generation process, according to the configuration, selecting batch generation mode or single sentence generation mode; Labeling entities: labeling entities for the filtered sentences, identifying the start position, end position, entity text and type of the entities in the sentences during the labeling process, and realizing accurate positioning of sensitive entities; Verify the labeling: verify and save the labeling results to ensure the integrity and traceability of the generated data, and the verification process includes checking the coverage, boundary accuracy and consistency with the generated target distribution of the entity labeling.

[0017] According to some embodiments of the present application, the evaluation results are obtained by multi-dimensional quality verification of the synthetic data set from the data layer, the entity layer and the semantic layer, including the steps of: Diversity verification: Data layer diversity: evaluate the diversity of the text itself through lexical diversity, syntactic structure and context mode, lexical diversity is calculated by word segmentation analysis to ensure that the generated text uses multiple expression methods; syntactic diversity analyzes the sentence structure and grammar mode to avoid generating single sentence type; context diversity observes the change of the context mode before and after the sentence to evaluate the performance of the text in different contexts; Entity layer diversity: Analyze the usage of Vanilla entities and Latent entities, including the number distribution, uniqueness, and type coverage of Vanilla entities and Latent entities. By comparing the frequency of Vanilla entities and Latent entities, check if there are entity duplicates or bias towards specific entities. Semantic layer diversity: Calculate the similarity between sentences based on TF-IDF vectors or semantic embeddings to assess semantic differences and the semantic performance of entities in different contexts.

[0018] According to some embodiments of the present application, the multi-dimensional quality verification further includes: Balance verification: Calculate the actual distribution of each entity type in the generated data and compare it with the target distribution model.

[0019] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a text desensitization data synthesis method based on a diversified generation strategy is provided for an embodiment of the present application; Figure 2 A flowchart of a target distribution and diversified strategy generation is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0022] It should be noted that although functional modules are divided in the system schematic diagram and logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0023] Terminology: 1. Text De-identification: Text de-identification refers to the process of identifying, replacing, masking, or falsifying sensitive information (such as names, ID numbers, addresses, contact information, etc.) in text data to avoid privacy leakage during data storage, transmission, and sharing. In the context of the increasing application of big data and artificial intelligence, text de-identification is widely used in medical, financial, e-commerce, social platform, and other scenarios. For example, when hospitals open case data for research, they need to de-identify patient names, addresses, and medical record numbers. When banks train risk control models, they need to process sensitive information such as customer ID numbers and bank card numbers. This invention focuses on the sensitive data identification aspect of text de-identification, laying the foundation for subsequent de-identification processing and de-identification effect evaluation.

[0024] 2. Synthetic Dataset: Synthetic Dataset refers to a data set generated by algorithms, rules, or models, which has similar distribution characteristics as real data but does not contain real sensitive information. Synthetic Dataset can play a role in data privacy protection and model evaluation, and is often used to supplement insufficient original data distribution.

[0025] Existing research has made several valuable attempts in text de-identification evaluation and synthetic data generation, but these works still have obvious limitations.

[0026] (1) Limitations of existing de-identification evaluation datasets: Existing de-identification evaluation methods mostly rely on public datasets that are not specifically designed for privacy protection, such as general NLP benchmarks or large model evaluation datasets. Such datasets have limited coverage of sensitive information, and the annotation system lacks uniformity and fine-grained specifications, resulting in macro-level analysis of de-identification effects and difficulty in accurately evaluating the processing capabilities of different categories of sensitive information.

[0027] (2) Deficiencies of synthetic data generation methods: Existing methods often rely on self-correction of a single model or limited human intervention, making it difficult to achieve large-scale, automated, and multi-dimensional quality assurance. Although there have been explorations of automated data generation frameworks, a complete closed-loop system covering data generation, screening, evaluation, and optimization has not yet been formed.

[0028] In view of the above shortcomings and deficiencies, this invention aims to propose a diversified text de-identification data synthesis method for de-identification effect evaluation. The purpose is to: build a systematic evaluation dataset covering multiple categories of sensitive information, enabling de-identification effects to be comprehensively measured at a fine-grained level; introduce target distribution and diversified generation strategies to improve the balance and diversity of synthetic data; combine feedback loop optimization mechanisms to ensure that generated data has stability and scalability in balance, diversity, and quality. In this way, this invention can provide unified, reliable, and efficient data support for the research and application of text de-identification technology.

[0029] Referring to Figure 1 and Figure 2 , Figure 1 A flowchart of a text desensitization data synthesis method based on diversified generation strategy provided for an embodiment of the present application; Figure 2 A flowchart of target distribution and diversified strategy generation provided for an embodiment of the present application.

[0030] In an embodiment, a diversified text sensitive data synthesis method for desensitization effect evaluation includes the steps of: constructing a sensitive entity system that fits the desensitization evaluation requirements, including general fields, medical fields, and financial fields, each field including multiple entity types; obtaining an original data set, counting entity distribution on the original data set and constructing a target distribution model, and designing diversified strategies based on entity types and sentence patterns; generating candidate corpus guided by the target distribution model and diversified strategies using a large language model, and performing character-level alignment annotation and consistency verification on the candidate corpus to generate a synthesized data set; performing multi-dimensional quality verification on the synthesized data set from the data layer, entity layer and semantic layer to obtain evaluation results, and feeding the evaluation results back to a closed-loop controller to adjust the quota and generation parameters to generate a final synthesized data set.

[0031] The present application establishes a target distribution and dynamically calculates the gap between the current distribution and the target distribution for unbalanced original data sets; through a tolerance range control and automatic calibration mechanism, the synthesized data maintains balance between entity categories, solving the problem of unbalanced data distribution generated by existing methods.

[0032] The present application combines sentence diversification and entity diversification to form a unified diversified generation strategy; sentence diversification ensures sentence richness under different contexts, expression methods and information density through attribute and category generation mechanism; entity diversification generates more sensitive entities with higher coverage under the guidance of diversified attribute values by introducing two types of entity pools, Vanilla entities and Latent entities, to improve data semantic level and realism.

[0033] The present application adopts a "calibration-generation-evaluation" cycle mode, and adjusts the strategy based on distribution gap, diversity index and naturalness score in each iteration; continuously optimizes the quality of generated data, realizes dynamic self-adaptation and stable convergence of the generation process; combines parallel processing and caching mechanism, and takes into account efficiency and controllability, to ensure the scalability and traceability of the synthesized data.

[0034] The application provides a text desensitization data synthesis method based on a diversified generation strategy, which integrates the organic combination of target distribution generation and automatic calibration mechanism, diversification strategy and closed-loop iterative optimization framework, and can continuously generate balanced, natural and diverse text containing sensitive entity data under different scenarios and application requirements. The existing technologies mostly focus on single data enhancement or random synthesis, which cannot guarantee the balance of entity distribution and the systematic diversity of corpus, leading to the generated data deviating from the real application scenario, and it is difficult to meet the requirements of text desensitization evaluation.

[0035] Stronger entity distribution balance: The existing methods generally rely on manual setting or static sampling in data distribution control, which easily leads to excessive scarcity or overabundance of individual entity types. The application uses target distribution generation and automatic calibration mechanism to make the entity distribution of the generated data highly consistent with the preset target, thereby ensuring the balance and representativeness of the data.

[0036] Rich data diversity: Existing technologies often only perform shallow diversification at the sentence level, ignoring the combination of entity diversity and semantic context, resulting in insufficient diversity of synthesized text. The application introduces a collaborative strategy of sentence diversification and entity diversification, which can generate Vanilla entities to meet basic sensitive information coverage and extend scenario-based expressions by combining context attribute values (such as medical reports) in Latent entities, thereby obtaining data closer to real context.

[0037] Quality optimization is more sustainable and stable: The application realizes the continuous improvement of data quality through the closed-loop iterative optimization framework of "calibration-generation-evaluation". The application can find distribution deviation or lack of diversity in each iteration and optimize the strategy, thereby gradually improving the balance and semantic quality of the data. This automated and adaptive iterative optimization mechanism ensures efficiency while significantly improving the authenticity and usability of synthesized data, providing better technical support for text desensitization evaluation tasks.

[0038] The detailed method is described as follows: I. Construction of sensitive entity system Before generating text desensitization data, a sensitive entity system needs to be constructed. According to domestic and foreign privacy protection specifications (such as GDPR "General Data Protection Regulation", China's "Personal Information Protection Law") and the entity recognition requirements of mainstream desensitization tools, the application designs and constructs a sensitive entity system that meets the needs of text desensitization evaluation. In order to better illustrate the application, Table 1 shows examples of sensitive entity field and entity type definitions, including but not limited to general, medical and financial fields, and defines 24 sensitive entity types as examples.

[0039] Table 1: Examples of sensitive entity field and entity type definitions

[0040] II. Target distribution and generation of diversification strategy On the basis of constructing the sensitive entity system, in view of the problems of unbalanced entity categories and monotonous data existing in the original data set, a data synthesis strategy of "target distribution driving + diversification strategy guiding" is proposed, as shown in the following formula: Figure 2 By setting the target distribution before data generation and introducing sentence diversification and entity diversification control in the generation process, the balance and diversity of the synthesized data can be effectively improved, and the generated data is more suitable for text desensitization evaluation tasks. Specifically, the following three aspects are included: 2.1 Entity distribution target generation The entity distribution target is the global control benchmark of the synthesized data. Due to the significant difference in the number of different entity categories in the original data set, if directly generated, it is easy to lead to the excessive concentration of some entity categories and the serious shortage of some other categories. Therefore, a reasonable distribution target needs to be set before synthesis.

[0041] (1) Distribution statistics and analysis: First, the entity categories in the original data set are counted to obtain the frequency distribution of each entity category. For example, in the original corpus, the name entity category may account for more than 30%, while the political inclination entity category accounts for less than 5%. Such distribution cannot directly meet the "comprehensive coverage" and "balanced training" requirements of desensitization evaluation.

[0042] (2) Target distribution setting: On the basis of the statistical results, the target distribution is constructed in combination with the task requirements. By setting the "total target number" parameter in the generation configuration file, the target number is specified for each entity type. At the same time, the "individual target number" field is set for specific entity types to support increasing the generation proportion of high-risk field entities such as finance and medicine.

[0043] (3) Tolerance and dynamic adjustment: In order to avoid the difficulty of generation caused by too strict constraints, the invention sets the upper and lower limit tolerance interval for each entity type through the tolerance parameter. The default tolerance is 10%, for example, if the target number is 100, the range of 90~110 is allowed. In the generation process, the entity types that need to be supplemented and pruned are found through dynamic calculation, if a certain entity type is insufficient or exceeds the standard, the generation quota of the subsequent batch is dynamically adjusted to ensure that the final overall distribution converges to the target interval. At the same time, the configuration file of diversification is automatically updated to ensure that the entity diversification degree is consistent with the target distribution.

[0044] 2.2 Sentence diversification generation Sentence diversification is a key strategy to improve the naturalness and expressiveness of synthetic text. The invention achieves fine-grained control over sentence generation through a multi-dimensional attribute control method, ensuring that different sensitive entities can appear in diverse contexts and expressions, thereby avoiding corpus monotony and pattern fixation. The specific process is as follows.

[0045] (1) Generate diverse sentence attributes: First, call the API interface of the large language model and ask the model in the form of a meta-question, such as: "What important attributes should be considered when generating sentences containing sensitive entities?". The model's return result usually contains 3-5 key dimensions, such as: text content, sensitive granularity, language style.

[0046] (2) Generate diverse sentence types: For the generated diverse sentence attributes, further call the API interface of the large language model to generate 5 different types.

[0047] For example, the attribute "text content" can be expanded to: medical case, bank transaction, job resume, news report, legal document, financial contract, social media dynamic, online customer service chat record; The attribute "sensitive granularity" can be expanded to: precise value, range value; The attribute "language style" can be expanded to: objective report, spoken dialogue, formal report, command instruction.

[0048] After generation, save the above results as two strategy files for subsequent data synthesis stage calling. In this way, during actual synthesis, only different attribute and attribute value combinations need to be selected to quickly generate diverse sentences, achieving attribute-driven expressiveness.

[0049] 2.3 Entity diversification generation Entity diversification is another core step to improve data authenticity and coverage. The invention divides entity diversification into Vanilla entity generation and Latent entity generation, which complement each other: Vanilla entity provides real and high-frequency basic entities, and Latent entity generates rare or contextualized extended entities in combination with sentence diversification strategies, thereby achieving the comprehensiveness and diversity of the entity system.

[0050] (1) Vanilla entity generation: The goal of Vanilla entity generation is to obtain a set of basic sensitive entity collection to ensure the authenticity and effectiveness of entities in the data synthesis process, which can cover the main sensitive categories. First, call the large language model API to ask questions as a data privacy expert, for example: "Assume you are a data privacy expert, please list 15 different disease entities that belong to sensitive information". This prompt ensures that the output disease entities are closely related to privacy-sensitive data, and limits the output quantity (15), ensuring that the result size is controllable. Then, process the returned results. The disease entities returned by the model may have synonyms, duplicates, or inconsistent formats. Remove duplicates and standardize these results. Finally, form a Vanilla entity pool for subsequent synthetic data.

[0051] (2) Latent entity generation: The goal of Latent entity generation is to generate sensitive entities with contextual features according to diversified sentence types, enhancing the diversity and real scenario coverage of synthetic data. In the sentence diversification section, several sentence attributes and corresponding attribute sentence types have been generated. For example, for the "text content" attribute, it may contain medical cases, bank transactions, and job resumes. When generating Latent entities, these sentence types are embedded into the prompt as context constraints. Based on different sentence types, contextual questions are constructed. For example, when the sentence type is "medical report", the prompt is: "Assume you are a data privacy expert, please list 15 different disease entities that belong to sensitive information in the medical report". Compared with Vanilla entities, this requires not only output of disease entities, but also compliance with the specified context. Then, call the large language model API for multiple attribute values to form a set of Latent entities under different contexts. All generated Latent entities and Vanilla entity pools complement each other.

[0052] III. Data synthesis based on diversification strategy This step integrates entity target distribution, sentence diversification, and entity diversification strategies to generate high-quality text desensitization data. The entire process can be summarized into three stages: sentence generation, entity annotation, and verification annotation.

[0053] (1) Sentence generation: First, load the target distribution file, sentence diversification configuration file, and diversification entity pool. During generation, automatically select batch generation mode or single sentence generation mode according to the configuration. Batch generation mode can generate multiple sentences at a time to improve efficiency, while single sentence generation mode generates sentences one by one to ensure fine control.

[0054] (2) Entity annotation: Next, the filtered sentences are annotated with entities. During the annotation process, the start and end positions of entities in the sentences, entity text, and type are identified to accurately locate sensitive entities. This stage supports single-type annotation and multi-round annotation methods to adapt to different complexity requirements of entity recognition. Multiple entity types can be processed in parallel during the annotation process, significantly improving efficiency through process-level parallelism, asynchronous parallelism, or hybrid parallelism strategies, while ensuring the accuracy of each entity type annotation.

[0055] (3) Verification and annotation: Finally, the annotation results are verified and saved to ensure the integrity and traceability of the generated data. The verification process includes checking the coverage, boundary accuracy, and consistency of the entity annotation with the target distribution. During the generation process, intermediate results are saved in real-time, including the original generated sentences, filtered sentences, and annotation results. The generated data for each entity type is archived separately to facilitate subsequent debugging, problem tracking, and iterative optimization. This closed-loop process not only ensures the diversity and annotatability of the data, but also provides a reliable foundation for subsequent "calibration-generation-evaluation" iterations.

[0056] Four, quality verification of synthetic data This step is a key link to ensure that the generated data meets the actual application requirements. A multi-dimensional quality evaluation system is used to comprehensively verify the generated data. The entire verification process can be summarized as balance verification and diversity verification.

[0057] 4.1 Balance verification On the one hand, balance verification is performed to ensure that the entity distribution of the generated data is consistent with the preset target distribution. The actual distribution of each entity type in the generated data is calculated and compared with the target distribution. Due to the use of a tolerance mechanism, a certain range of distribution deviation is allowed. If the number of entities is within the tolerance range, the balance verification is passed. Balance verification can effectively ensure that different entity types are fully utilized in the data set, avoiding over-generation or under-generation of a few entities, thereby providing a balanced data foundation for text desensitization evaluation tasks.

[0058] 4.2 Diversity verification On the other hand, diversity verification is performed from three dimensions: data layer, entity layer, and semantic layer, to evaluate the richness of the generated data in expression, structure, and semantics.

[0059] (1) Data layer diversity: Evaluate the diversity of the text itself through vocabulary diversity, syntactic structure, and context pattern. Vocabulary diversity calculates the vocabulary richness through word segmentation analysis to ensure that the generated text uses multiple expression methods; syntactic diversity analyzes sentence structure and grammar patterns to avoid generating single sentence types; context diversity evaluates the text's performance ability in different contexts by observing the changes in the pre and post sentence context patterns.

[0060] (2) Entity layer diversity: Analyze the usage of entities, including the number distribution, uniqueness, and type coverage. By comparing the frequency of Vanilla entities and Latent entities, check if there are entity duplicates or biases towards certain specific entities, and ensure the diversity and coverage of entity combinations in sentences.

[0061] (3) Semantic layer diversity: Calculate the similarity between sentences based on TF-IDF vectors or semantic embeddings, evaluate the semantic difference and the semantic performance of entities in different contexts. Semantic layer diversity can verify that the generated data retains the semantic accuracy of entities while providing rich semantic expression, thereby enhancing the generalization ability of the model in real scenarios.

[0062] During the diversity verification process, text naturalness evaluation, annotation accuracy check, semantic consistency analysis, and linguistic quality evaluation are performed simultaneously. Naturalness evaluation is scored by a large language model API, and sentences below the threshold are marked as abnormal; annotation accuracy verification checks the correctness of entity boundaries and types; semantic consistency analysis checks the matching degree of entities and context; linguistic quality evaluation scores from the dimensions of grammar, fluency, and readability. Finally, the verification results are generated into a visual report, including distribution comparison charts, radar charts, heat maps, and interactive HTML reports, and the comprehensive evaluation report is generated by summarizing the indicators, providing a comprehensive reference for data quality.

[0063] Through the coordinated verification of balance and diversity in two stages, it is ensured that the generated data meets the preset standards in terms of quantity, expression, and semantic level, thereby providing high-quality, diversified, and annotatable data basis for text desensitization evaluation tasks.

[0064] Five, "calibration-generation-evaluation" closed-loop iterative framework Based on the previous four modules: 1) sensitive entity system construction, 2) target distribution and diversity strategy generation, 3) data synthesis based on diversity strategy, 4) synthesis data quality verification. This invention builds a "calibration-generation-evaluation" closed-loop iterative framework, realizing a sustainable optimization data generation process, its process is shown in Table 2 Algorithm 1.

[0065] Table 2: "Calibration-Generation-Evaluation" Closed-Loop Iterative Process Algorithm

[0066] The variable names and function names in Algorithm 1 are summarized and explained as follows: original_dataset: The original dataset. As the input of the entire process, it is the basis data that needs to be expanded and optimized.

[0067] max_iterations: Maximum number of iterations. Controls the upper limit of the optimization loop, preventing the process from running indefinitely.

[0068] current_dataset: Current dataset. The version of the dataset that is dynamically modified and expanded in each iteration, is the core object of the entire process.

[0069] base_strategies: Base diversification strategies. The basic rule set for sentence and entity diversification, initialized at the beginning of the process.

[0070] gap: Distribution gap. Calculated at the beginning of each iteration, measures the gap between the number of entities in the current dataset and the preset target.

[0071] diversity: Diversity index. Measures the richness of the current dataset in terms of vocabulary, syntax, semantics, etc.

[0072] current_strategy: Current iteration strategy. Dynamically adjusted according to the gap and diversity of this round, generates a specific plan to guide the data synthesis of this round.

[0073] new_data_batch: Newly generated data batch. In a single iteration, a small batch of synthetic data is generated according to the current_strategy.

[0074] final_dataset: Final synthetic dataset. After all iterations of optimization, the final output of high-quality dataset.

[0075] quality_report: Quality assessment report. The final output of the visual comprehensive report, fully displays the evaluation results of final_dataset in terms of balance and diversity.

[0076] IterativeSynthesis function: Iterative synthesis main function. As the entrance of the algorithm, responsible for organizing and calling all steps, completing the whole process from data initialization, iterative optimization to final verification.

[0077] TrimDataset function: Trim dataset function. Used to delete samples in the dataset that exceed the upper limit of the target number, called at the beginning of the process and after each new data is merged, to ensure the balance of entity distribution.

[0078] InitializeStrategies function: Initialize strategies function. Executed once at the beginning of the process, generates basic sentence and entity diversification strategies, as the starting point for subsequent iteration adjustment.

[0079] AnalyzeCurrentState function: Analyze the current state function. Called at the beginning of each iteration, it comprehensively evaluates the current_dataset and returns the current distribution gap and diversity metrics.

[0080] TerminationConditionMet function: Determine the termination condition function. Based on the input gap and diversity, determine whether the dataset quality has met the standard, thus deciding whether to end the iteration loop early.

[0081] AdjustStrategyForIteration function: Adjust the iteration strategy function. This is the core of the iteration optimization, which dynamically adjusts the generation strategy based on the current state analysis results to make up for deficiencies, such as prioritizing the generation of rare entity types.

[0082] SynthesizeDataBatch function: Synthesize data batch function. According to the adjusted strategy of this round, call the language model to generate and annotate a batch of new data.

[0083] MergeDatasets function: Merge datasets function. Incorporate the newly generated new_data_batch into the current_dataset to expand the size and diversity of the dataset.

[0084] FinalBalanceValidation function: Final balance validation function. After all iterations are completed, conduct a comprehensive entity distribution balance check on the final dataset.

[0085] FinalDiversityValidation function: Final diversity validation function. Conduct a comprehensive diversity assessment of the final dataset, covering multiple levels such as data, entities and semantics.

[0086] GenerateVisualReport function: Generate visual report function. Summarize all the final validation results and generate a comprehensive quality report with pictures and text.

[0087] The "calibration-generation-evaluation" closed-loop iteration framework continuously optimizes the quality and distribution balance of synthetic data through three-stage cyclic iteration.

[0088] (1) Calibration phase: The calibration phase is the starting point of each iteration, responsible for analyzing the current state and developing optimization strategies. The system first calculates the entity distribution and diversity indicators of the current dataset, compares them with the target distribution to obtain the gap analysis. Based on the gap analysis, the system intelligently adjusts the generation strategy, such as enabling sentence diversification, entity diversification, or example sentence functions. At the same time, the system diagnoses the current problems, such as large distribution gap, insufficient diversity, etc., and provides corresponding improvement suggestions. Finally, check if the termination condition is met, if met, exit the iteration loop.

[0089] (2) Generation phase: The generation phase executes specific data generation tasks according to the strategy adjustment of the calibration phase. The system first generates a strategy file, including a target distribution file (indicating the number of entities to be supplemented), sentence diversification configuration, and entity diversification data. Then update the NER configuration, pass the strategy file path and generation feature configuration to the data generation module. The data generation module calls the large language model API to generate synthetic data containing target entities. After generation, the system merges the new data with the existing data and prunes according to the target distribution to ensure that it does not exceed the tolerance range.

[0090] (3) Evaluation phase: The evaluation phase evaluates the quality of the generated data to provide feedback for the next iteration. The system saves the detailed data of the current iteration, including the dataset, distribution gap, diversity indicators, etc. Then run the quality evaluation script to evaluate the data quality from multiple dimensions, including balance verification, diversity verification, naturalness evaluation, etc. The evaluation results are saved to the corresponding iteration directory to provide the basis for subsequent analysis and optimization.

[0091] Among them, the entity types in the general field include: name, age, gender, nationality, occupation, nationality, education background, marital status, political inclination, family members, geographical location, travel information; the entity types in the medical field include: disease, drug, clinical manifestation, medical procedure, allergy information, fertility information; the entity types in the financial field include: salary amount, investment product, tax record, credit record, entity asset, transaction information.

[0092] Among them, the entity distribution is counted on the original dataset and the target distribution model is constructed, including the steps: count the entity categories in the original dataset to obtain the statistical results, and obtain the frequency distribution of each entity type; based on the statistical results, combine the task requirements to construct the target distribution, set the "total target number" parameter in the generation configuration file, and specify the target number for all entity types, set individual target numbers for specific entity types through the "individual target number" field. Set the upper and lower limit tolerance interval for each entity type through the tolerance parameter.

[0093] Among them, the diversification strategy includes: sentence diversification generation and entity diversification generation.

[0094] The sentence diversification generation includes the steps of generating diversified sentence attributes: first, calling the API interface of the large language model to ask the model in the form of a meta-question to obtain the diversified sentence attributes that the sentence of the sensitive entity is concerned about, including: text content, sensitive granularity, and language style; generating diversified sentence types: for the generated diversified sentence attributes, further calling the API interface of the large language model to generate multiple different types: the attribute “text content” is expanded to: medical cases, bank transactions, job resumes, news reports, legal documents, financial contracts, social media updates, and online customer service chat records; the attribute “sensitive granularity” is expanded to: precise value and range value; and the attribute “language style” is expanded to: objective report, spoken dialogue, formal report, and command instruction.

[0095] The entity diversification generation includes two parts: Vanilla entity generation and Latent entity generation, the Vanilla entity generation provides real and high-frequency basic entities, and the Latent entity generation generates rare or contextualized extended entities in combination with the sentence diversification strategy.

[0096] According to the target distribution model and the diversification strategy, the large language model generates candidate corpus, including the steps of: generating sentences: loading the target distribution file model, the sentence diversification configuration file and the diversified entity pool, during the generation process, batch generation mode or single sentence generation mode is selected according to the configuration; labeling entities: labeling entities for the filtered sentences, during the labeling process, the start position, end position, entity text and type of the entities in the sentences are identified to realize accurate positioning of sensitive entities; verifying the labeling: verifying and saving the labeling results to ensure the integrity and traceability of the generated data, the verification process includes checking the coverage, boundary accuracy and consistency with the generated target distribution of the entity labeling.

[0097] In terms of generation controllability, “batch generation” and “single sentence generation” modes are supported, which can be flexibly switched according to evaluation needs. The batch mode is suitable for quickly building large-scale basic corpus, meeting the performance evaluation of desensitization algorithms in the “data sufficient” scenario; the single sentence mode is suitable for targeted supplement of special cases, solving the problem of “long tail scene coverage deficiency” in batch generation, ensuring the comprehensiveness of evaluation. At the same time, by loading the “target distribution file model”, “sentence diversification configuration file” and “diversified entity pool”, the generation process is completely controllable, avoiding the deviation of data features caused by the “free generation” of the large language model, and ensuring that the synthesized corpus can effectively simulate the data distribution in the actual desensitization task.

[0098] In terms of annotation accuracy, the annotation process not only identifies entity types, but also accurately records the starting position, ending position and entity text, achieving "character-level" sensitive entity positioning, solving the problem of "ambiguous boundaries" in traditional annotation, and providing an objective basis for subsequent evaluation of the "coverage" and "precision" of the desensitization algorithm. Moreover, the annotation logic strictly corresponds to the "sensitive entity system" constructed in the early stage, ensuring that the annotation results match the evaluation scenario and avoiding evaluation bias caused by generalized annotation.

[0099] In terms of quality reliability, the verification process includes "entity annotation coverage", "boundary accuracy" and "consistency with target distribution", which eliminates unqualified corpus from three aspects, ensuring that sensitive entities are not missed, annotation positions are correct, and entity distribution conforms to the real scene. The annotation results are "saved" and associated with the generated parameters to form a complete data pedigree, which facilitates subsequent analysis and provides support for "abnormal result tracing" in desensitization evaluation.

[0100] This embodiment ensures that the corpus is close to the real scene through "controllable generation logic", provides evaluation benchmarks through "fine-grained annotation", and guarantees data quality through "multi-dimensional verification". The three work together to lay the foundation for the effectiveness of the subsequent synthetic dataset, ultimately serving the core goal of "accurately evaluating the effect of desensitization algorithm".

[0101] Among them, the evaluation result of the synthetic dataset is obtained by multi-dimensional quality verification from the data layer, entity layer and semantic layer, including the steps: diversity verification: data layer diversity: evaluate the diversity of the text itself through vocabulary diversity, syntactic structure and context mode, vocabulary diversity is calculated by word segmentation analysis to ensure that the generated text uses multiple expression methods; syntactic diversity analyzes the sentence structure and grammar pattern to avoid generating a single sentence type; context diversity observes the changes in the context mode before and after the sentence to evaluate the text's performance ability in different contexts; entity layer diversity: analyze the usage of Vanilla entities and Latent entities, including the number distribution, uniqueness and type coverage of Vanilla entities and Latent entities, by comparing the usage frequency of Vanilla entities and Latent entities, check if there are entity duplication or bias towards specific entities; semantic layer diversity: calculate the similarity between sentences based on TF-IDF vector or semantic embedding to evaluate semantic difference and semantic performance of entities in different contexts.

[0102] The application can comprehensively guarantee the quality of the generated text by evaluating vocabulary diversity, syntactic structure and context pattern. With the help of word segmentation analysis, the lexical richness is calculated to measure the lexical diversity, which can encourage the generated text to use various expressions and avoid monotonous repetition of words, making the text more suitable for actual language use scenarios. By analyzing syntactic diversity, focusing on sentence structure and grammar patterns, it can effectively avoid the generation of single sentence type, making the sentence of the text more diverse and enhancing readability and naturalness. By observing the changes in the context pattern to consider the context diversity, it helps to evaluate the performance of the text in different contexts, so that it can better adapt to various actual application contexts.

[0103] The use of Vanilla entities and Latent entities in the entity layer diversity verification link is of great significance. Specifically, the number distribution, uniqueness and type coverage of Vanilla entities and Latent entities are examined, and the use frequency of the two is compared, so that the problem of entity repetition or bias towards specific entities can be accurately checked. This helps to ensure that the entities in the synthesized data set are more reasonable and balanced, avoiding the impact of uneven entity distribution on the accuracy of the de-sensitization effect evaluation, so that the evaluation based on the data set can more comprehensively and objectively reflect the true situation.

[0104] In the semantic layer diversity verification, the similarity between sentences is calculated based on TF-IDF vectors or semantic embeddings to evaluate semantic differences and the semantic performance of entities in different contexts. This approach can deeply mine the semantic information of the text, clearly understand the differences and relationships between different sentences, and accurately grasp the meaning changes of entities in diverse contexts, thereby ensuring that the synthesized data set is rich and reasonable in the semantic layer, providing a solid semantic foundation for subsequent accurate and reliable evaluation of the de-sensitization effect, and avoiding evaluation bias due to defects in the semantic layer.

[0105] Further multi-dimensional quality verification also includes balance verification: calculating the actual distribution of each entity type in the generated data and comparing it with the target distribution model. Through this comparison, the accuracy of the data can be accurately controlled. The target distribution model is constructed based on the statistics of the original data set, reflecting the reasonable distribution state of each type of entity in the real scenario. Comparing the distribution of entity types in the actual generated data with the target distribution model can clearly know whether there is deviation in the generation process, such as whether a certain type of entity appears too much or too little in the generated data, so that the possible unreasonable data generation can be discovered and corrected in time, ensuring that the synthesized data set can be closest to the real data appearance.

[0106] The embodiment helps to improve the reliability of desensitization evaluation. The effect evaluation of the desensitization algorithm often depends on the quality and rationality of the data. Only when the entity type distribution in the generated data meets the expected balanced state, and the desensitization algorithm is tested based on these data, can an objective and convincing evaluation result be obtained. If the entity distribution is unbalanced, some entity-related desensitization may be overrepresented or underrepresented, which may affect the accurate judgment of the effect of the entire desensitization algorithm. The balance verification can avoid such problems and make the evaluation more reliable.

[0107] The memory, as a kind of non-transient computer readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transient memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory can optionally include remote memory relative to the processor, which can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The device embodiments described above are only schematic, and units described as separate components can or can not be physically separated, and can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0109] In addition, one embodiment of the present application also provides a computer readable storage medium, which stores computer executable instructions. The computer executable instructions are executed by a processor or a controller, for example, by a processor in the above terminal embodiment, so that the above processor executes the diversified text sensitive data synthesis method for desensitization effect evaluation in the above embodiment.

[0110] Those of ordinary skill in the art will appreciate that all or certain steps, systems in the methods disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is common knowledge to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0111] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are included in the scope defined by the claims of the present application.

[0112] The above specific embodiments of the present application do not constitute a limitation on the scope of protection of the present application. Any other corresponding changes and modifications made according to the technical concept of the present application should be included in the scope of protection of the claims of the present application.

Claims

1. A method for diversification of text sensitive data synthesis for desensitization effect evaluation, characterized in that, The method comprises the steps of: Constructing a sensitive entity system that meets the needs of the evaluation of desensitization fitting, including general field, medical field, and financial field, each field including multiple entity types; Obtaining an original data set, statistically analyzing entity distribution on the original data set, constructing a target distribution model, and designing diversification strategies based on entity types and sentence patterns; According to the target distribution model and the diversification strategy, guiding the large language model to generate candidate corpus, and performing character-level alignment annotation and consistency verification on the candidate corpus, and generating a synthetic data set based on the candidate corpus; From the data layer, entity layer and semantic layer, the synthetic data set is verified in multiple dimensions to obtain the evaluation results, and the evaluation results are fed back to the closed-loop controller to adjust the quota and generation parameters to generate the final synthetic data set.

2. The method for diversification of text sensitive data synthesis for desensitization effect evaluation as claimed in claim 1, wherein, The entity types of the general field include: name, age, gender, nationality, occupation, nationality, education background, marital status, political inclination, family members, geographical location, travel information; The entity types of the medical field include: disease, drug, clinical manifestation, medical procedure, allergy information, and fertility information; The entity types of the financial field include: salary amount, investment product, tax record, credit record, entity asset, and transaction information.

3. The method for diversification of text sensitive data synthesis for desensitization effect evaluation as claimed in claim 1 wherein, Statistically analyzing entity distribution on the original data set and constructing a target distribution model, comprising the steps of: Statistically analyzing the entity categories in the original data set to obtain statistical results, and obtaining the frequency distribution of each entity type; Based on the statistical results, the target distribution is constructed in combination with the task requirements, and the "total target number" parameter is set in the generation configuration file, and the target number is specified for all entity types, and the "individual target number" field is used to set individual targets for specific entity types.

4. The method for diversification of text sensitive data synthesis for desensitization effect evaluation of claim 3, wherein, Statistically analyzing entity distribution on the original data set and constructing a target distribution model, further comprising the steps of: Set the upper and lower limit tolerance interval for each type of entity by the tolerance parameter.

5. The method for diversification of text sensitive data synthesis for desensitization effect evaluation as claimed in claim 1 wherein, The diversification strategy includes sentence diversification generation and entity diversification generation.

6. The method for diversification of text sensitive data synthesis for desensitization effect evaluation of claim 5, wherein, Sentence diversification generation includes the steps of: Generating diversified sentence attributes: first, call the API interface of the large language model, ask the model in the form of "meta question" to get the diversified sentence attributes that the sensitive entity sentence should pay attention to, and the diversified sentence attributes include: text content, sensitive granularity, language style; Generating diversified sentence types: for the generated diversified sentence attributes, further call the API interface of the large language model to generate multiple different types: The attribute "text content" is expanded to: medical case, bank transaction, job resume, news report, legal document, financial contract, social media dynamic, online customer service chat record; The attribute "sensitive granularity" is expanded to: accurate value, range value; The attribute "language style" is expanded to: objective report, oral dialogue, formal report, command instruction.

7. The method for diversification of text sensitive data synthesis for desensitization effect evaluation as claimed in claim 5, wherein, Entity diversification generation includes: Vanilla entity generation and latent entity generation, Vanilla entity generation provides real and high-frequency basic entities, and Latent entity generates rare or contextualized extended entities in combination with sentence diversification strategies.

8. The method for diversification of text sensitive data synthesis for desensitization effect evaluation as claimed in claim 1 wherein, According to the target distribution model and the diversification strategy, guiding the large language model to generate candidate corpus, comprising the steps of: Generate sentences: Load target distribution file model, sentence diversification configuration file and diversification entity pool. During the generation process, select batch generation mode or single sentence generation mode according to the configuration; Label entities: Perform entity labeling on the filtered sentences. In the labeling process, identify the start position, end position, entity text and type of the entity in the sentence, and achieve accurate positioning of sensitive entities; Verify labeling: Verify and save the labeling results to ensure the integrity and traceability of the generated data. The verification process includes checking the coverage, boundary accuracy and consistency with the generated target distribution of entity labeling.

9. The method for diversified text sensitive data synthesis for desensitization effect evaluation of claim 4, wherein, From the data layer, entity layer and semantic layer, the evaluation results of the synthesized data set are obtained through multi-dimensional quality verification, including the following steps: Diversity verification: Data layer diversity: Evaluate the diversity of the text itself through vocabulary diversity, syntactic structure and context mode. Vocabulary diversity calculates the vocabulary richness through word segmentation analysis to ensure that the generated text uses multiple expressions. Syntactic diversity analyzes sentence structure and grammar patterns to avoid generating single sentence types. Context diversity observes the changes in the context mode before and after the sentence to evaluate the text's performance ability in different contexts. Entity layer diversity: Analyze the usage of Vanilla entities and Latent entities, including the number distribution, uniqueness and type coverage of Vanilla entities and Latent entities. By comparing the usage frequency of Vanilla entities and Latent entities, check whether there are problems such as entity repetition or bias towards specific entities. Semantic layer diversity: Calculate the similarity between sentences based on TF-IDF vectors or semantic embeddings to evaluate semantic difference and semantic performance of entities in different contexts.

10. The method for diversification of text sensitive data synthesis for desensitization effect evaluation of claim 9, wherein, Multi-dimensional quality verification also includes: Balance verification: Calculate the actual distribution of each entity type in the generated data and compare it with the target distribution model.

Images