Privacy data desensitization method and system for large language model field

Abstract

The application relates to the technical field of artificial intelligence large model security, in particular to a privacy data desensitization method and system for the field of large language models, in the aspect of privacy identification, a light large language model is used for context semantic reasoning, the limitation that regular and named entity recognition can only match explicit format privacy is broken through, implicit associated privacy is accurately mined, and the problem of non-standard privacy missing detection is fundamentally solved. In the aspect of desensitization replacement, for structured data, same distribution simulation generation is adopted to keep statistical characteristics, for unstructured text, placeholder replacement is carried out according to semantic roles to keep complete syntax structure, and cross-paragraph associated privacy is implemented to cut off the combination restoration path, so that the desensitized corpus can maintain the integrity of semantic logic while erasing sensitive information. A special privacy discrimination model is introduced to inspect the desensitized data sentence by sentence, automatic backflow reprocessing of unqualified data is realized, a closed loop verification mechanism is formed, and bidirectional balance between desensitization safety and model precision is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence large model security technology, specifically to privacy data desensitization methods and systems for the field of large language models. Background Technology

[0002] Existing large-scale model training data sources mostly come from internal enterprise business data, chat logs, contract texts, work order data, user interaction corpora, etc. These data are a mixture of structured (documents, accounts, codes), semi-structured (form text), and unstructured natural text, embedding three types of sensitive data: explicit privacy, contextual implicit privacy, and cross-paragraph privacy. Current mainstream anonymization technologies in the industry fall into three categories, each with its own unavoidable technical flaws:

[0003] Traditional desensitization schemes based on fixed regular expression rules rely solely on pre-defined regular expressions to match fixed-format privacy fields. They can only identify standardized explicit privacy such as ID cards and mobile phone numbers, but cannot identify context-derived privacy (such as inferring a natural person's identity indirectly through place of origin and date of birth, or sporadic descriptions of confidential project amounts). Using uniform asterisk masks and random character replacements can disrupt the original sentence's syntax and semantic logic. After desensitization, the text's word order and semantic features shift, leading to fitting bias and a sharp drop in question-answering accuracy when large models are fine-tuned based on this data.

[0004] A single NER named entity recognition desensitization solution relies on training an entity recognition model using a general entity annotation dataset. However, the general model has poor adaptability to niche privacy fields in various industries. Customized business secrets, internal numbers, and project confidentiality fields for government and enterprises lack entity labels, resulting in a high rate of implicit privacy loss. Furthermore, entity replacement uses random filling, which does not match the semantic roles of the original text and disrupts the distribution of the corpus.

[0005] Full encryption / differential privacy desensitization solution: Differential privacy adds noise that distorts the semantic features of the original text. After encryption, the data cannot be directly used for fine-tuning of large models and requires additional decryption and conversion, which increases the training cost of large models. Moreover, the encrypted data destroys the original text features and seriously affects the semantic rules learned by LLM. Summary of the Invention

[0006] In view of this, the purpose of this application is to provide a method and system for de-identifying privacy data in the field of large language models, so as to solve the problems in the background art.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] This application presents a privacy-preserving data anonymization method for the field of large language models, comprising the following steps:

[0009] S1, Obtain the original mixed corpus;

[0010] S2, extract structured privacy information from the original mixed corpus based on a pre-built structured privacy rule base; and extract unstructured privacy information and cross-contextual related privacy information from the original mixed corpus based on a pre-fine-tuned lightweight large language model;

[0011] S3, extract the structural features of the structured privacy information, and generate virtual replacement data for the structured privacy information based on the structural features; and perform virtual data replacement on the structured privacy information of the original mixed corpus based on the virtual replacement data;

[0012] S4, identify the semantic role attributes in the unstructured privacy information, and replace the unstructured privacy information in the original mixed corpus with placeholders based on the semantic role attributes;

[0013] S5, perform aggregation and linkage desensitization on the cross-context related privacy information in the original mixed corpus, wherein the semantic cohesion of the information after aggregation and linkage desensitization pointing to the original event is less than or equal to a preset risk threshold.

[0014] S6, the original mixed corpus after the virtual data replacement, the placeholder replacement, and the aggregation linkage desensitization is used as the preliminary desensitized corpus. Based on the constructed privacy discrimination model, the privacy residue of the preliminary desensitized corpus is judged. If the preliminary desensitized corpus has privacy residue, return to step S2 until the preliminary desensitized corpus has no privacy residue. When the preliminary desensitized corpus has no privacy residue, the desensitized corpus is obtained. The desensitized corpus is used to train the target large language model.

[0015] This application also provides a privacy-preserving data anonymization system for the field of large language models, including:

[0016] The acquisition module is used to acquire the original mixed corpus;

[0017] The privacy information extraction module is used to extract structured privacy information from the original mixed corpus based on a pre-built structured privacy rule base; and to extract unstructured privacy information and cross-context related privacy information from the original mixed corpus based on a pre-fine-tuned lightweight large language model.

[0018] A structured privacy replacement module is used to extract the structural features of the structured privacy information, generate virtual replacement data for the structured privacy information based on the structural features, and perform virtual data replacement on the structured privacy information of the original mixed corpus based on the virtual replacement data.

[0019] An unstructured privacy replacement module is used to identify semantic role attributes in the unstructured privacy information and to perform placeholder replacement on the unstructured privacy information of the original mixed corpus based on the semantic role attributes.

[0020] The cross-context-related privacy replacement module is used to aggregate and desensitize cross-context-related privacy information in the original mixed corpus, wherein the semantic cohesion of the aggregated and desensitized information pointing to the original event is less than or equal to a preset risk threshold.

[0021] The verification and loop module is used to take the original mixed corpus after the virtual data replacement, the placeholder replacement and the aggregation linkage desensitization as the initial desensitized corpus, and to judge the privacy residue of the initial desensitized corpus based on the constructed privacy discrimination model. If the initial desensitized corpus has privacy residue, it returns to the privacy information extraction module until the initial desensitized corpus has no privacy residue. When the initial desensitized corpus has no privacy residue, the desensitized corpus is obtained, wherein the desensitized corpus is used to train the target large language model.

[0022] The beneficial effects of this application are as follows: The privacy-sensitive data desensitization method and system for the field of large language models, at the privacy identification level, employs a lightweight large language model for contextual semantic reasoning, overcoming the limitation of regular expressions and named entity recognition that can only match explicit format privacy, accurately uncovering implicit relational privacy, and fundamentally solving the problem of missed detection of non-standard privacy. At the desensitization and replacement level, structured data is generated using identically distributed simulation to maintain statistical characteristics; unstructured text is replaced based on semantic roles to preserve the complete syntactic structure; and cross-paragraph relational privacy is desensitized through aggregation and linkage to cut off the combination and restoration path, ensuring that the desensitized corpus maintains the integrity of semantic logic while removing sensitive information. Furthermore, a dedicated privacy discrimination model is introduced to inspect the desensitized data sentence by sentence, enabling automatic reprocessing of unqualified data and forming a closed-loop verification mechanism. This achieves a two-way balance between desensitization security and model accuracy, providing a high-quality training data foundation that is both compliant and usable for the deployment of large models in vertical fields. Attached Figure Description

[0023] The present application will be further described below with reference to the accompanying drawings and embodiments:

[0024] Figure 1 This is a flowchart illustrating the complete process of a privacy data desensitization method for the large language model domain in one embodiment of this application.

[0025] Figure 2 This is a flowchart illustrating a privacy data desensitization method for the field of large language models, as shown in one embodiment of this application;

[0026] Figure 3This is a schematic diagram of a structured privacy information extraction process in one embodiment of this application;

[0027] Figure 4 This is a schematic diagram illustrating the process of extracting unstructured privacy information and cross-context-related privacy information in one embodiment of this application;

[0028] Figure 5 This is a logical structure diagram of a privacy data desensitization system for the field of large language models, as shown in one embodiment of this application. Detailed Implementation

[0029] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0030] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the layers related to this application and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0031] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of this application; however, it will be apparent to those skilled in the art that embodiments of this application may be practiced without these specific details.

[0032] Figure 1 This is a flowchart illustrating the complete process of a privacy data anonymization method for large language models in one embodiment of this application. Figure 1 As shown, the overall logical flow in this application includes:

[0033] (1) Input of raw mixed corpus;

[0034] (2) Layered privacy identification module (rule base accurately identifies explicit privacy + LLM semantic association mining of implicit privacy);

[0035] (3) Privacy label classification and categorization;

[0036] (4) Adaptive type-based desensitization and replacement module (structured homogeneous virtual data replacement / unstructured semantic placeholder replacement / associative privacy aggregation desensitization);

[0037] (5) De-identified data caching;

[0038] (6) Privacy Residue Detection and Verification Module (Dedicated Privacy Judgment Model for Secondary Inspection);

[0039] (7) Non-compliant data is returned to the privacy identification stage for reprocessing;

[0040] (8) Qualified desensitized data;

[0041] (9) Semantic alignment fine-tuning module;

[0042] (10) Adapt to large model training dataset output.

[0043] Based on the above logical flow, this application includes the following five core functional units: a two-layer hierarchical privacy identification unit, a privacy hierarchical classification unit, a multi-strategy adaptive desensitization and replacement unit, a closed-loop privacy verification unit, and an LLM semantic alignment and adaptation unit.

[0044] Figure 2 This is a flowchart illustrating a privacy data desensitization method for the field of large language models, as shown in one embodiment of this application. Figure 2 As shown, the privacy data anonymization method for the large language model domain in this application mainly includes the following steps:

[0045] S1, Obtain the original mixed corpus;

[0046] The original mixed corpus usually contains data in three forms simultaneously:

[0047] (1) Structured data, including customer number, ID card number, mobile phone number, amount, etc.;

[0048] (2) Unstructured data, including contracts, report texts, etc.;

[0049] (3) Semi-structured / multi-dimensional distributed information, such as emails, meeting minutes, work order records, etc., which are distributed in multiple paragraphs.

[0050] S2, extract structured privacy information from the original mixed corpus based on a pre-built structured privacy rule base; and extract unstructured privacy information and cross-contextual related privacy information from the original mixed corpus based on a pre-fine-tuned lightweight large language model;

[0051] Step S2 is the content of the two-layer hierarchical privacy identification unit in this application. This application extracts structured privacy information, unstructured privacy information, and cross-contextual related privacy information from the original mixed corpus through a pre-built structured privacy rule base and a lightweight large language model. The following is an explanation of structured privacy information, unstructured privacy information, and cross-contextual related privacy information:

[0052] (1) Structured privacy information; privacy data with a strict fixed format, usually existing independently as fields or key-value pairs, and that can be directly matched by regular expressions. For example:

[0053] ID number (110101199003071234);

[0054] Mobile phone number (13912345678);

[0055] Bank card number (6222000000000000);

[0056] Unified Social Credit Code for Enterprises (91440101MA5XXXXXX);

[0057] Fixed format internal project number (e.g., XM-ZF-2026-0012);

[0058] (2) Unstructured privacy information; privacy entities embedded within natural language text, without a fixed format, and that can be identified and located using only the internal context of a single sentence or paragraph. For example:

[0059] The sentence "Zhang San paid 5 million yuan to XX Technology today" contains the words "Zhang San", "XX Technology", and "5 million yuan".

[0060] (3) Cross-contextual association of privacy information; composed of multiple fragmented descriptions scattered across different paragraphs, chapters, or even different documents. Individually, none of these fragments constitute privacy or pose a very low risk. However, when these fragments are pieced together, a complete and concrete portrait of a sensitive event or individual can be reconstructed. For example:

[0061] Paragraph A (Page 1): "The core algorithm module is codenamed XXX."

[0062] Paragraph B (page 4): "The total consideration for this merger and acquisition transaction is proposed to be RMB 120 million."

[0063] Paragraph C (page 7): "The seller's final authorized signatory is Li Si."

[0064] Combined Risks: When these three fragments are read separately, one only knows a code name, a sum of money, and a person's name; but when they are put together, they form a complete top-secret profile of a business merger and acquisition: "A certain technology company sells XXX technology for 120 million, and the signatory is Li Si."

[0065] After understanding the definitions of the three types of privacy information mentioned above, the following section will introduce the specific methods for extracting these three types of privacy information.

[0066] S2-1, Extraction of structured privacy information;

[0067] Figure 3 This is a schematic diagram of a structured privacy information extraction process in one embodiment of this application, such as... Figure 3 As shown, the structured privacy feature extraction process in this application includes:

[0068] S2-1-1 constructs a structured privacy rule base, creating a dynamically iterative, layered privacy rule base divided into general privacy rules (such as ID card, bank card, mobile phone number, etc.) and industry-customized rules (such as internal enterprise project codes, confidential contract numbers, government and enterprise exclusive file codes, etc.). It supports users adding and modifying industry-specific private rules online, specifically including:

[0069] S2-1-1-1, Construct a general privacy rule layer and an industry-specific custom rule layer, wherein the industry-specific custom rule layer is used to extend industry-specific privacy rules;

[0070] The general layer defines rules for common privacy information such as ID cards, mobile phone numbers, and bank card numbers. The industry-specific rule layer defines rules for specific privacy information between industries, such as financial SWIF codes and medical record numbers.

[0071] S2-1-1-2, Configure the matching mode and verification function for each privacy type in the general privacy rule layer and the industry-customized rule layer, wherein the matching mode includes the morphological features of the candidate field using regular expressions, the morphological features include length, character type and delimiter, and the verification function is used to verify the privacy information content;

[0072] The matching pattern for each privacy type defines the format of structured privacy information, including length, character type, and delimiter. For example, the regularization expression for a bank card number is:

[0073]

[0074] Where \b represents a word boundary anchor, \d{4}[-]? represents 4 digits plus an optional separator, {3} represents 3 consecutive occurrences, and \d{4,7} represents 4 to 7 digits, ending with a number.

[0075] The above regular expression includes length (16-19 characters), character type (pure numbers), and separator ("-" or ""). By matching the pattern, information that meets the format requirements can be quickly matched.

[0076] Then, validation functions are set for privacy types. For example, the FSM (FiniteStateMachine) validator validates the internal logic of the information to filter out fields that are mathematically consistent.

[0077] S2-1-1-3, Construct a deterministic finite automaton for the matching patterns of each privacy type in the general privacy rule layer and the industry-customized rule layer;

[0078] S2-1-1-4, Construct a deterministic finite state machine based on the matching patterns of all privacy types, and construct a structured privacy rule library based on the verification function of each privacy type in the deterministic finite state machine, the general privacy rule layer, and the industry-customized rule layer.

[0079] Finally, to address the matching efficiency issue with massive datasets, all regular expressions are constructed into a huge Dependent Automaton (DFA). When text characters flow through the DFA, only one left-to-right traversal is needed (time complexity O(n)) to simultaneously match all rules, thus improving matching efficiency.

[0080] S2-1-2 utilizes regularization and a finite state machine (FSM) to match structured explicit privacy, marking privacy locations, privacy types, and data structure features to generate a first-level privacy label, specifically including:

[0081] S2-1-2-1, The original mixed corpus is input into the deterministic finite state machine for automatic scanning to obtain candidate privacy fields and the privacy types of the candidate privacy fields;

[0082] Deterministic finite automata (DFA) enable multi-pattern parallel matching by compiling all regular expressions in the general rule layer and the industry-defined rule layer into a single state transition table.

[0083] The character stream of the input raw mixed corpus is sequentially fed into the DFA. The DFA queries the state transition table based on the current state and the input characters to jump to the next state. When the character sequence ends or reaches a word boundary, if the DFA is currently in an accepting state, it is determined that the character sequence completely matches the regular expression pattern corresponding to that accepting state, and the system records the character sequence as a candidate privacy field.

[0084] S2-1-2-2, The candidate privacy fields are validated based on the validation function of the privacy type, and the candidate privacy fields that pass the validation are used as structured privacy fields;

[0085] Invoke the validation function pre-bound to this type. This validation function uses a deterministic algorithm to mathematically verify the internal logical consistency of the candidate fields.

[0086] For example, when the privacy type is a bank card number, the Luhn algorithm is used; when the privacy type is a mobile phone number, a number range + length verification is used. Specific verification functions can directly reference existing verification methods, which will not be elaborated upon here.

[0087] S2-1-2-3, extract the location, privacy type and structural features of the structured privacy field to obtain structured privacy information, wherein the structural features include field length, character type distribution, numerical range, encoding format and fixed prefix / suffix identifier;

[0088] For structured privacy fields that pass the verification, the system performs metadata extraction.

[0089] This operation involves three tasks:

[0090] First, locate and record the starting and ending character indices of the field in the original corpus to form position coordinates;

[0091] Second, record the privacy type output in S2-1-2-1;

[0092] Third, analyze the field's morphological structure to extract its length (total number of characters), character type distribution (the proportion or absolute number of numbers, uppercase letters, lowercase letters, and separators), numerical range (if the field contains numerical substrings, record their minimum and maximum value ranges), encoding format (the position of separators between segments and the number of segments), and fixed prefix or suffix identifiers (if a constant character sequence exists at the beginning or end of the field, record it). The above information is combined to form the complete structured privacy information for this field.

[0093] Finally, a primary privacy label is generated for the structured privacy information. This label is then passed to the privacy classification unit for subsequent decision-making.

[0094] S2-2, Extraction of unstructured privacy information and cross-contextualized privacy information;

[0095] Figure 4 This is a schematic diagram illustrating the extraction process of unstructured privacy information and cross-context related privacy information in one embodiment of this application. The extraction process is as follows: Figure 4 As shown, it includes:

[0096] S2-2-1 fine-tunes a lightweight model with a large number of parameters, abandoning the general NLP entity extraction model and adopting a lightweight LLM pre-trained on a small amount of industry privacy corpus.

[0097] S2-2-1-1, Obtain original business corpus samples and construct training labels for the original business corpus samples, wherein the training labels include inference basis samples, entity annotations of privacy fields, related entity groups across sentences or paragraphs, and the association level of related entity groups;

[0098] The tags contain three types of information: First, the inference basis sample, which is a text describing in natural language why the context is determined to contain privacy and the logic of the relationship between fragments; this tag is used to teach lightweight large language models to learn privacy information. For example: the original text "The total amount of the XXX project is 120 million, approved by Li Si.", the tag contains the inference basis "XXX and 120 million belong to the same merger and acquisition event", the entity list contains "XXX" (project), "120 million" (amount), "Li Si" (person's name), and the association group contains the first two and the association level is 0.95.

[0099] Second, entity annotation of privacy fields, that is, the start and end positions and type of each privacy entity in the original text;

[0100] Third, the system groups entities across sentences or paragraphs and assigns them association levels. This involves grouping multiple entities located in different physical locations into the same group and assigning each group a value between 0 and 1, representing the risk intensity of reconstructing a specific sensitive event after combining the entities within the group. This labeling structure allows the model to learn both entity recognition and association reasoning tasks simultaneously.

[0101] S2-2-1-2, Input the original business corpus sample into the lightweight large language model to obtain the generated text, wherein the generated text includes the reasoning basis and the privacy entity list;

[0102] The token sequence of the original business corpus samples is input into a lightweight large language model under the current parameter state. The model performs multi-layer Transformer decoding and generates the target sequence through autoregression on a token-by-token basis. This generated sequence is structured text, containing a natural language description of the reasoning basis and a JSON structure of a list of privacy entities.

[0103] At this stage, the model does not output association grouping information or association level; it only outputs independent entities and their types.

[0104] S2-2-1-3, Calculate the loss between the training labels and the generated text based on a pre-constructed loss function, wherein the mathematical expression for the loss is:

[0105]

[0106]

[0107]

[0108] In the formula, For the loss, The cross-entropy loss is used to measure the difference between the generated text and the training labels. To compare learning loss, The token sequence of the original business corpus sample. For the token location index, The set of token indices used as the basis for reasoning. A set of token indexes for privacy entities. For the first Tokens for each location, For historical token sequences, This is the entity loss weighting coefficient. To train the indexes of two different privacy entities in the labels, These are the entity representation vectors of two different privacy-preserving entities output by the lightweight large language model. This represents an index representing two distinct privacy entities belonging to the same associated entity group. For the intra-group association level of two different privacy entities, This represents an index representing two distinct privacy entities that do not belong to the same associated entity group. Represents the similarity function;

[0109] The text sequence generated by the model in S2-2-1-2 is compared token-by-token with the training label text in S2-2-1-1. Simultaneously, the vectors corresponding to the tokens of entities in the hidden layers of the model are extracted for spatial distance measurement. The weighted sum of the two errors yields the total loss, and this loss scalar is used to calculate the gradients of each trainable parameter through backpropagation.

[0110] In the loss function, the total loss It includes overall text loss and contrastive learning loss. Overall text loss The text sequence generated by the model is divided into two functional regions. The reasoning logic text generated by the cross-entropy constraint model for the corresponding reasoning basis region is consistent with the manually annotated reasoning basis in terms of word order and semantics. For the corresponding privacy entity region, the entity name, type, and JSON format generated by its cross-entropy constraint model perfectly match the annotation. Weight coefficients This value is used to adjust the relative importance of entity recognition accuracy; a value greater than 1 enhances the model's sensitivity to entity boundaries and type discrimination. The cross-entropy is calculated separately for each region and then summed to ensure that the model generates fluent inference text without sacrificing entity extraction accuracy.

[0111] Comparative learning loss It is divided into positive and negative pairs. Positive pairs consist of two entities in the training labels that belong to the same related entity group. The cosine similarity of their representation vectors is calculated, and the similarity is determined by the group's association level. The weights are accumulated. The optimization objective is to maximize this accumulated value, that is, to force entities within the same group to be close to each other in the latent space. It acts as a soft boundary modulator: groups with high association levels (close to 1) receive a greater degree of convergence, while groups with low association levels (close to 0.3) only receive a slight convergence. Negative pairs are two entities belonging to different associated entity groups in the training labels; the cosine similarity of their representation vectors is calculated and the negative values ​​are summed. The optimization objective is to minimize this value, i.e., to force entities from different groups to move further apart in the latent space.

[0112] This loss transforms the grouping information in the training labels into geometric constraints in a high-dimensional space, ensuring that entity representation vectors within the same association group form compact clusters, while entity representation vectors from different association groups maintain clear boundaries. During the inference phase, the system can perform unsupervised dynamic clustering based on the geometric distance between vectors, eliminating the need for the model to explicitly generate grouping labels in the output, thus decoupling the training supervision signal from the unsupervised clustering during the inference phase.

[0113] S2-2-1-4, Freeze the parameters of the lightweight large language model, and fine-tune the adapter layer of the lightweight large language model based on the loss.

[0114] The entire original pre-trained weights of the lightweight large language model are frozen, and only the gradients calculated in S2-2-1-3 are applied to the parameters of the low-rank adapter modules inserted in the side paths of each Transformer sublayer (Q and V projection matrices in multi-head attention and up-projection matrices in the feedforward network). This operation preserves the model's general semantic understanding capabilities while achieving industry-specific privacy-related identification capabilities with a minimal number of parameter updates (less than 1% of the total model).

[0115] S2-2-2 extracts unstructured and cross-contextually related privacy information, using paragraphs and multi-sentence contexts as the identification granularity, and mining implicit privacy without a fixed format based on semantic association logic: including fragmented descriptions of natural persons, project amounts written in parts, and confidential data that can only be reconstructed by piecing together cross sentences, specifically including:

[0116] S2-2-2-1, The original mixed corpus is input into a pre-fine-tuned lightweight large language model to obtain a privacy entity list, and the entity representation vector of each privacy entity in the privacy entity list is extracted from the pre-fine-tuned lightweight large language model.

[0117] S2-2-2-2, cluster the entity representation vectors of multiple privacy entities to obtain multiple candidate clusters;

[0118] Using all extracted entity representation vectors as input, density-based clustering or hierarchical clustering algorithms are employed, with cosine distance as the metric, to divide the vectors in the semantic space into several candidate clusters. During clustering, by setting the neighborhood radius parameter and the minimum number of samples parameter, spatial distribution structures with different densities can be automatically identified.

[0119] Specifically, this application employs HDBSCAN clustering. Compared to traditional DBSCAN, HDBSCAN does not require pre-setting the global neighborhood radius ε, and can automatically adapt to regions of different densities, avoiding under-segmentation of high-density regions or over-segmentation of low-density regions due to a fixed threshold.

[0120] The method for determining key hyperparameters is as follows:

[0121] Minimum cluster size (min_cluster_size): Based on the definition of cross-context associated privacy (at least two fragment combinations), min_cluster_size is set to 3. This value includes at least two associated entities plus a noise tolerance margin, ensuring that the clustering results retain only entity groups with substantial combination risks.

[0122] Minimum number of samples (min_samples): To enhance the robustness of the algorithm to outliers, min_samples = 2 is set, meaning that each core point contains at least 2 other samples in its neighborhood.

[0123] Cluster selection method: The 'eom' (Excess of Mass) strategy is adopted to prioritize the extraction of cluster structures with significant stability and avoid excessive fragmentation.

[0124] S2-2-2-3, calculate the cohesion of multiple entity representation vectors within each candidate cluster, and count the number of entity representation vectors within each candidate cluster;

[0125] Cohesion The calculation uses a normalized Gaussian potential function:

[0126]

[0127] In the formula, The number of entities within the cluster. and For indexes of two different entities within a cluster, For kernel width coefficient, and Two different entities within the cluster.

[0128] This formula calculates the Gaussian kernel similarity of all pairwise vector pairs within a cluster and takes the average. The output value range is (0,1], with larger values ​​indicating closer clustering of entities. Simultaneously, it counts the number of entities within each candidate cluster. This is used for subsequent threshold determination.

[0129] S2-2-2-4 identifies candidate clusters whose number of entity representation vectors is greater than or equal to a preset number threshold and whose cohesion is greater than or equal to a preset cohesion threshold as target clusters, and identifies the privacy entities corresponding to multiple entity representation vectors within the target cluster as cross-context-related privacy information; and identifies the privacy entities corresponding to entity representation vectors outside the target cluster as unstructured privacy information.

[0130] In this application, a dual-threshold design ensures that cross-contextual desensitization is triggered only when multiple entities form a sufficiently dense and numerous cluster in the semantic space. If the above dual judgment conditions are not met, it is classified as unstructured privacy information and pushed to the semantic role placeholder replacement branch.

[0131] S2-3, Generate multi-level privacy labels based on privacy type and contextual risk level, and map them to differentiated de-identification strategies:

[0132] S2-3-1 establishes a four-level privacy risk rating system. Based on the privacy type and cross-contextuality strength output in stages S2-1 and S2-2, the system assigns a risk level label L1 to L4 to each privacy record. The level definitions are as follows:

[0133] Level L1 (Extremely High Risk): Contains strong identifiers that can directly and uniquely identify natural or legal entities, including ID card numbers, bank card numbers, passport numbers, and corporate unified social credit codes. A breach of this level of privacy constitutes a major security incident, necessitating lossless virtual data replacement (routing to S3).

[0134] Level L2 (High Risk): A set of associated privacy entities composed of ≥3 entities from multiple locations across different dimensions, with a cross-contextual cohesion Ψ(C) ≥ 0.75. This level of combination can reconstruct a complete profile of sensitive events, and requires aggregated and linked desensitization (routing to S5).

[0135] Level L3 (Medium Risk): Isolated, unstructured privacy entities embedded in natural language, including names of people, organizations, specific amounts, and project codes, and not belonging to any highly cohesive association groups. This level requires semantic role placeholder replacement (routing to S4).

[0136] Level L4 (Low Risk): Contains only vague time descriptions, generic terms (such as "a certain customer"), or invalid placeholder contexts remaining after desensitization verification. No additional desensitization processing is required, and it can be directly allowed to proceed to S6.

[0137] S2-3-2, Construct a hierarchical decision routing table. Based on the mapping logic in the table below, the system automatically directs privacy data carrying different levels of labels to the corresponding de-identification sub-modules. The decisions are shown in Table 1:

[0138] Table 1. Examples of Privacy Decisions

[0139]

[0140] S2-3-3, Tag Persistence and Transmission. The system stores level tags in a metadata structure, for example:

[0141] {privacy_level:Lx,original_span:[start,end],group_id:(optional),source_type:'structured'|'unstructured'|'cross_context'}

[0142] Among them, privacy_level:Lx represents the privacy risk level;

[0143] original_span:[start,end] represents the coordinates of the original text position;

[0144] group_id: (optional) is the associated group ID;

[0145] `source_type:'structured'|'unstructured'|'cross_context'` specifies the data format source.

[0146] S3, extract the structural features of the structured privacy information, and generate virtual replacement data for the structured privacy information based on the structural features; and perform virtual data replacement on the structured privacy information of the original mixed corpus based on the virtual replacement data;

[0147] The process of replacing structured privacy data and de-identifying it includes:

[0148] S3-1, parse the format template of the structured privacy information, and divide the format template into static segments and dynamic segments according to the structural features. The static segment includes fixed delimiters, fixed unit codes, and fixed prefix / suffix identifiers. The dynamic segment includes numerical sequences, letter sequences, mixed sequences, and variable-encoded suffixes.

[0149] Structured privacy information has a fixed format template, which consists of alternating constant character sequences and variable character sequences.

[0150] Based on the structural features (field length, character type distribution, fixed prefix / suffix identifier, and delimiter position) extracted from S2-1-2-3, the system marks each character position in the format template as a static or dynamic segment.

[0151] Static segments are character sequences that remain unchanged across all samples, including fixed prefixes (such as “CUST-”), fixed unit codes (such as “CN”), and fixed separators (such as “-” or “ / ”). They are preserved as is during the virtual data generation process and do not participate in sampling replacement.

[0152] The dynamic segment is a character sequence that varies across different samples, including variable years, serial numbers, and letter combinations. It is resampled and generated based on a statistical distribution model. This partitioning operation decouples the generation task from the direct copying of the fixed part to the distributed sampling of the variable part, simplifying the subsequent generation process.

[0153] For example, the format template of the structured privacy information "CUST-2026-0087" is parsed as follows: static segment "CUST-" and two separators "-"; dynamic segment "2026" (year numerical sequence) and "0087" (four-digit serial number).

[0154] S3-2, For the dynamic segment, based on the distribution of its character type and the statistical distribution characteristics of its numerical interval, random sampling is performed using the Monte Carlo inverse transform method to generate a preliminary replacement sequence;

[0155] The system performs sampling and generation independently for each dynamic segment. First, based on the statistical distribution of the dynamic segment in historical data (for numerical dynamic segments, this includes the mean, variance, minimum, and maximum values; for character dynamic segments, it includes the probability of occurrence of each character type), an empirical cumulative distribution function is constructed. Subsequently, the generated data follows a uniform distribution. random numbers ,calculate This involves drawing a sample value from an empirical distribution that conforms to the original statistical characteristics. For numerical sequences, the sampling results are rounded and truncated to ensure they fall within the valid interval.

[0156] For a letter sequence, each character is sampled independently according to its type distribution probability. The mathematical basis of this method is the inverse transform sampling theorem, which ensures that the statistical distribution of the generated sequence is asymptotically consistent with the original data.

[0157] S3-3, The preliminary replacement sequence is fed into a finite state machine verifier bound to the current privacy type to perform internal logic self-consistency verification;

[0158] The system calls the pre-bound verification function in S2-1-1-2 based on the type of the current structured privacy information (such as ID card number, bank card number, internal project number). This verification function is implemented in the form of a deterministic finite state machine (DFSM) or a deterministic algorithm, taking the initial replacement sequence generated in S3-2 as input and performing state transition calculations bit by bit.

[0159] S3-4, If the internal logic self-consistency verification fails, return to the method of using the Monte Carlo inverse transform to perform random sampling and generate an initial replacement sequence until the internal logic self-consistency verification is passed or the maximum number of loops is reached; if the internal logic self-consistency verification is passed, the initial replacement sequence is used as the final replacement sequence;

[0160] If S3-3 returns "Reject", the process returns to S3-2 to re-execute Monte Carlo sampling, generating a new preliminary replacement sequence, and then performs S3-3 verification again. This process forms a sampling-verification loop until verification passes or the preset maximum number of loops is reached (usually set to 100). The maximum number of loops is set to prevent system deadlock due to a very low probability of continuous sampling failures. If verification passes within the maximum number of loops, the current preliminary replacement sequence is marked as the final replacement sequence; if it still fails after reaching the maximum number of loops, the system executes a safety net strategy: it generates a purely random value based on the character type distribution of the dynamic segment and abandons the FSM verification constraint, or selects a default valid value from a preset static safe replacement table.

[0161] S3-5, the static segment and the final replacement sequence are concatenated according to the format template to obtain the virtual replacement data of the structured privacy information.

[0162] The system concatenates all static segment character sequences with the verified final replacement sequence according to their position order in the original format, following the original format template defined in S3-1. The concatenation process does not add extra characters or delete any static or dynamic segments, ensuring that the length of the output fields, character type distribution, delimiter positions, and fixed prefixes / suffixes are completely consistent with the original format template.

[0163] S4, identify the semantic role attributes in the unstructured privacy information, and replace the unstructured privacy information in the original mixed corpus with placeholders based on the semantic role attributes;

[0164] The process of replacing unstructured privacy information with placeholders includes:

[0165] S4-1, Obtain unstructured privacy entities and their semantic role labels;

[0166] For each entity, extract its position coordinates in the original text and the semantic role label synchronously output by the lightweight large language model in S2-2-1-2 during the generation phase. The semantic role labels are obtained by classifying the syntactic functions assumed by the entity in the sentence, including categories such as subject (SUBJ), object (OBJ), indirect object / institutional attribute (IOBJ), amount attribute (AMOUNT), project attribute (PROJ), and time attribute (TIME).

[0167] S4-2: Query the semantic role-placeholder mapping table to determine the normalized placeholder that each entity should be replaced with. The semantic role-placeholder mapping table is shown in the following table:

[0168] Table 2. Example of semantic role-placeholder mapping table

[0169]

[0170] S4-3: Perform granularity-aware replacement operations, and the system executes differentiated replacement strategies according to the type of the entity. For entities of person name, institution name, and project code types, the original text segment is directly and completely replaced with the placeholder determined in S4-2.

[0171] For entities of amount and quantity types, the measurement unit words in the original text need to be retained to maintain grammatical coherence, but the specific values are covered by placeholders; that is, the specific numbers or numerical expressions are deleted, and the unit words (such as "ten thousand yuan", "one hundred million yuan", "yuan") are retained and placed after the placeholders, or "five million yuan" is entirely replaced with [Project Approved Funding] according to preset rules, while the amount semantics is implicitly retained through the context. During the replacement process, the system records the mapping relationship of the replacement operation, that is, the corresponding relationship between each replaced entity and the corresponding placeholder.

[0172] For example: the original sentence "Zhang San paid five million yuan to XX Technology." after replacement becomes "[Business User] paid [Project Approved Funding] to [Partner Institution]." The subject-predicate-object structure of the sentence, the preposition "to", the auxiliary word "le" and the punctuation marks are all retained as they are, and only the entity stems are covered by the placeholders.

[0173] S5: Perform aggregated linkage desensitization on cross-context associated private information in the original mixed corpus, wherein the semantic cohesion degree of the information after aggregated linkage desensitization pointing to the original event is less than or equal to a preset risk threshold;

[0174] The process of performing aggregated linkage desensitization on cross-context associated private information includes:

[0175] S5-1: Extract the semantic role labels of multiple private entities in the cross-context associated private information;

[0176] The system retrieves multiple privacy entities belonging to the same association group from the target cluster determined in S2-2-2-4. For each entity, it backtracks to the semantic role labels synchronously output by the lightweight large language model in stage S2-2-1-2 during generation, including categories such as subject, object, institution modifier, monetary modifier, and time modifier. This semantic role information does not change due to the clustering result and is an independent attribute of the entity's syntactic function in its original sentence. The system aggregates these role labels according to the association group to which the entity belongs, providing a basis for subsequently assigning different placeholders.

[0177] S5-2, Select placeholders that match the semantic role label from the pre-constructed orthogonal placeholder pool, wherein the cosine similarity between any two placeholders in the orthogonal placeholder pool in the high-dimensional semantic space is less than a preset similarity threshold.

[0178] The system calls a pre-built pool of orthogonal placeholders, which is a collection of placeholders and is constructed as follows:

[0179] In the pre-trained word vector space, pairwise cosine similarity is calculated for all candidate placeholder words, and a subset of placeholder words whose similarity to any two words is lower than a preset similarity threshold (usually set to 0.2) is selected and included in the pool.

[0180] The mathematical property of this pool is that any two placeholder words within the pool are approximately orthogonal in direction in the high-dimensional semantic space, and the absolute value of their word vector inner product is extremely small. The system traverses each entity in the current association group, queries the mapping table based on its semantic role label, and initially filters out a set of candidate placeholder words that match the role.

[0181] Subsequently, a group compatibility check is performed on the placeholders in the set: the cosine similarity between the currently assigned placeholders and the candidate placeholders is calculated one by one, and only the candidates whose similarity with all the assigned placeholders is lower than the pool construction threshold are selected as the final assignment results.

[0182] This allocation strategy ensures that entities within the same association group are replaced with placeholders that are independent of each other in the semantic space, cutting off the possibility of the combination pointing to the same event at the vector level.

[0183] S5-3, Replace the privacy entity based on the placeholder to obtain preliminary replaced privacy information;

[0184] S5-4, calculate the semantic cohesion of the text that initially replaces the privacy information. When the semantic cohesion is greater than or equal to a preset risk threshold, return to the process of selecting a placeholder word that matches the semantic role label from a pre-constructed orthogonal placeholder pool until the semantic cohesion is less than the preset risk threshold. When the semantic cohesion is less than the preset risk threshold, use the initial replacement privacy information as the final replacement privacy information.

[0185] The system extracts the representation vectors corresponding to each placeholder in the initial replacement text generated by S5-3 (the placeholder vectors can be derived from pre-trained word embeddings or the hidden layer representation of lightweight LLM), and constructs a temporary vector set. It then calculates the semantic cohesion of the temporary vector set (the calculation formula is described above).

[0186] If the semantic cohesion is greater than or equal to the preset risk threshold, it means that the replaced placeholder still has event-directedness that can be identified by the clustering algorithm. It is determined that the association has not been completely severed. The system returns to S5-2, reassigns a new set of mutually orthogonal placeholders to the association group from the orthogonal placeholder pool, and repeats S5-3 and S5-4.

[0187] If the semantic cohesion is less than the preset risk threshold, it means that the replaced placeholders are evenly distributed in the semantic space and cannot be classified into the same event cluster by the clustering algorithm. The association has been severed, and the system marks the current preliminary replacement privacy information as the final replacement privacy information and submits it to the subsequent process.

[0188] The iterative mechanism in S5 ensures that the effectiveness of aggregated desensitization does not depend on the randomness of a single allocation, but provides deterministic guarantees through closed-loop verification.

[0189] S6, the original mixed corpus after the virtual data replacement, the placeholder replacement, and the aggregation linkage desensitization is used as the preliminary desensitized corpus. Based on the constructed privacy discrimination model, the privacy residue of the preliminary desensitized corpus is judged. If the preliminary desensitized corpus has privacy residue, return to step S2 until the preliminary desensitized corpus has no privacy residue. When the preliminary desensitized corpus has no privacy residue, the desensitized corpus is obtained. The desensitized corpus is used to train the target large language model.

[0190] In this application, a privacy discriminant model is used to perform privacy residual discrimination. The method for constructing the privacy discriminant model includes:

[0191] (1) Obtain training samples and privacy residual labels of training samples, wherein the training samples include positive samples containing privacy residuals and negative samples not containing privacy residuals;

[0192] The privacy discrimination model is a binary classification model, and its training data must cover both positive and negative samples. The way the positive and negative samples are constructed directly affects the model's discrimination ability in the actual desensitization pipeline.

[0193] Positive samples are text data containing privacy remnants, and their sources include two categories: the first category is text fragments containing explicit or implicit privacy extracted from the original un-de-identified corpus, labeled as 1; the second category is de-identified failed samples that have been processed by this de-identification system but intentionally retain some privacy (such as only replacing explicit privacy while retaining implicit associations), labeled as 1.

[0194] Negative samples are text data that do not contain any privacy residues. The sources include two categories: the first category is compliant desensitized text data that has been fully processed by this desensitization system and confirmed by manual review to be free of residues, with the label set to 0; the second category is general text in public corpora that does not contain any personal or commercially sensitive information (such as Wikipedia and public news), with the label set to 0.

[0195] The ratio of positive to negative samples should be controlled between 1:1 and 1:3 to prevent the model from making biased predictions due to class imbalance.

[0196] (2) Input the training samples into the preset base model to obtain the privacy residue judgment result;

[0197] In this application, a lightweight pre-trained language model (such as RoBERTa-base or BERT-base-Chinese, with approximately 110M to 125M parameters) is selected as the base model.

[0198] (3) Calculate the judgment loss of the privacy residual label and the privacy residual judgment result based on the preset binary cross-entropy loss function;

[0199] The mathematical expression for the binary classification cross-entropy loss function is:

[0200]

[0201] In the formula, For binary classification, cross-entropy loss, For batch size, For sample index, For the sample Privacy residue label, This represents the privacy residual probability predicted by the pedestal model.

[0202] (4) The base model is trained based on the judgment loss to obtain the privacy discrimination model.

[0203] During training, minimize To achieve this, the parameters of the base model are updated using gradient descent or its variants (such as the AdamW optimizer).

[0204] The privacy discrimination model obtained through training is used to judge the privacy residue of the initial desensitized data, and the decision is made based on the judgment result to enter the next round of loop or directly output the desensitized data.

[0205] After obtaining the anonymized corpus for training the target large language model, it cannot be directly used for fine-tuning / training. Before fine-tuning / training the target large language model, a lightweight semantic alignment subtask is set up: the original small batch of unanonymized business corpus and the large batch of anonymized corpus are used to construct training samples, allowing the target large language model to learn the mapping relationship between the original semantics and the anonymized placeholder semantics, adapt to the semantic expression habits of the anonymized placeholder words, and offset the feature bias caused by the anonymization replacement.

[0206] The process of semantic subtask alignment training includes:

[0207] (1) Extract aligned training sample datasets from the original mixed corpus and the desensitized corpus;

[0208] The core objective of the semantic alignment subtask is to enable the target large language model to map placeholders in the anonymized text to the semantic representation neighborhoods of the corresponding entities in the original text. To achieve this objective, paired training samples need to be constructed, with each pair containing the original and anonymized versions of the same semantic content.

[0209] (2) Input the original mixed corpus into the target large language model, and extract the first global representation vector from the penultimate hidden layer of the target large language model. The desensitized corpus is then input into the target large language model, and the second global representation vector is extracted from the penultimate hidden layer of the target large language model. ;

[0210] The system will process the original samples separately. and desensitized samples The input is a large language model. This model is a large vertical domain model (typically with 70B to 130B parameters) to be formally fine-tuned. The model performs a forward propagation for each input but does not update any parameters. The system sets a hook in the penultimate layer of the model (i.e., after the last Transformer decoder layer and before the output layer) to intercept the hidden state output of that layer.

[0211] For each input text, the hidden representation vectors of all tokens are averaged and pooled, or the vector corresponding to the [CLS] flag is taken to obtain a fixed-dimensional global representation vector. This global vector encodes the model's understanding of the overall semantics of the input text. The two-channel operation yields... and Both serve as inputs for subsequent loss calculations.

[0212] (3) Calculate the first global representation vector and the second global representation vector semantic alignment loss The semantic alignment loss The mathematical expression is:

[0213]

[0214] In the formula, This refers to the batch sample size. To align the training sample indices, Indicates the first A raw mixed corpus, Indicates the first A desensitized corpus, This represents the hidden layer representation extraction function. These are the KL divergence weighting coefficients. The input to the target large language model is The probability distribution of the output vocabulary at that time. The input to the target large language model is The probability distribution of the output vocabulary at that time. Indicates KL divergence;

[0215] In the loss function, the first term To distribute the error loss, this term directly constrains the global representation vectors of the original sample and the desensitized sample in the hidden layer of the model to be as close as possible. The second term... The KL divergence loss constrains the model's behavior in the output prediction distribution, making the model tend to produce similar next word prediction probabilities for original and de-identified texts with the same semantics.

[0216] (4) Based on the semantic alignment loss Semantic alignment training is performed on the target large language model.

[0217] The system minimizes semantic alignment loss. To achieve this, the gradient descent method is used to update the parameters of the target large language model.

[0218] After the target large language model is trained by subtask alignment, it is adapted to the desensitized corpus generated earlier. At this point, the desensitized corpus generated earlier can be used to train / fine-tune the target large language model after alignment training.

[0219] To prevent infinite loops, this application also provides the following reflow processing flow:

[0220] S6-1, Reflux particle size control and incremental reprocessing:

[0221] When the privacy discrimination model determines that there are privacy remnants in the initially de-identified corpus, the system first analyzes the attention heatmap output by the privacy discrimination model to locate the specific paragraph or sentence coordinates (accurate to the sentence boundary) where the privacy remnants are located. The reflow operation does not reprocess the entire corpus, but only packages the sentence fragment containing the remnants and the context windows of the two sentences before and after it (to ensure the integrity of cross-contextual association) into micro-batches, and sends them back to S2-1 (structured rule base scanning) and S2-2 (lightweight LLM semantic recognition) for incremental reprocessing.

[0222] It is important to note that when reflow reprocessing is triggered, the system appends the `skip_cache=True` flag to the micro-batch metadata sent back to S2, forcing the S2 stage to skip cache lookups and directly perform incremental identification. Simultaneously, the system proactively removes any existing historical cache entries for that segment in the S5-5 cache layer based on the `original_span` coordinates. This design ensures that incremental reprocessing after reflow can fully execute the dynamic annealing strategy defined in S6-2, preventing stale cache results from blocking the correct reprocessing flow.

[0223] S6-2, Dynamic Threshold Adjustment and Strategy Annealing:

[0224] To prevent the same residual fragment from repeatedly triggering reflow, the system uses an annealing mechanism to dynamically adjust the recognition sensitivity:

[0225] First reflow: Tighten the tolerance of the structured rule base's validation function (e.g., reduce the allowed number of retries for Luhn validation from 100 to 50), and lower the entity discrimination temperature parameter of the lightweight LLM from 0.7 to 0.3 to reduce generation randomness and improve recall.

[0226] Second reflow: If the first reflow still fails, the system further activates high-precision regular anchor points (such as forced matching of internationally accepted formats) and enables lightweight LLM beam search (beam size=5) to replace random sampling generation.

[0227] S6-3, Maximum number of backflows and forced degradation strategy:

[0228] The system is set to a maximum of 3 reflow attempts. If the privacy remnant verification still fails after 3 reflow attempts, the system will no longer loop indefinitely, but will instead execute a forced degradation strategy:

[0229] Mark the segment as "Pending Manual Review", remove it from the automatic training dataset, store it in the pending review queue, and trigger an alarm notification.

[0230] Simultaneously, a safe placeholder copy (the global default placeholder) that has undergone S3 virtual replacement is used to urgently overwrite the fragment, ensuring that the overall training dataset construction progress is not blocked. After manual review and approval, this fragment can be used to replace and update the dataset in the next version.

[0231] This mechanism strikes a balance between system robustness and security, completely eliminating the risk of infinite loops.

[0232] like Figure 5 As shown, this application also provides a privacy-preserving data anonymization system for the field of large language models, including:

[0233] The acquisition module is used to acquire the original mixed corpus;

[0234] The privacy information extraction module is used to extract structured privacy information from the original mixed corpus based on a pre-built structured privacy rule base; and to extract unstructured privacy information and cross-context related privacy information from the original mixed corpus based on a pre-fine-tuned lightweight large language model.

[0235] A structured privacy replacement module is used to extract the structural features of the structured privacy information, generate virtual replacement data for the structured privacy information based on the structural features, and perform virtual data replacement on the structured privacy information of the original mixed corpus based on the virtual replacement data.

[0236] An unstructured privacy replacement module is used to identify semantic role attributes in the unstructured privacy information and to perform placeholder replacement on the unstructured privacy information of the original mixed corpus based on the semantic role attributes.

[0237] The cross-context-related privacy replacement module is used to aggregate and desensitize cross-context-related privacy information in the original mixed corpus, wherein the semantic cohesion of the aggregated and desensitized information pointing to the original event is less than or equal to a preset risk threshold.

[0238] The verification and loop module is used to take the original mixed corpus after the virtual data replacement, the placeholder replacement and the aggregation linkage desensitization as the initial desensitized corpus, and to judge the privacy residue of the initial desensitized corpus based on the constructed privacy discrimination model. If the initial desensitized corpus has privacy residue, it returns to the privacy information extraction module until the initial desensitized corpus has no privacy residue. When the initial desensitized corpus has no privacy residue, the desensitized corpus is obtained, wherein the desensitized corpus is used to train the target large language model.

[0239] This application presents a privacy-preserving data desensitization method and system for the field of large language models. At the privacy identification level, it employs a lightweight large language model for contextual semantic reasoning, overcoming the limitation of regular expressions and named entity recognition, which can only match explicit format privacy. This accurately uncovers implicit relational privacy, fundamentally solving the problem of missed detection of non-standard privacy. At the desensitization and replacement level, structured data is generated using identical distribution simulation to maintain statistical characteristics. Unstructured text is replaced based on semantic roles to preserve the complete syntactic structure. Cross-paragraph relational privacy is addressed through aggregated and linked desensitization to cut off the combination and restoration path, ensuring that the desensitized corpus maintains the integrity of semantic logic while removing sensitive information. Furthermore, a dedicated privacy discrimination model is introduced to inspect the desensitized data sentence by sentence, enabling automatic reprocessing of unqualified data and forming a closed-loop verification mechanism. This achieves a two-way balance between desensitization security and model accuracy, providing a high-quality training data foundation that is both compliant and usable for the deployment of large models in vertical domains.

[0240] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0241] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory so that the terminal performs any of the methods in this embodiment.

[0242] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0243] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0244] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0245] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0246] In the above embodiments, although the present application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. The embodiments of the present application are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0247] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A privacy-preserving data anonymization method for the field of large language models, characterized in that, include: S1, Obtain the original mixed corpus; S2, extract structured privacy information from the original mixed corpus based on a pre-built structured privacy rule base; And based on a pre-tuned lightweight large language model, unstructured privacy information and cross-context-related privacy information are extracted from the original mixed corpus; S3, extract the structural features of the structured privacy information, and generate virtual replacement data for the structured privacy information based on the structural features; and perform virtual data replacement on the structured privacy information of the original mixed corpus based on the virtual replacement data; S4, identify the semantic role attributes in the unstructured privacy information, and replace the unstructured privacy information in the original mixed corpus with placeholders based on the semantic role attributes; S5, perform aggregation and linkage desensitization on the cross-context related privacy information in the original mixed corpus, wherein the semantic cohesion of the information after aggregation and linkage desensitization pointing to the original event is less than or equal to a preset risk threshold. S6, the original mixed corpus after the virtual data replacement, the placeholder replacement, and the aggregation linkage desensitization is used as the preliminary desensitized corpus. Based on the constructed privacy discrimination model, the privacy residue of the preliminary desensitized corpus is judged. If the preliminary desensitized corpus has privacy residue, return to step S2 until the preliminary desensitized corpus has no privacy residue. When the preliminary desensitized corpus has no privacy residue, the desensitized corpus is obtained. The desensitized corpus is used to train the target large language model.

2. The privacy data anonymization method for the field of large language models according to claim 1, characterized in that, The method for constructing the structured privacy rule base includes: Construct a general privacy rule layer and an industry-specific custom rule layer, wherein the industry-specific custom rule layer is used to extend industry-specific privacy rules; In the general privacy rule layer and the industry-customized rule layer, a matching mode and a verification function are configured for each privacy type. The matching mode includes the morphological features of the candidate fields using regular expressions. The morphological features include length, character type and delimiter. The verification function is used to verify the privacy information content. Construct a deterministic finite automaton for the matching patterns of each privacy type in the general privacy rule layer and the industry-customized rule layer; A deterministic finite state machine is constructed based on the matching patterns of all privacy types, and a structured privacy rule library is constructed based on the verification functions of each privacy type in the deterministic finite state machine, the general privacy rule layer, and the industry-customized rule layer.

3. The privacy data anonymization method for the field of large language models according to claim 2, characterized in that, Structured privacy information is extracted from the original mixed corpus based on a pre-built structured privacy rule base, including: The original mixed corpus is input into the deterministic finite state machine for automatic scanning to obtain candidate privacy fields and their privacy types. The candidate privacy fields are validated based on the validation function of the privacy type, and the candidate privacy fields that pass the validation are used as structured privacy fields. The location, privacy type, and structural features of the structured privacy fields are extracted to obtain structured privacy information. The structural features include field length, character type distribution, numerical range, encoding format, and fixed prefix / suffix identifiers.

4. The privacy data anonymization method for the field of large language models according to claim 1, characterized in that, Based on a pre-tuned, lightweight large language model, unstructured privacy information and cross-contextually related privacy information are extracted from the original mixed corpus, including: The original mixed corpus is input into a pre-tuned lightweight large language model to obtain a list of privacy entities, and the entity representation vector of each privacy entity in the list of privacy entities is extracted from the pre-tuned lightweight large language model. Clustering the entity representation vectors of multiple privacy entities yields multiple candidate clusters; Calculate the cohesion of multiple entity representation vectors within each candidate cluster, and count the number of entity representation vectors within each candidate cluster; Candidate clusters with a number of entity representation vectors greater than or equal to a preset number threshold and a cohesion greater than or equal to a preset cohesion threshold are designated as target clusters. Multiple privacy entities corresponding to entity representation vectors within the target cluster are designated as cross-context-related privacy information. Privacy entities corresponding to entity representation vectors outside the target cluster are designated as unstructured privacy information.

5. The privacy data anonymization method for the field of large language models according to claim 1, characterized in that, The fine-tuning process of the lightweight large language model includes: Obtain original business corpus samples and construct training labels for the original business corpus samples, wherein the training labels include inference basis samples, entity annotations of privacy fields, related entity groups across sentences or paragraphs, and the association level of related entity groups; The original business corpus samples are input into a lightweight large language model to obtain generated text, wherein the generated text includes reasoning basis and a list of privacy entities; The loss between the training labels and the generated text is calculated based on a pre-constructed loss function, wherein the mathematical expression for the loss is: In the formula, For the loss, The cross-entropy loss is used to measure the difference between the generated text and the training labels. To compare learning loss, The token sequence of the original business corpus sample. For the token location index, The set of token indices used as the basis for reasoning. A set of token indexes for privacy entities. For the first Token for each location, For historical token sequences, This is the entity loss weighting coefficient. To train the indexes of two different privacy entities in the labels, These are the entity representation vectors of two different privacy-preserving entities output by the lightweight large language model. This represents an index representing two distinct privacy entities belonging to the same associated entity group. For the intra-group association level of two different privacy entities, This represents an index representing two distinct privacy entities that do not belong to the same associated entity group. Represents the similarity function; The parameters of the lightweight large language model are frozen, and the adapter layer of the lightweight large language model is fine-tuned based on the loss.

6. The privacy data anonymization method for the field of large language models according to claim 1, characterized in that, Based on the structural features, virtual replacement data for the structured privacy information is generated, including: The structured privacy information format template is parsed, and the format template is divided into static segments and dynamic segments according to the structural features. The static segments include fixed delimiters, fixed unit codes, and fixed prefix / suffix identifiers. The dynamic segments include numerical sequences, letter sequences, mixed sequences, and variable-encoded suffixes. For the dynamic segment, based on the distribution of its character type and the statistical distribution characteristics of its numerical range, the Monte Carlo inverse transform method is used for random sampling to generate a preliminary replacement sequence; The initial replacement sequence is fed into a finite state machine verifier bound to the current privacy type to perform internal logic self-consistency verification. If the internal logic self-consistency verification fails, the process returns to the previous step of using the Monte Carlo inverse transform method for random sampling to generate an initial replacement sequence, until the internal logic self-consistency verification is passed or the maximum number of iterations is reached; if the internal logic self-consistency verification is passed, the initial replacement sequence is used as the final replacement sequence. The static segment and the final replacement sequence are concatenated according to the format template to obtain the virtual replacement data of the structured privacy information.

7. The privacy data anonymization method for the field of large language models according to claim 1, characterized in that, Aggregate and desensitize cross-context related privacy information in the original mixed corpus, including: Extract semantic role labels for multiple privacy entities from the cross-context associated privacy information; Placeholders matching the semantic role label are selected from a pre-constructed orthogonal placeholder pool, wherein the cosine similarity between any two placeholders in the orthogonal placeholder pool in the high-dimensional semantic space is less than a preset similarity threshold. The privacy entity is replaced based on the placeholder to obtain preliminary replaced privacy information; Calculate the semantic cohesion of the text that initially replaces the privacy information. If the semantic cohesion is greater than or equal to a preset risk threshold, return to the process of selecting a placeholder word that matches the semantic role label from a pre-constructed orthogonal placeholder pool until the semantic cohesion is less than the preset risk threshold. If the semantic cohesion is less than the preset risk threshold, use the initial replacement privacy information as the final replacement privacy information.

8. The privacy data anonymization method for the field of large language models according to claim 1, characterized in that, The training method for the privacy discrimination model includes: Obtain training samples and privacy residual labels of the training samples, wherein the training samples include positive samples containing privacy residuals and negative samples not containing privacy residuals; The training samples are input into a preset base model to obtain the privacy residue judgment result; The judgment loss of the privacy residue label and the privacy residue judgment result is calculated based on the preset binary classification cross-entropy loss function; The pedestal model is trained based on the judgment loss to obtain a privacy discrimination model.

9. The privacy data anonymization method for the field of large language models according to claim 1, characterized in that, Before using desensitized corpora to train the target large language model, the following steps are also included: An aligned training sample dataset is extracted from the original mixed corpus and the anonymized corpus; The original mixed corpus is input into the target large language model, and the first global representation vector is extracted from the penultimate hidden layer of the target large language model. The desensitized corpus is then input into the target large language model, and the second global representation vector is extracted from the penultimate hidden layer of the target large language model. ; Calculate the first global representation vector and the second global representation vector semantic alignment loss The semantic alignment loss The mathematical expression is: In the formula, This refers to the batch sample size. To align the training sample indices, Indicates the first A raw mixed corpus, Indicates the first A desensitized corpus, This represents the hidden layer representation extraction function. These are the KL divergence weighting coefficients. The input to the target large language model is The probability distribution of the output vocabulary at that time. The input to the target large language model is The probability distribution of the output vocabulary at that time. Indicates KL divergence; Based on the semantic alignment loss Semantic alignment training is performed on the target large language model.

10. A privacy-preserving data anonymization system for the field of large language models, characterized in that: include: The acquisition module is used to acquire the original mixed corpus; The privacy information extraction module is used to extract structured privacy information from the original mixed corpus based on a pre-built structured privacy rule base; and to extract unstructured privacy information and cross-context related privacy information from the original mixed corpus based on a pre-fine-tuned lightweight large language model. A structured privacy replacement module is used to extract the structural features of the structured privacy information and generate virtual replacement data for the structured privacy information based on the structural features; And based on the virtual replacement data, the structured privacy information of the original mixed corpus is virtually replaced; An unstructured privacy replacement module is used to identify semantic role attributes in the unstructured privacy information and to perform placeholder replacement on the unstructured privacy information of the original mixed corpus based on the semantic role attributes. The cross-context-related privacy replacement module is used to aggregate and desensitize cross-context-related privacy information in the original mixed corpus, wherein the semantic cohesion of the aggregated and desensitized information pointing to the original event is less than or equal to a preset risk threshold. The verification and loop module is used to take the original mixed corpus after the virtual data replacement, the placeholder replacement and the aggregation linkage desensitization as the initial desensitized corpus, and to judge the privacy residue of the initial desensitized corpus based on the constructed privacy discrimination model. If the initial desensitized corpus has privacy residue, it returns to the privacy information extraction module until the initial desensitized corpus has no privacy residue. When the initial desensitized corpus has no privacy residue, the desensitized corpus is obtained, wherein the desensitized corpus is used to train the target large language model.

Images