An efficient sensitive data detection method

Abstract

The present application relates to a kind of sensitive data efficient detection method, belong to data processing and data detection identification technical field.The present application includes the following steps: by flow mirror image copy data stream in transmission and obtain data stream, analysis into analysable text format;Collect sensitive data detection rule, by general reasoning big model to build rule base;Sensitivedata is quickly screened using rule base;For the data that is not screened out by rule base, it is divided into non-sensitive data and unidentified data by agent;Using domain knowledge and security field big model fine-tuning sensitive data detection big model to detect unidentified data, and dynamically update rule base based on result.The present application combines the efficiency of rule matching and the semantic understanding ability of big model, realizes the efficient and accurate detection of sensitive data, improves the adaptability and continuous optimization ability of the method, and provides reliable technical support for data security and privacy protection.

Description

TECHNICAL FIELD

[0001] The present application relates to a sensitive data efficient detection method, belonging to the technical field of data processing and data detection and identification. BACKGROUND

[0002] Under the current digital wave, data is growing explosively, covering personal social, medical health, enterprise operation, government services and other fields, and implying a large amount of sensitive information such as personal identity, financial data, enterprise secrets and special group information. The risk of data leakage is increasing day by day, and the consequences are unpredictable.

[0003] Traditional sensitive data detection methods are divided into three categories: rule matching, traditional machine learning models and large model-based analysis. The rule matching method relies on manual rule setting, which is effective for fixed format data, but difficult to deal with complex data, and the rule maintenance is difficult;

[0004] Traditional machine learning models such as decision tree, Naive Bayes and SVM require a large amount of labeled data for training, while sensitive data labeling faces the problems of difficult data acquisition, high cost and high privacy risk. At the same time, these models have high dependence on feature engineering, and it is difficult to select features in complex data environment, which affects the performance and generalization ability of the model. Moreover, when the data distribution changes or new types of sensitive data appear, the model usually needs to be retrained, which has poor adaptability.

[0005] Although large models perform well in semantic understanding, they require high computing resources, the decision-making process is difficult to explain, and further adaptation is required when processing data in specific fields.

[0006] Based on the above situation, it is urgent to combine rule matching and large model analysis to fully utilize the advantages of each technology and make up for its shortcomings. Some research combines rule-based methods with traditional machine learning models, but this combination still has not made satisfactory breakthroughs when facing complex semantics and new data types. There are also attempts to directly apply large models to sensitive data detection tasks, but due to factors such as cost and explainability, it is difficult to promote on a large scale. Therefore, the present application aims to propose a sensitive data detection method based on rules and large models, which cleverly combines the advantages of both technologies to provide an innovative and practical solution for identifying sensitive information in public data, effectively dealing with complex and changing data environments and growing security challenges. SUMMARY

[0007] To solve the problems of the prior art, the present application provides a sensitive data efficient detection method, which can efficiently and accurately detect various types of sensitive data.

[0008] The technical scheme of the present application is: a sensitive data efficient detection method, comprising:

[0009] The data flow in the transmission is copied and obtained by flow mirroring, and is parsed into an analyzable text format;

[0010] Sensitive data detection rules are collected, and a rule base is constructed by a general reasoning large model;

[0011] Sensitive data is quickly screened by using the rule base;

[0012] For data that is not screened by the rule base, it is divided into non-sensitive data and unidentified data by an intelligent agent;

[0013] A sensitive data detection large model is fine-tuned using domain knowledge and a security domain large model to detect unidentified data, and the rule base is dynamically updated based on the results.

[0014] Further, the method specifically comprises the following steps:

[0015] S1. Data acquisition and conversion: In the data transmission process, the data flow is copied and backed up in real time by flow mirroring, generating an independent mirror copy; based on the mirror copy, the data flow is obtained by using packet capture technology without interfering with the original business transmission; the captured data flow is parsed and format standardized, and structured, unstructured and semi-structured data are uniformly processed into an analyzable text format;

[0016] S2. Rule base construction: Collect various sensitive data detection rules, input the collected rules into a general reasoning large model, and use the semantic understanding and reasoning ability of the large model to efficiently optimize, integrate and correlate the rules, and construct a high-efficiency and adaptive rule base;

[0017] S3. Using the constructed rule base, the converted data is quickly screened, and sensitive data is screened out; the screened sensitive data is directly marked as sensitive data; the data that is not screened is input into the next step for operation;

[0018] S4. For data that is not screened by the rule base, an intelligent agent is constructed, trained and learned based on non-sensitive data rules and unidentified data rules, which can distinguish non-sensitive data and unidentified data; the data that is not screened by the rule base is input into the intelligent agent, and the intelligent agent further divides it into non-sensitive data and unidentified data;

[0019] S5. A general large model in the security field is selected, an adapter is added, and the large model is fine-tuned using sensitive data detection domain professional knowledge to make it more focused on the sensitive data detection task; the unidentified data in S4 is input into the fine-tuned large model for detection, and the judgment results of non-sensitive data and sensitive data are output, and the rule base is updated in a timely manner according to the output results of the large model.

[0020] Further, the S1 comprises:

[0021] S11. Deploying traffic mirroring at the network key node of data transmission, real-time copying all data streams in transmission, generating mirror copies independent of the original transmission link, synchronously completing backup storage, for ensuring not to interfere with the original data transmission process;

[0022] S12. In the isolated environment where the mirror copy is located, capturing all data streams through packet capture tools to obtain complete transmission data content; the captured data includes government data, medical data, financial data, education data, enterprise data, and personal data, etc. of multiple fields;

[0023] S13. For the captured data, distinguishing structured, unstructured, and semi-structured types, and through analysis, extraction, encoding conversion, etc. processing, converting them into text format that can be directly used for analysis.

[0024] Further, the S2 comprises:

[0025] S21. Collecting multi-dimensional sensitive data detection rules, including regular sensitive data format rules, sensitive keyword rules, and specific pattern rules, and storing them according to rule types;

[0026] S22. Normalizing the collected rules, removing duplicate rules, correcting logical conflict rules, and converting rules described in natural language into structured expressions, for ensuring uniform rule format and enabling large model recognition;

[0027] S23. Inputting the pre-processed rules into a general reasoning large model, analyzing the logical association between rules through the semantic understanding ability of the large model, and performing optimization operations such as merging redundant items, supplementing missing conditions, and expanding applicable scenarios on the rules; based on the optimization results output by the large model, integrating rules across types, establishing hierarchical associations between rules, and forming a rule network covering multiple scenarios and multiple types;

[0028] S24. Storing the integrated rule set as a structured rule library, verifying the completeness and accuracy of the rule library through test data sets, and finally forming a comprehensive, efficient, and scenario-adaptive sensitive data detection rule library.

[0029] Further, the S3 comprises:

[0030] S31. Converting each rule in the rule library into a feature vector , wherein is the feature dimension; for format rules, regular expression pattern features are extracted; for keyword rules, semantic vectors are generated using word embeddings; and specific pattern rules are converted into Boolean feature vectors through logical expressions;

[0031] S32. The converted text data is segmented into words to generate a word sequence ; the feature vector of each word is extracted , and a data matrix is constructed , where n is the number of words;

[0032] S33. For the feature vector of each rule , the cosine similarity between it and each word vector in the data matrix is calculated:

[0033]

[0034] Set the similarity threshold , if , then determine that the word matches the rule ;

[0035] S34. For the words that match the rule, combine the rule weight and the matching strength to calculate the sensitivity score ; The sensitivity score is determined by large model training, based on similarity normalization;

[0036] S35. Set the sensitivity threshold , when , determine that the data belonging to the word is sensitive data; for consecutive matching words, use a sliding window mechanism to calculate the cumulative score:

[0037]

[0038] where is the window length, starting from the th word in the word sequence, covering the th to th word, if , then all data in the entire window is marked as sensitive; the rest of the data is treated as non-sensitive data as input for the next step.

[0039] Further, the S4 includes:

[0040] S41. Collect non-sensitive data samples and unidentified data samples , two types of samples are labeled, non-sensitive is labeled and unidentified label is ; feature extraction is performed on the samples to generate a feature matrix , is the total number of samples, is the feature vector of the th sample, is the feature dimension, and the label vector ;

[0041] S42. Construct an agent model based on bidirectional LSTM, including input layer, hidden layer and output layer; the input layer receives the feature vector ; the hidden layer extracts the context semantic features through bidirectional LSTM and calculates the hidden state:

[0042]

[0043] wherein, , represent the forward and reverse hidden states at time t, , represent the forward and reverse hidden states at time t-1 and t+1, represents the input vector at time t;

[0044] concatenate the bidirectional hidden states to get , and output the probability that the sample belongs to unidentified data through a fully connected layer :

[0045]

[0046] wherein, is the Sigmoid activation function, and are learnable parameters;

[0047] S43. Convert the unfiltered data output by S3 into a feature vector and input it into the trained agent to obtain the prediction probability of each sample ; set the classification threshold , if , it is determined as non-sensitive data, and if , it is determined as unidentified data;

[0048] S44. Randomly extract part of the classification results for manual verification, calculate the precision and recall , is the true positive, is the false positive, is the false negative, and if Or Below the preset threshold, use error samples to expand the training set, retrain the agent to optimize the classification performance.

[0049] Further, the S5 comprises:

[0050] S51. Collect sensitive data detection field expertise, extract entities, entity relationship to build a set of triples Wherein , For entities, For relationships; based on the importance of rules and the confidence of relationships, assign weights to each triple , the calculation formula is:

[0051]

[0052] Wherein, The normalized frequency of triples in the rule base, The confidence of the relationship , The weight coefficient;

[0053] S52. On the basis of the general large model in the security field, add a domain adaptation layer to realize the focused adaptation of the large model to the sensitive data detection task through parameter adjustment; insert an adapter module in the large model, and the output is:

[0054]

[0055] Wherein, The original output features of the large model, , The dimension reduction coefficient, The feature dimension, The activation function, And The bias term;

[0056] Collect sensitive data and non-sensitive data, as well as professional knowledge in the field of sensitive data detection, to build a fine-tuning sample set Wherein The feature vector of the th sample, The label, 0 represents non-sensitive data, and 1 represents sensitive data;

[0057] Train the model with the fine-tuning sample set, and the loss function is:

[0058]

[0059]

[0060]

[0061] wherein, is the classification cross-entropy loss; is the knowledge guidance loss; is the model predicted sensitive probability of the i-th sample; is the total number of samples; is the embedding vector of the entity or the relation; is the weight coefficient, denotes the set of all triplets, denotes the weight of a single triplet is the predicted probability of the i-th sample. The application also provides a sensitive data efficient detection system, which comprises a module for executing the sensitive data efficient detection method.

[0062] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the sensitive data efficient detection method when executing the program.

[0063] The application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the sensitive data efficient detection method.

[0064] The application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the sensitive data efficient detection method.

[0065] The application has the following beneficial effects:

[0066] 1. The application can quickly acquire and process data by packet capture, mirroring, and data stream conversion during data transmission; the constructed rule library is comprehensive and optimized by a large model, and can quickly match and identify sensitive data.

[0067] 2. The application accurately identifies various types of sensitive data from multiple dimensions by comprehensively using various rule types and large models for precise detection; it is suitable for various types of data and has wide versatility and adaptability.

[0068] 3. The constructed agent can subdivide detection of different types of data, improving detection flexibility.

[0069] 4. The application dynamically updates the rule library according to the output results of the large model, maintaining the timeliness and accuracy of the detection method.

[0070] 5. The application detects sensitive data in a timely manner and issues warnings, reducing data security risks, forming a closed-loop data security management system, and assisting in all-round protection of data security. ​

[0071] 6. The method of this invention achieves accurate detection of sensitive data by combining the high efficiency of rule matching with the semantic understanding capabilities of large-scale models. Rule matching can quickly locate sensitive data with clear formats and features, while large-scale models can handle sensitive information with varied formats and complex semantics. By integrating these two technologies, this invention not only improves the accuracy and efficiency of sensitive data detection but also enhances the adaptability and interpretability of the method. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0073] Figure 1 This is a flowchart illustrating the architecture of an efficient sensitive data detection method provided by the present invention.

[0074] Figure 2 This is a flowchart illustrating a method for efficient detection of sensitive data provided by the present invention;

[0075] Figure 3 This is a module connection diagram of a high-efficiency sensitive data detection method provided by the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the present invention, and not all of it. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention. It should be noted that, without conflict, the various embodiments and features in this application can be arbitrarily combined.

[0077] Furthermore, the execution subject of this invention can cover various types of terminals, such as computers, smartphones, tablets, wearable devices, mobile internet devices, and other terminals with text processing capabilities. This invention does not impose any limitations on these.

[0078] Example 1: As Figures 1-3 As shown, a method for efficient detection of sensitive data includes the following steps:

[0079] S1. Data acquisition and conversion: During data transmission, the data stream is copied and backed up in real time through traffic mirroring to generate an independent mirror copy; based on the mirror copy, the data stream is acquired using packet capture technology without interfering with the original business transmission; the captured data stream is parsed and standardized, and structured, unstructured and semi-structured data are uniformly processed into an analyzable text format;

[0080] S2. Rule base construction: Collect various sensitive data detection rules, input the collected rules into a general reasoning large model, and use the semantic understanding and reasoning ability of the large model to optimize, integrate and correlate the rules to construct a rule base;

[0081] S3. Using the constructed rule base, the converted data is quickly screened to screen out sensitive data; the screened sensitive data is directly marked as sensitive data; the data that is not screened is operated in the next step;

[0082] S4. For data that is not screened by the rule base, construct an agent and train and learn based on non-sensitive data rules and unidentified data rules; input the data that is not screened by the rule base into the agent, and further divide it into non-sensitive data and unidentified data by the agent;

[0083] S5. Select a general large model in the security field, add an adapter, and fine-tune the large model using sensitive data detection field expertise; input the unidentified data in S4 into the fine-tuned large model for detection, output the judgment results of non-sensitive data and sensitive data, and update the rule base in a timely manner according to the output results of the large model.

[0084] Further, the S1 includes:

[0085] S11. Deploy traffic mirroring at network key nodes of data transmission to copy all data streams in real time, generate a mirror copy independent of the original transmission link, and complete backup storage synchronously to ensure that the original data transmission process is not disturbed;

[0086] S12. In the isolated environment where the mirror copy is located, capture the mirror data stream in full through a packet capture tool to obtain complete transmission data content; the captured data includes multiple domain types of government data, medical data, financial data, education data, enterprise data and personal data;

[0087] S13. For the captured data, distinguish between structured, unstructured and semi-structured types, and convert them into a text format that can be directly used for analysis through parsing, extraction, encoding conversion processing.

[0088] This step generates an independent mirror copy by mirroring the data stream in transmission in real time, and then obtains the complete data stream based on the copy by packet capturing technology without interfering with the original transmission. The obtained data covers multiple fields such as government affairs, medical treatment, finance, education, enterprise and personal data, and then it is uniformly converted into an analyzable standardized data format, providing a basis for subsequent processing.

[0089] Further, the S2 comprises:

[0090] S21. Collect multi-dimensional sensitive data detection rules, including regular sensitive data format rules, sensitive keyword rules, and specific mode rules, and store them by rule type; wherein the specific mode rule can include the characteristics of specific combination in the text, for example, a rule based on a scenario combination feature can be used;

[0091] S22. Normalize the collected rules, remove duplicate rules, correct logical conflict rules, and convert natural language description rules into structured expressions to ensure uniform rule format and enable large model recognition;

[0092] S23. Input the preprocessed rules into a general reasoning large model, analyze the logical association between rules through the semantic understanding ability of the large model, and perform optimization operations such as merging redundant items, supplementing missing conditions, and expanding applicable scenarios on the rules; based on the optimization results output by the large model, integrate the rules across types, establish hierarchical associations between rules, and form a rule network covering multiple scenarios and multiple types;

[0093] S24. Store the integrated rule set as a structured rule library, verify the completeness and accuracy of the rule library through a test data set, and finally form a rule library for sensitive data detection.

[0094] This step first collects and classifies multiple types of rules such as regular sensitive data format rules, sensitive keyword rules, and specific mode rules, and then inputs them into a general reasoning large model after preprocessing and normalization. With the help of the semantic understanding and reasoning ability of the large model, the rules are optimized, integrated, and associated, and finally a comprehensive, efficient, and adaptive sensitive data detection rule library is constructed.

[0095] Further, the S3 comprises:

[0096] S31. Convert each rule in the rule library into a feature vector , wherein is the feature dimension; for format rules, extract regular expression pattern features; for keyword rules, use word embedding to generate semantic vectors; and for specific mode rules, convert them into Boolean feature vectors through logical expressions;

[0097] S32. The converted text data is segmented to generate a word sequence ; a feature vector of each word is extracted , and a data matrix is constructed , where n is the number of words.

[0098] S33. For each rule, a feature vector is calculated , and the cosine similarity between the feature vector and each word vector in the data matrix is calculated:

[0099]

[0100] A similarity threshold is set , and if , it is determined that the word matches the rule .

[0101] S34. For the words that match the rule, the sensitive score is calculated based on the rule weight and the matching strength . The sensitive score is determined by large model training based on similarity normalization.

[0102] S35. A sensitive threshold is set , and when , it is determined that the data belonging to the word is sensitive data. For consecutive matching words, a sliding window mechanism is used to calculate the cumulative score:

[0103]

[0104] where is the window length, starting from the th word in the word sequence, covering the th to th word. If , the data in the entire window is marked as sensitive. The remaining data is treated as non-sensitive data as input for the next step.

[0105] The above steps use the constructed rule library to quickly scan and match the converted standardized data: first, the rules and data are converted into feature vectors, and the matching degree is calculated by cosine similarity; then, the data sensitive score is calculated by combining the rule weight and the matching strength, the sensitive threshold is set, and the sensitive data is determined by the word unit score and the sliding window cumulative score; finally, the sensitive data is marked and stored, and the false positives are reduced through context filtering, achieving efficient identification and marking of sensitive data, providing a basis for subsequent protection processing.

[0106] Further, the S4 comprises:

[0107] S41. Collecting non-sensitive data samples and unidentified data samples , both types of samples are labeled, non-sensitive is labeled and unidentified label is ; feature extraction is performed on the samples to generate a feature matrix , is the total number of samples, is the feature vector of the th sample, is the feature dimension, and the label vector ;

[0108] S42. Constructing an agent model based on bidirectional LSTM, including input layer, hidden layer and output layer; the input layer receives the feature vector ; the hidden layer extracts context semantic features through bidirectional LSTM and calculates the hidden state:

[0109]

[0110] wherein, , represent the forward and reverse hidden states at time t, , represent the forward and reverse hidden states at time t-1 and t+1, represents the input vector at time t;

[0111] concatenate the bidirectional hidden states to get , and output the probability that the sample belongs to unidentified data through a fully connected layer :

[0112]

[0113] wherein, is the Sigmoid activation function, and are learnable parameters;

[0114] S43. Convert the non-screened data output by S3 into a feature vector and input it into the trained agent to get the prediction probability of each sample ; set the classification threshold , if , it is determined as non-sensitive data, and if , it is determined as unidentified data.

[0115] S44. Randomly extract part of the classification results for manual verification, calculate the precision and recall rate , true positive, false positive, false negative, if or is lower than the preset threshold, use the error sample to expand the training set, retrain the agent to optimize the classification performance.

[0116] Through the above steps, for the non-sensitive data identified in S3, first build an agent based on bidirectional LSTM, use non-sensitive data and unidentified data samples for training, and optimize the model parameters through cross-entropy loss function, so that it can distinguish the two types of data based on data features and semantic information. Input the non-sensitive data into the trained agent, divide it into non-sensitive data and unidentified data according to the prediction probability output by the model, and check the results through precision and recall rate, and update the model if necessary to improve the classification accuracy, and provide clear objects for subsequent large model detection.

[0117] Further, the S5 comprises:

[0118] S51. Collect professional knowledge in the field of sensitive data detection, extract entities and relationships between entities to build a set of triples , wherein , entity, relationship; based on the importance of the rule and the confidence of the relationship, assign a weight to each triple , the calculation formula is:

[0119]

[0120] wherein, is the normalized frequency of the triple in the rule base, is the confidence of the relationship , is the weight coefficient;

[0121] S52. Add a domain adaptation layer based on a general large model in the security field, and realize the focused adaptation of the large model to the sensitive data detection task through parameter adjustment; insert an adapter module in the large model, and the output is:

[0122]

[0123] wherein, is the original output feature of the large model, , is the dimension reduction coefficient, is the feature dimension, is an activation function, and is a bias term;

[0124] Collect sensitive data and non-sensitive data, and professional knowledge in the field of sensitive data detection, build a fine-tuning sample set wherein is the feature vector of the i-th sample, is the label, 0 represents non-sensitive data, and 1 represents sensitive data; Train the model with the fine-tuning sample set, and the loss function is:

[0125]

[0126]

[0127]

[0128]

[0129] wherein, is the classification cross-entropy loss; is the knowledge-guided loss; is the model predicted sensitive probability of the i-th sample; is the total number of samples; is the embedding vector of the entity or relationship; is the weight coefficient, denotes the set of all triples, denotes the weight of a single triple is the predicted probability of the i-th sample. The above steps add an adapter to the general large model in the security field, fine-tune the model using a hybrid loss function to adapt to the sensitive data detection task. The unclassified data is input into the fine-tuned large model, and the sensitive / non-sensitive determination result is output; based on the model output, new features are extracted, and the rule base is dynamically updated, forming a "detection-feedback-optimization" closed loop, continuously improving the detection performance.

[0130] The present application also provides a sensitive data efficient detection system, the system comprises:

[0131] The data acquisition and conversion module is used for real-time copying and backup of data streams through flow mirroring in the data transmission process, generating independent mirror copies; based on the mirror copies, the data stream is acquired by using the packet capture technology without interfering with the original business transmission; the captured data stream is parsed and format standardized converted, and the structured, unstructured and semi-structured data are uniformly processed into an analyzable text format;

[0132]

[0133] ​​​Rule base construction module: used for collecting various sensitive data detection rules, inputting the collected rules into a general reasoning large model, optimizing, integrating and correlation analyzing the rules by means of semantic understanding and reasoning capability of the large model, and constructing a rule base;

[0134] Rule base matching module: used for screening the converted data quickly by using the constructed rule base, screening out sensitive data; directly marking the screened sensitive data as sensitive data; and inputting the un-screened data into the agent construction module for operation;

[0135] Agent construction module, used for constructing an agent for the data un-screened by the rule base, training and learning based on non-sensitive data rules and unidentified data rules; inputting the data un-screened by the rule base into the agent, and further dividing the data into non-sensitive data and unidentified data by the agent;

[0136] Large model fine-tuning and detection module, used for selecting a general large model in the security field, adding an adapter, fine-tuning the large model by using professional knowledge in the sensitive data detection field, inputting the unidentified data in S4 into the fine-tuned large model, detecting, outputting the judgment results of non-sensitive data and sensitive data, and updating the rule base in time according to the output results of the large model.

[0137] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the sensitive data efficient detection method when executing the program.

[0138] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the sensitive data efficient detection method.

[0139] Embodiment 2: as shown in the following table, a sensitive data efficient detection method comprises the following steps: Figures 1-3

[0140] S1. Data acquisition and conversion: in the data transmission process, the data stream is copied and backed up in real time through flow mirroring to generate an independent mirror copy; based on the mirror copy, the data stream is acquired by using packet capture technology without interfering with the original business transmission; the acquired data covers multiple fields such as government affairs, medical treatment, finance, education, enterprise and personal data. The captured data stream is parsed and standardized converted, and the structured, unstructured and semi-structured data are uniformly processed into an analyzable text format, thereby providing a standardized input for subsequent detection;

[0141] Further, the S1 comprises:

[0142] ​S11. Configure port mirroring function (using SPAN or RSPAN technology) at the core node of data transmission (such as network switch, server export gateway, data center interface, etc.), and copy all original data streams (including structured data such as database interaction information, semi-structured data such as API interface message, unstructured data such as file transmission content, etc.) flowing through the target node to the independent mirror port in real time, forming a mirror data stream completely consistent with the original data stream.

[0143] S12. Import the mirror data stream into a dedicated isolated server (physically or logically isolated from the production network), and backup in real time through high-speed storage devices (such as SSD array) to ensure the integrity and timing consistency of the mirror data, and not affect the normal transmission of the original data.

[0144] S13. Deploy packet capture tools (such as Wireshark, Suricata, etc.) on the isolated server, and set packet capture rules for the mirror data stream: filter by data source (such as government system IP segment, medical database port, financial transaction protocol, etc.), data type (such as HTTP / HTTPS message, FTP file, database query statement, etc.), only capture target data for detection, reduce invalid data processing amount. Perform integrity check (such as checksum comparison) on the captured mirror data to ensure its consistency with the original mirror data stream, and avoid data distortion caused by transmission or storage anomalies.

[0145] S14. Analyze the captured mirror data and identify its format type: structured data (such as JSON, XML, database table structure data): extract field name, field value and hierarchical relationship; semi-structured data (such as log file, CSV, Markdown document): identify separator, label and key information block; unstructured data (such as plain text, scanned copy, audio clip): distinguish types by format identifier (such as file header information).

[0146] S15. For structured / semi-structured data: use special parsing tools (such as JSON parser, Logstash) to convert it to a unified key-value pair text format (encoded as UTF-8), preserving the original semantic association;

[0147] For unstructured data: use OCR technology (such as PaddleOCR) to extract text content from images, and use voice-to-text tools (such as Baidu AI speech recognition) to convert audio to text, and finally unify it into an analyzable text format.

[0148] S16. Remove irrelevant information in the converted text (such as transmission protocol header, redundant spaces, error encoding characters), correct data format errors (such as unify the date format to "YYYY-MM-DD"). Temporarily mask the sensitive fields (such as ID number, bank card number) in the mirror data that have been identified by the preset simple rules (such as replace the middle 10 characters with "*"), and only keep the feature fragments (such as the first 6 administrative division codes and the last 4 tail numbers) for detection, to prevent the mirror data from being leaked in the preprocessing stage.

[0149] S17. Store the preprocessed standardized mirror data into the detection special database (such as Elasticsearch) in the isolated environment, and establish multi-dimensional index according to data source, type, timestamp, to ensure that the subsequent rule matching, agent analysis and other steps can quickly call the mirror data, and improve the detection efficiency.

[0150] S2. Rule library construction: collect various sensitive data detection rules, input the collected rules into the general reasoning large model, and optimize, integrate and correlation analyze the rules with the help of the semantic understanding and reasoning ability of the large model, to construct a comprehensive, efficient and adaptive rule library;

[0151] Further, the S2 comprises:

[0152] S21. Collect three types of sensitive data detection rules, specifically: regular sensitive data format rules: rules based on fixed format features, such as "18-digit ID number (first 6 digits address code + 8 digits birth date + 3 digits sequence code + 1 digit check code)" "11-digit mobile phone number (beginning with 13 / 14 / 15 / 17 / 18 / 19)" "20-digit bank card number (including card issuing bank identifier + personal account identifier)" and the like; sensitive keyword rules: rules based on core sensitive words, such as "password" "key" "medical record" "credit report" "enterprise business secret" "government secret" "student performance" and the like; specific pattern rules: rules based on scene-based combination features, such as "name + ID number + home address" combination (complete personal privacy information), "transaction amount + bank card number + payment time" combination (financial sensitive operation), "patient ID + diagnosis result + medication record" combination (medical sensitive information) and the like.

[0153] S22. Remove duplicate rules: such as "11-digit mobile phone number" and "mobile phone number (11 digits)", keep the former as the standard rule.

[0154] S23. Resolve logical conflicts: As two rules define "bank card number is 19 digits" and "bank card number is 20 digits" respectively, confirm "bank card number length is 16-20 digits" by querying "Bank Card Business Management Measures", and merge into "16-20 digit bank card number" rule.

[0155] S24. Adopt a general reasoning large model (such as DeepSeek R1), and input the preprocessed rules as input, and set the prompt word as: "Perform the following processing on the input sensitive data detection rules: 1. Merge redundant rules; 2. Supplement rule applicable scenarios; 3. Establish associations between different types of rules; 4. Optimize the accuracy of rule expression."

[0156] S25. The large model outputs optimized rules as follows: Redundancy merging: "16-digit bank card number", "19-digit bank card number", "20-digit bank card number" are merged into "16-20-digit bank card number" (covering the full format), and the verification rules are supplemented. Scene supplement: Supplement the applicable scenarios of "keyword'medical record'" rule "applicable to hospital systems, medical centers, and telemedicine platforms", and clarify the business boundaries of the rule. Cross-type association: Establish "format rule + keyword rule" association, such as "18-digit ID number (format rule)" associated with keywords "ID card" and "identity information", forming the strengthened rule "appearance of keyword 'ID card' and matching 18-digit format"; Establish "specific pattern rule + format rule" association, such as "patient ID (format rule: letter + 8-digit number)" combined with "diagnosis result (keyword rule)" to form "patient ID + diagnosis result" medical sensitive pattern.

[0157] Through the above steps, the constructed rule library not only retains the core features of the original rules, but also realizes cross-type association and scene adaptation through large model optimization, providing efficient and comprehensive rule support for subsequent sensitive data detection.

[0158] S3. Use the constructed rule library to quickly filter the converted data and filter out sensitive data; directly mark the filtered sensitive data as sensitive data; for data that has not been filtered, proceed to the next step;

[0159] Further, the S3 comprises:

[0160] S31. Select 3 typical rules from the rule library and convert them into 300-dimensional feature vectors (based on a general word embedding model):

[0161] Rule 1 (format rule): "16-20 digit bank card number", vector contains "financial account" "numeric sequence" "length feature" and other dimensions;

[0162] Rule 2 (keyword rule): "password reset", vector contains "account operation" "sensitive behavior" and other semantic features;

[0163] Rule 3 (specific pattern rule): "name + login password" combination, vector contains the association feature of "identity + permission certificate".

[0164] S32. Convert data features to text data, and the converted text data is: "User Wang Moumou initiates password reset, binds bank card number 6222021234567890123, verification code 123456". After segmentation, the word sequence is generated: [user, Wang Moumou, initiates, password reset, binds, bank card number, 6222021234567890123, verification code, 123456], extract 300-dimensional feature vector of each word, construct 8x300 data matrix (total 8 words)

[0165] S33. For the feature vector of each rule , calculate the cosine similarity with each word vector in the data matrix ;

[0166] Set the similarity threshold , the calculation result is as follows:

[0167] The similarity of rule 1 and "6222021234567890123" (≥0.75), determine matching;

[0168] The similarity of rule 2 and "password reset" (≥0.75), determine matching;

[0169] The similarity of rule 3 and "Wang Moumou" (name) (0.75), the similarity with "password reset" is sim=0.80 (0.75), because rule 3 is a combination pattern, it needs to match "name + password related words" at the same time, so it is not determined as a whole match.

[0170] S34. The rule weight is preset as: rule 1 (0.85), rule 2 (0.90), rule 3 (0.95), and the matching strength "6222021234567890123": , ; "password reset": , .

[0171] S35. Set the sensitivity threshold Both “6222021234567890123” (0.99 ≥ 0.8) and “password reset” (1.11 ≥ 0.8) are marked as sensitive tokens.

[0172] S36. Window length The cumulative score is calculated for the continuous tokens “password reset + bank card number 6222021234567890123”: The segment is marked as sensitive data.

[0173] The “password reset” “6222021234567890123” and “password reset + bank card number 6222021234567890123” segments are directly marked as sensitive data; “user” “Wang S S” “initiate” “bind” “verification code” “123456” are further processed in S4.

[0174] S4. For non-sensitive data and data that is difficult to detect directly through the rule base, an intelligent agent is constructed. The intelligent agent is trained and learned based on non-sensitive data rules and unidentified data rules, and can deeply analyze the characteristics and semantic information of the data to distinguish non-sensitive data and unidentified data. The non-sensitive data in step 3 is input into the intelligent agent, which further divides it into non-sensitive data and unidentified data.

[0175] Further, the S4 includes:

[0176] S41. Based on the non-sensitive data output by S3 (such as “user” “Wang S S” “initiate” “bind” “verification code” “123456”), supplement the same type of data (open source data set) to construct a training set. Convert the text into a 300-dimensional feature vector to construct a training matrix The label vector is .

[0177] S42. Use a bidirectional LSTM network, which includes: an input layer that receives a 300-dimensional feature vector; a bidirectional LSTM layer with a hidden layer dimension of 128 to extract context semantic features; a fully connected layer that concatenates the bidirectional hidden state to output the prediction probability. Optimizer: Adam, learning rate 0.001; loss function: binary cross-entropy loss; batch size: 32, training rounds: 50; validation set: reserve 20% samples for validation;

[0178] S43. Convert the data that is not filtered out by S3 into a feature vector: “user” is a feature vector " Wang Moumou " is the feature vector " 123456 " is the feature vector , etc. Input the trained agent to get the prediction probability of each sample ;

[0179] S44. Set the classification threshold If , it is determined to be non-sensitive data, and if , it is determined to be unidentified data. The prediction results are shown in Table 1:

[0180] Table 1 is the prediction result

[0181] Data item Predicted probability Classification result User 0.12 Non-sensitive data Wang 0.85 Unidentified data Initiate 0.09 Non-sensitive data Binding 0.15 Non-sensitive data Verification code 0.40 Non-sensitive data 123456 0.82 Unidentified data

[0182] Unidentified data (such as " Wang Moumou " may be a real name, and " 123456 " may be a verification code) is input into the large model of the next step for accurate detection, and non-sensitive data is no longer operated.

[0183] This step realizes further subdivision of ambiguous data that cannot be accurately identified by the rule base through agent classification, providing a basis for subsequent fine processing.

[0184] S5. Select a general large model in the security field (such as Foundation-Sec), add an adapter, and use sensitive data detection field knowledge to fine-tune the large model to make it more focused on sensitive data detection tasks; input the unidentified data in S4 into the fine-tuned large model for efficient and accurate detection, output the determination results of non-sensitive data and sensitive data, and update the rule base in a timely manner according to the output results of the model.

[0185] Further, the S5 includes:

[0186] S51. Integrate S2 rule base and sensitive data detection field knowledge to construct a triple set in descending order of weight, supporting fast retrieval;

[0187] S52. Insert an adapter in the general large model in the security field, and output the formula:

[0188]

[0189] Where is the original output feature of the large model, , and are bias terms.

[0190] S53. Loss function:

[0191]

[0192] wherein is the classification cross-entropy loss, is the knowledge guidance loss, which constrains the model output to be consistent with the domain knowledge; the learning rate is 5e-5, and the training is performed for 100 rounds.

[0193] S54. The unclassified data output by S4 ("Wang Someone", "verification code", "123456") is input into the fine-tuned model, and the sensitive probability is output; the sensitive probability of "Wang Someone" is 0.92, which is determined as sensitive data; the sensitive probability of "verification code" is 0.35, which is determined as non-sensitive data; the sensitive probability of "123456" is 0.28, which is determined as non-sensitive data.

[0194] S55. Based on the model output, the following is added in the rule library of S2:

[0195] (1) "real name" is a sensitive keyword rule;

[0196] (2) "6-digit number +'verification code'" is a specific pattern rule (determined as non-sensitive)

[0197] Through this process, the unclassified data output by S4 is accurately classified, and the rule library is dynamically updated, improving the overall detection performance.

[0198] The method of the present application compares the advantages and disadvantages of traditional rule-based, machine learning-based, and large model-based methods.

[0199] Table 2: Comparison of advantages and disadvantages of each method

[0200] Method type Advantages Disadvantages Rule-based only Fast detection speed, suitable for structured and well-defined data (such as ID number) matching; low initial development cost, no data labeling or model training required. Limited coverage, unable to handle variant data (such as "hidden middle digit mobile number") and semantically related data (such as "family member's bank card"); high maintenance cost, requiring continuous rule updates and difficulty in adapting to data format changes; Machine learning only Automatic learning of feature variants (such as "11-digit number + 'phone' recognized as mobile number"); high inference efficiency after training, suitable for batch processing of structured / semi-structured data. Dependence on large amounts of labeled data, poor generalization ability for unseen feature combinations (such as "6-digit number + 'pick up'"). Based on large models only Strong semantic understanding, capable of handling ambiguous data (such as "fill in social security number on the form"); no need for manual feature design, directly processing unstructured data (such as emails, chat records). Low detection efficiency, high cost, time-consuming for batch processing of large-scale data; lack of domain knowledge may lead to misjudgment (such as "desensitized mobile number" misjudged as sensitive); poor interpretability, high cost of training and fine-tuning (requires large amounts of domain data and computing resources). The method of the present invention Integrating the advantages of rule efficiency, machine learning feature learning, and large model semantic understanding, the method processes data in layers, balancing efficiency and coverage; combining rule library and continuous optimization and update, reducing misjudgment and reducing manual maintenance cost; Complex initial architecture building, requiring simultaneous construction of rule library, agent, and large model adapter; high technical stack requirements, needing to consider rule engine, machine learning, large model fine-tuning, and other multi-field technologies.

[0201] The method of the present application combines the efficiency of rule matching and the semantic understanding ability of large models to achieve accurate detection of sensitive data. Rule matching can quickly locate sensitive data with clear format and characteristics, while large models can handle sensitive information with variable format and complex semantics. By integrating the two technologies, the present application not only improves the accuracy and efficiency of sensitive data detection, but also enhances the adaptability and interpretability of the method.

[0202] The specific embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A method for efficient detection of sensitive data, characterized in that: The method comprises: Copy the data stream in the transmission by flow mirroring and obtain the data stream, and parse into an analyzable text format; Collect sensitive data detection rules, and construct a rule base by a general reasoning large model; Screen sensitive data quickly by using the rule base; For data that is not screened out by the rule base, divide it into non-sensitive data and unidentified data by an intelligent agent; Use domain knowledge and a security domain large model to fine-tune a sensitive data detection large model to detect the unidentified data, and dynamically update the rule base based on the result; The method comprises: S41. Collect samples of non-sensitive data and unidentified data samples The two types of samples were labeled separately, with the non-sensitive label being... and unidentified tags Feature extraction is performed on the samples to generate a feature matrix. , The total number of samples, For the first The feature vector of each sample For feature dimensions, label vector ; S42. Constructing a bidirectional LSTM-based agent model, including an input layer, a hidden layer, and an output layer; the input layer receives the feature vector ; the hidden layer extracts the context semantic features through bidirectional LSTM and calculates the hidden state: ； wherein, , denotes the forward, backward hidden state at time t, denotes the forward hidden state at time t-1, denotes the backward hidden state at time t+1, denotes the input vector at time t; concatenating the bidirectional hidden states output the prediction probability of the i-th sample through the fully connected layer : ； wherein, is a Sigmoid activation function, and are learnable parameters; S43. The data not screened out is converted into a feature vector, input into the trained intelligent agent, and the prediction probability of each sample is obtained ; a classification threshold is set , it is determined as non-sensitive data, and if , it is determined as unidentified data ;​ The method comprises: S51. Collect sensitive data detection field expertise, from which to extract entities, entity relationship to build a set of triples wherein , is an entity, is a relationship; based on the importance of the rules and the confidence of the relationship, assign a weight to each triple , the calculation formula is: ； in, The normalized frequency of triples in the rule base. For relationship confidence level These are the weighting coefficients; S52. On the basis of a security domain general large model, add a domain adaptation layer to realize the focused adaptation of the large model to the sensitive data detection task through parameter adjustment; insert an adapter module in the large model, and the output of the adapter module is: ； wherein, is the original output feature of the large model, , is the dimension reduction coefficient, is the feature dimension, is the activation function, and is the bias term; Collecting sensitive data and non-sensitive data, and professional knowledge in the field of sensitive data detection, constructing fine-tuning sample set wherein is the feature vector of the th sample, is the label, 0 represents non-sensitive data, and 1 represents sensitive data; Train the model with the fine-tuning sample set, and the loss function is: ； ； ； in, For classification cross-entropy loss; Knowledge-guided loss; As the first The model predicts the sensitivity probability for each sample; The total number of samples; An embedding vector for an entity or relation; These are the weighting coefficients. Denotes the set of all triples. Represents a single triple The weight, Let be the predicted probability of the i-th sample.

2. The method of claim 1, wherein: The method specifically comprises the following steps: S1. Data acquisition and conversion: in the data transmission process, copy and backup the data stream in real time by flow mirroring to generate an independent mirror copy; Based on the mirror copy, use packet capture technology to obtain the data stream without interfering with the original business transmission; analyze and format-standardize the captured data stream, and uniformly process structured, unstructured and semi-structured data into an analyzable text format; S2. Rule base construction: collect various sensitive data detection rules, input the collected rules into a general reasoning large model, and use the semantic understanding and reasoning ability of the large model to optimize, integrate and correlation analyze the rules to construct a rule base; S3. Use the constructed rule base to quickly screen the converted data, and screen out sensitive data; directly mark the screened sensitive data as sensitive data; For data that is not screened out, proceed to the next step; S4. For data that is not screened out by the rule base, construct an intelligent agent, and train and learn based on non-sensitive data rules and unidentified data rules; Input the data that is not screened out by the rule base into the intelligent agent, and further divide it into non-sensitive data and unidentified data by the intelligent agent; S5. Select a general large model in the security domain, add an adapter, and fine-tune the large model using sensitive data detection domain professional knowledge; Input the unidentified data in S4 into the fine-tuned large model, detect it, output the judgment result of non-sensitive data and sensitive data, and update the rule base in a timely manner according to the output result of the large model.

3. The method of claim 2, wherein: The S1 comprises: S11. Deploy flow mirroring at a network key node of data transmission to copy the full amount of data stream in the transmission in real time, generate a mirror copy independent of the original transmission link, complete backup storage synchronously, and ensure that the original data transmission process is not disturbed. S12. In the isolated environment where the mirror copy is located, the mirror data stream is captured in full by a packet capture tool to obtain complete transmission data content; the captured data includes multi-domain types of government data, medical data, financial data, educational data, enterprise data, and personal data; S13. For the captured data, structured, unstructured, and semi-structured types are distinguished, and through parsing, extraction, and encoding conversion processing, they are uniformly converted into a text format that can be directly used for analysis.

4. The method of claim 2, wherein: The S2 includes: S21. Collect multi-dimensional sensitive data detection rules, including regular sensitive data format rules, sensitive keyword rules, and specific pattern rules, and store them according to rule types; S22. Normalize the collected rules to remove duplicate rules, correct logical conflict rules, and convert natural language description rules into structured expressions to ensure uniform rule format and enable large model recognition; S23. Input the pre-processed rules into a general reasoning large model, analyze the logical association between the rules through the semantic understanding ability of the large model, and perform optimization operations such as merging redundant items, supplementing missing conditions, and expanding applicable scenarios on the rules; based on the optimization results output by the large model, integrate the rules across types, establish hierarchical associations between the rules, and form a rule network that covers multiple scenarios and multiple types; S24. Store the integrated rule set as a structured rule library, verify the completeness and accuracy of the rule library through a test data set, and finally form a rule library for sensitive data detection.

5. The method of claim 2, wherein: The S3 includes: S31. Convert each rule in the rule base into a feature vector where is the feature dimension; for regular expression pattern features, extract regular expression patterns; for keyword rules, use word embeddings to generate semantic vectors; and for specific pattern rules, convert to Boolean feature vectors through logical expressions;​ S32. The converted text data is segmented to generate a word sequence ; a feature vector of each word is extracted , and a data matrix is constructed , n is the number of words; S33. For each rule's feature vector , compute its cosine similarity with the feature vectors of the words in the data matrix ​ ； Setting a similarity threshold , if , then determining that the word matches the rule ; S34. For words that match to a rule, combine rule weight and match strength to compute sensitivity score ; determined by large model training, based on cosine similarity normalization; S35. Set sensitivity threshold When the words belong to sensitive data; for the continuously matched words, a sliding window mechanism is used to calculate the cumulative score: ； wherein, is the window length, starting from the word of the word sequence, covering the to the word, if then the data within the whole window is marked as sensitive; the rest of the data is treated as non-sensitive data as input for the next step.

6. The method of claim 2, wherein: The S4 includes: S44. Randomly extract part of the classification results for manual verification, calculate the precision and recall rate , true positive, false positive, false negative, if or is lower than the preset threshold, use the error sample to expand the training set, retrain the intelligent agent to optimize the classification performance.

7. A sensitive data efficient detection system, characterized by, The system includes a module for executing a sensitive data efficient detection method according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement a sensitive data efficient detection method according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement a sensitive data efficient detection method according to any one of claims 1 to 6.

Images