Full-flow data desensitization method and system for sensitive data

Abstract

The invention belongs to the technical field of data desensitization. A full-flow data desensitization method for sensitive data comprises the following steps: acquiring all full-flow data entering and exiting a system, and performing preliminary data cleaning to obtain cleaned full-flow data; performing sensitivity identification on the cleaned full-flow data by using a regular expression or a machine learning model to obtain full-flow sensitive data; performing serialized format conversion on the full-flow sensitive data, and establishing an index structure to obtain data to be desensitized; desensitizing the to-be-desensitized data by using a preset desensitization strategy to obtain desensitized data; carrying out distributed storage on the desensitized data to obtain stored desensitized data; and querying and analyzing the stored desensitized data by using a query and analysis tool to obtain a query and analysis result for business decision. According to the method, real-time identification, desensitization processing and serialization storage are carried out on the full-flow data, and the risk that sensitive information is exposed can be effectively avoided.

Description

Technical Field

[0001] The present application belongs to the technical field of data desensitization, and in particular, relates to a full-flow data desensitization method and system for sensitive data. Background Art

[0002] In the current era of big data, with the explosive growth of data volume and the increasing demand for data analysis in various industries, enterprises and institutions need to process a large amount of sensitive information. This information includes personally identifiable information (PII), financial data, medical records, etc., and its privacy and security must be ensured when processing this data. However, traditional data protection methods, such as static encryption and simple access control, often cannot effectively deal with dynamic and complex security threats. For example, unauthorized access, data leakage, and abuse by insiders are all common security risks. In addition, existing systems often lack sufficient flexibility and scalability when processing full-flow data, and it is difficult to adapt to the efficient processing needs in large-scale data environments.

[0003] To solve these problems, companies urgently need a new solution that can provide strong data privacy protection without affecting data analysis and usage efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a full-flow data desensitization method and system for sensitive data in response to the above technical problems.

[0005] In a first aspect, the present application provides a full-flow data desensitization method for sensitive data, the method comprising: Obtaining all full flow data entering and exiting the system, and performing preliminary data cleaning to obtain the cleaned full flow data; Using regular expressions or machine learning models to perform sensitivity identification on the cleaned full flow data to obtain full flow sensitive data; The full-flow sensitive data is converted into a serialized format, and an index structure is established to obtain the data to be desensitized; Desensitizing the data to be desensitized using a pre-set desensitization strategy to obtain desensitized data; Distributed storage is performed on the desensitized data to obtain stored desensitized data; The stored desensitized data is queried and analyzed using query and analysis tools to obtain query and analysis results for business decision-making.

[0006] In some practicable methods, the step of obtaining all the full flow data entering and leaving the system and performing preliminary data cleaning to obtain the cleaned full flow data includes: Using a data capture tool, the network traffic data entering and leaving the system is monitored and collected to obtain the full traffic data; The full flow data is cleaned to obtain the cleaned full flow data.

[0007] In some practicable methods, the step of using a regular expression or a machine learning model to identify the sensitivity of the cleaned full flow data to obtain full flow sensitive data includes: Get sensitive data types; Using regular expressions or machine learning models, for defined sensitive data types, pattern matching is performed in the cleaned full flow data to obtain the labeled full flow data; The full flow sensitive data is obtained according to the marked full flow data.

[0008] In some practicable methods, the step of converting the full flow sensitive data into a serialized format and establishing an index structure to obtain the data to be desensitized includes: Serializing the full-flow sensitive data and converting it into a standardized format suitable for storage and retrieval to obtain structured data; For the structured data, an index structure is established to obtain the data to be desensitized.

[0009] In some practicable methods, the step of desensitizing the data to be desensitized by using a preset desensitization strategy to obtain desensitized data includes: Obtaining the type and sensitivity of the data to be desensitized; In the desensitization strategy library, a desensitization strategy is selected according to the type and sensitivity of the data to be desensitized to obtain a target desensitization strategy; The target desensitization strategy is used to desensitize the data to be desensitized to obtain the desensitized data.

[0010] In some practicable methods, the step of distributing and storing the desensitized data to obtain the stored desensitized data includes: Based on the hash value or primary key of the data, the desensitized data is distributedly stored to obtain multiple sharded desensitized data; The multiple sharded desensitized data are dispersedly stored on different nodes of the distributed storage system to obtain stored desensitized data.

[0011] In some practicable manners, the step of using a query and analysis tool to query and analyze the stored desensitized data to obtain query and analysis results for business decision-making includes: Build queries and analyze permissions; According to the query and analysis authority, query and analyze the stored desensitized data using query and analysis tools to obtain query and analysis results for business decision-making; Based on the query and analysis results used for business decision-making, industry extension associations are performed to obtain extended results.

[0012] In a second aspect, the present application provides a full-flow data desensitization system for sensitive data, applying the aforementioned full-flow data desensitization method for sensitive data, the system comprising: An acquisition unit, used to acquire all the full flow data entering and leaving the system, and perform preliminary data cleaning to obtain the cleaned full flow data; An identification unit, used to perform sensitivity identification on the cleaned full flow data using a regular expression or a machine learning model to obtain full flow sensitive data; A processing unit, used to convert the full-flow sensitive data into a serialized format and establish an index structure to obtain the data to be desensitized; A desensitizing unit, used to desensitize the data to be desensitized by using a pre-set desensitization strategy to obtain desensitized data; A storage unit, used for performing distributed storage on the desensitized data to obtain stored desensitized data; The query and analysis unit is used to query and analyze the stored desensitized data using query and analysis tools to obtain query and analysis results for business decision-making.

[0013] In a third aspect, the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned full-flow data desensitization method for sensitive data.

[0014] In a fourth aspect, the present application provides a computer program, which, when executed by a processor, implements the steps of the aforementioned full-flow data desensitization method for sensitive data.

[0015] Beneficial effect: the present application provides a method for desensitizing the full-flow data for sensitive data, obtains all the full-flow data entering and leaving the system, and performs preliminary data cleaning to obtain the cleaned full-flow data; uses regular expressions or machine learning models to identify the sensitivity of the cleaned full-flow data to obtain full-flow sensitive data; converts the full-flow sensitive data into a serialized format, and establishes an index structure to obtain the data to be desensitized; uses a pre-set desensitization strategy to desensitize the data to be desensitized to obtain desensitized data; performs distributed storage on the desensitized data to obtain stored desensitized data; uses query and analysis tools to query and analyze the stored desensitized data to obtain query and analysis results for business decision-making. The above method realizes real-time identification, desensitization processing and serialized storage of full-flow data, which can effectively avoid the risk of sensitive information being exposed during use and transmission. In addition, the architecture design of distributed storage and efficient indexing is adopted to ensure that high-performance query and data processing capabilities can still be provided in a large-scale data environment. This solution provides enterprises and institutions with a powerful tool to meet the challenges of modern data security and privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 The present invention is a flowchart of a method for desensitizing full-flow data for sensitive data in one embodiment. DETAILED DESCRIPTION

[0018] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all couplings of one or more related listed items.

[0020] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0021] Some terms involved in this application are explained below to facilitate understanding of this application: Regular expressions are a powerful text processing tool that can identify and extract strings that match specific patterns.

[0022] Long Short-Term Memory (LSTM) is a special recurrent neural network (RNN) architecture that can remember the previous and next information of long sentences, solving the problem of RNN (when the time interval is large, the network will forget the previous information, resulting in the gradient disappearance problem and the formation of long-term dependency problems) and avoiding long-term dependency.

[0023] Precision, recall, and F1 score are metrics used to evaluate the performance of classification models.

[0024] like Figure 1 As shown, in the first aspect, the present application provides a full-flow data desensitization method for sensitive data, the method comprising: S100, obtaining all the full flow data entering and leaving the system, and performing preliminary data cleaning to obtain the cleaned full flow data.

[0025] Specifically, obtaining the cleaned full flow data includes the following steps: S101, using a data capture tool to monitor and collect network traffic data entering and leaving the system to obtain the full traffic data.

[0026] S102, performing data cleaning on the full flow data to obtain the cleaned full flow data.

[0027] Specifically, deploy network sniffers or other data capture tools on key network nodes, such as servers, firewalls, routers, etc., to monitor and record all packets passing through. Use network sniffers or other data capture tools to monitor and collect all network traffic data entering and leaving the system in real time. These tools can capture various network activity data including HTTP requests, database queries, file transfers, etc., so as to comprehensively cover all traffic of the system. Exemplarily, use tools such as Wireshark, tcpdump, etc. to capture packets passing through the network interface. These tools can capture all data in the network, including but not limited to HTTP requests, database queries, API calls, etc.

[0028] Among them, the captured data is stored in the original format. Next, the data that is valuable for business analysis is screened out from the captured full-flow data, such as data from specific ports, data from specific protocols, etc. The collected full-flow data is preliminarily processed to filter out invalid data packets and noise data. Specific operations include removing duplicate data packets, eliminating incomplete or damaged data streams, and filtering out irrelevant traffic to ensure that the data processed subsequently has sufficient accuracy and reliability. In addition, the cleaned data can be integrated into a consistent format to facilitate subsequent analysis and processing. Exemplarily, the data is converted into a specific format (such as JSON, CSV) or imported into a database.

[0029] S200, using regular expressions or machine learning models to perform sensitivity identification on the cleaned full flow data to obtain full flow sensitive data.

[0030] Specifically, obtaining full flow sensitive data includes the following steps: S201, obtaining sensitive data types.

[0031] First, define what types of data are sensitive data, which can be pre-set based on relevant data protection rules, such as personally identifiable information (PII), financial data, health records, etc. Next, classify the data and clarify the characteristics and identification criteria of each type of data.

[0032] S202, using regular expressions or machine learning models, for defined sensitive data types, perform pattern matching in the cleaned full flow data to obtain the labeled full flow data.

[0033] Among them, regular expressions are used to match sensitive data in specific formats, such as phone numbers, credit card numbers, email addresses, etc.

[0034] For example, a regular expression is used to replace all non-numeric characters. This cleans and standardizes the phone number data, removing any non-numeric characters and leaving only numbers.

[0035] 1 UPDATE customers 2 SET phone_number = 3 REGEXP_REPLACE(phone_number, '[^0-9]', '', 'g') 4 WHERE phone_number ~ '[^0-9]'; Use the regular expression [^0-9] to match any non-digit character and replace it with an empty string ''. 'g' means global replacement, that is, all matching places will be replaced.

[0036] phone_number ~ '[^0-9]': This is a regular expression match condition that matches any value in the phone_number column that contains any non-numeric characters.

[0037] Use a regular expression to match phone numbers that contain 10 numeric characters and format them as xxx-xxx-xxxx: 1 UPDATE customers 2 SET phone_number = 3 REGEXP_REPLACE(phone_number, '([0-9]{3})([0-9]{3})([0-9]{4})', 4 '\1-\2-\3') 5 WHERE phone_number ~'^[0-9]{10}$'; Use regular expressions ([0-9]{3})([0-9]{3})([0-9]{4}) to match specific patterns in phone numbers Replace the pattern with \1-\2-\3, where \1, \2, and \3 are the three groups of numbers captured in the regular expression phone_number ~ '^[0-9]{10}$': This is a regular expression matching condition that matches the value of the phone_number column that contains only 10 numeric characters.

[0038] Filter sensitive data 1 SELECT 2 CONCAT(firstname, ' ', UPPER(SUBSTRING(lastname, 1, 1)), '.') 3 AS fullname, 4 emails, 5 (SELECT CONCAT(' - -', 6 RIGHT(phone_number, 4)) AS masked_phone_number) 7 FROM customers; The SUBSTRING() function is used to extract the first character of lastname. The first 1 is the starting position parameter, which specifies the starting position of the substring to be extracted. In this case, it is the first character in the lastname string.

[0039] The second 1 is the length parameter, which specifies the length of the substring to be extracted. Here, it means to extract only one character.

[0040] The query results will be - - followed by the last four digits of the original phone_number value.

[0041] Password encryption: Set the "password_salt" field to a randomly generated string that is obtained by converting a random number into text format, performing MD5 encryption, and then intercepting the first 16 characters.

[0042] In addition, a machine learning model can also be used to perform pattern matching in the cleaned full traffic data for defined sensitive data types. Exemplarily, the machine learning model can be a long short-term memory network.

[0043] Using long short-term memory networks to identify data may include the following steps: Collect a large amount of text data that contains known sensitive data instances, and annotate the sensitive data instances by experts or automated tools. These annotations will serve as labels for training models.

[0044] Preprocess and vectorize large amounts of text data. Text preprocessing: includes cleaning text, removing irrelevant characters, standardization (such as unifying date and time formats), word segmentation, etc. Vectorization: converting text into a format that can be processed by machine learning models, such as word embeddings or character-level representations.

[0045] An initial LSTM network is constructed, and the preprocessed and vectorized data is input into the initial LSTM network for training to obtain a LSTM network. The labeled sensitive data is used as the training target, and the weights of the LSTM are adjusted through the back propagation algorithm to minimize the difference between the predicted and actual labels.

[0046] The performance of the long short-term memory network is tested on an independent validation set, and its accuracy, recall rate, F1 score and other indicators are evaluated to obtain the evaluation results.

[0047] Adjust the parameters of the long short-term memory network according to the evaluation results. For example, adjust the learning rate, number of layers, number of hidden units, etc. of the long short-term memory network to improve the model performance.

[0048] Apply the trained long short-term memory network to perform pattern matching in the full traffic data to identify sensitive data in real-time or batch data streams. The long short-term memory network can also be optimized again according to the continuous monitoring of its performance.

[0049] Exemplarily, construct a Bi-LSTM: It is composed of a forward LSTM and a backward LSTM. One LSTM network learns from the beginning of a sentence, and the other LSTM network learns from the end of a sentence.

[0050] CRF (Conditional Random Field): It is a discriminative model used to solve the labeling bias problem. It models using P(Y|X) and is globally normalized. Application fields: part-of-speech tagging, word segmentation, named entity recognition, etc.

[0051] Take named entities as an example: Instance: I go to Beijing word_alphabet: {o: "我", 1: "去", 2: "北", 3: "京", 4: <unk> ,5: <cpad>} label_alphabet: {o;b,1:m,2:e,3:s,4:o,5: <start> ,6: <end>} Among them, b indicates the entity start, m indicates the entity middle, e indicates the entity end, s indicates a single entity, and o indicates a non-entity other. 5: <start>and 6: <end>Indicates the start and end of a sequence word_nums:6 label_nums:7 There are 7 labels in total, so there are 7 possibilities for each word to be predicted.

[0052] Emit score: the score from word_index to label_index word_index:0 1 2 3 label_index:5 4 4 0 2 6 First: From word_index 1 → label_index 4 score emit[1] [4] When the previous label is 0, the current label is predicted to be m and e Using CRF can avoid being predicted as s and o.

[0053] Transition score (T): the score from the previous label to the current label word_index :0 1 2 3 label_index:5 4→4 0 2 6 Second: The score from label_index 4 to label_index 4 is recorded as T[4][4] The scores from word_ndex=1 to label_index=4 are obtained through the first and second, where emit[1][4]+T[4][4] In order to take the whole situation into consideration, CRF also adds the previous score to the current score, thus expressing the overall score of the predicted fishing and hunting: score[1][4]=score[0][4]+emit[1][4]+T[4][4] The overall formula is:

[0054] Among them, x represents the word_index sequence and y represents the predicted label_index sequence.

[0055] S203, obtaining the full flow sensitive data according to the marked full flow data.

[0056] Specifically, sensitive data is extracted from the full flow data according to the information marked in step S202. The identified full flow sensitive data is marked and associated with the original full flow data for subsequent processing and auditing. The marked full flow sensitive data will be used as output for subsequent serialization, desensitization and storage operations. This ensures the correct processing and tracking of sensitive data in subsequent processes.

[0057] S300, converting the full flow sensitive data into a serialized format, and establishing an index structure to obtain the data to be desensitized.

[0058] Specifically, obtaining the data to be desensitized includes the following steps: S301, serialize the full-flow sensitive data and convert it into a standardized format suitable for storage and retrieval to obtain structured data.

[0059] S302: Establish an index structure for the structured data to obtain the data to be desensitized.

[0060] Specifically, the identified full-flow sensitive data is serialized and converted into a standardized format suitable for storage and retrieval, such as MD5 format after conversion from JSON format. This conversion not only helps the structured storage of data, but also makes the data more compatible and readable during transmission and processing. Through this serialization, the originally chaotic and disordered sensitive data is systematically represented as structured data.

[0061] Based on the serialized structured data, efficient index structures such as B-tree and inverted index are established. These index structures are designed to speed up subsequent data retrieval, query and desensitization processing. By building a powerful index mechanism, specific sensitive data records can be quickly located and processed, thereby improving overall processing efficiency.

[0062] The full flow of sensitive data after serialization and indexing is output as the data to be desensitized. These desensitized data are ready for subsequent desensitization strategy implementation to ensure the security and operability of the data in the further processing process.

[0063] S400, using a pre-set desensitization strategy, desensitizes the data to be desensitized to obtain desensitized data.

[0064] Specifically, obtaining the desensitized data includes the following steps: S401, obtaining the type and sensitivity of the data to be desensitized; S402, in a desensitization strategy library, according to the type and sensitivity of the data to be desensitized, a desensitization strategy is selected to obtain a target desensitization strategy; S403: Desensitize the data to be desensitized by using a target desensitization strategy to obtain the desensitized data.

[0065] Specifically, identify the type of data to be desensitized, such as personally identifiable information (PII), financial data, health records, etc. Assess the sensitivity of the data and determine the sensitivity level of the data. This may involve factors such as compliance requirements and potential risks of data leakage. In other words, the types of sensitive information contained in the data set can be identified through automated tools or manual review, and their sensitivity can be rated based on the potential impact of data leakage. Extract metadata from the data to be desensitized, such as the source of the data, the owner of the data, the purpose of the data use, etc. This information will help to select the appropriate desensitization strategy later. In addition, the retention of metadata enables partial or full restoration of desensitized data in special circumstances to ensure business continuity and data consistency.

[0066] It should be noted that a desensitization strategy library can be established, which contains multiple desensitization strategies, each targeting a specific type of data and sensitivity level. Desensitization strategies (methods) are all conventional methods. Next, according to the type and sensitivity of the data to be desensitized, the most appropriate desensitization strategy is selected from the strategy library. For example, for highly sensitive data, a strong encryption method may be selected; while for low-sensitivity data, a simple mask or replacement strategy may be used. Next, according to the selected desensitization strategy, the sensitive information in the data set is processed. All operations and results in the desensitization process can be recorded, including the application of desensitization strategies, data comparison before and after desensitization, etc., to facilitate subsequent audits and compliance checks.

[0067] S500, performing distributed storage on the desensitized data to obtain stored desensitized data.

[0068] Specifically, obtaining the stored desensitized data includes the following steps: S501, based on the hash value or primary key of the data, the desensitized data is distributedly stored to obtain multiple shards of desensitized data.

[0069] S502, dispersively storing the plurality of sharded desensitized data on different nodes of the distributed storage system to obtain stored desensitized data.

[0070] Specifically, according to the scale of the desensitized data and business needs, select a suitable distributed storage engine, such as Hadoop HDFS, Cassandra or Elasticsearch. These storage engines have high scalability and high availability, and can support the storage and fast access of large-scale data. The selected storage engine should be able to fully adapt to the reading and writing needs of the data, and provide good load balancing and fault tolerance to ensure the stability and reliability of data storage. Next, after selecting the storage engine, the desensitized data is sharded and divided into multiple shards according to certain rules (such as based on the hash value or primary key of the data). These shards will be stored in different nodes of the distributed storage system. The application of sharding technology helps to improve the reading and writing performance of data, while reducing the load of a single node and preventing the occurrence of system bottlenecks. The design of the distributed storage system should ensure the reasonable distribution of data shards to achieve efficient resource utilization and data access. After the desensitized data is stored in the distributed system (existing distributed system), the data is backed up and managed. The backup operation can be set on different nodes to ensure redundant storage of data to cope with potential hardware failures or data loss. At the same time, an effective management mechanism is implemented to monitor, maintain and optimize the stored data to ensure high availability and security of the data.

[0071] S600, using query and analysis tools, query and analyze the stored desensitized data to obtain query and analysis results for business decision-making.

[0072] Specifically, obtaining query and analysis results for business decision-making includes the following steps: S601, build query and analysis permissions; S602, querying and analyzing the stored desensitized data using query and analysis tools according to the query and analysis authority to obtain query and analysis results for business decision-making; S603, performing association extension according to the query and analysis results used for business decision-making to obtain an extension result.

[0073] It should be noted that query and analysis permissions should be defined and assigned according to the business needs and data security requirements of different users. Different user groups may have different permission levels. For example, data administrators can access all data, while ordinary business users may only have access to a subset of data that has been desensitized. The definition of permissions should be based on factors such as data sensitivity, user responsibilities, and compliance requirements to ensure that access and use of sensitive data complies with company policies and laws and regulations.

[0074] Next, according to business needs, select appropriate query and analysis tools (such as Spark, Presto, etc.) to process the stored masked data. These tools should be able to efficiently process large-scale data and integrate seamlessly with the permission control mechanism. During the tool configuration process, it is necessary to ensure that the query tool can correctly implement user permission settings and can quickly respond to user query requests. Then, in the configured query tool, users perform query and analysis operations according to the assigned permissions. The query process may include operations such as filtering, aggregating, sorting, and classifying masked data to extract information useful for business decisions. For example, by querying historical transaction data, customer behavior trends can be analyzed, or by aggregating sales data, key business indicators can be calculated. Ultimately, the system will generate query and analysis results for business decisions, providing data support for subsequent decision-making processes.

[0075] Finally, after obtaining the query and analysis results for business decision-making, these results are further interpreted and associated with extended analysis. For example, if the query results show that the sales of a certain product are growing rapidly, the relevant market factors, customer feedback, and competition can be further analyzed to explore the drivers of growth. Through this extended analysis, enterprises can deeply understand the complex relationships behind the data and identify potential opportunities and risks. Based on the results of extended analysis, more detailed and in-depth business insights are generated. Extended results can be used to adjust market strategies, optimize operational processes, or set new business goals. Enterprises can apply these extended results to actual business decisions to enhance competitiveness and market responsiveness. Ultimately, these extended results enrich the initial query and analysis conclusions.

[0076] It should be noted that, in step S603, the following steps are also included: Construct an industry extension association model, the architecture of which is a conventional model architecture.

[0077] The query and analysis results used for business decision-making are used as training data to train the industry extension association model. After a large amount of training, a trained industry extension association model is obtained; The query and analysis results currently used for business decision-making are used as input data and input into the trained industry extension association model to obtain the extension results. This extension result can enrich the initial query and analysis conclusions and provide a more comprehensive reference for the company's strategic formulation, thus improving the user experience.

[0078] In a second aspect, the present application provides a full-flow data desensitization system for sensitive data, applying the aforementioned full-flow data desensitization method for sensitive data, the system comprising: An acquisition unit, used to acquire all the full flow data entering and leaving the system, and perform preliminary data cleaning to obtain the cleaned full flow data; An identification unit, used to perform sensitivity identification on the cleaned full flow data using a regular expression or a machine learning model to obtain full flow sensitive data; A processing unit, used to convert the full-flow sensitive data into a serialized format and establish an index structure to obtain the data to be desensitized; A desensitizing unit, used to desensitize the data to be desensitized by using a pre-set desensitization strategy to obtain desensitized data; A storage unit, used for performing distributed storage on the desensitized data to obtain stored desensitized data; The query and analysis unit is used to query and analyze the stored desensitized data using query and analysis tools to obtain query and analysis results for business decision-making.

[0079] In a third aspect, the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned full-flow data desensitization method for sensitive data.

[0080] In a fourth aspect, the present application provides a computer program, which, when executed by a processor, implements the steps of the aforementioned full-flow data desensitization method for sensitive data.

[0081] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0082] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0083] The protection scope of the present disclosure is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.< / end> < / start> < / end> < / start> < / cpad> < / unk>

Claims

1. A full-flow data desensitization method for sensitive data, characterized in that: Methods include: Obtaining all full flow data entering and exiting the system, and performing preliminary data cleaning to obtain the cleaned full flow data; Using regular expressions or machine learning models to perform sensitivity identification on the cleaned full flow data to obtain full flow sensitive data; The full-flow sensitive data is converted into a serialized format, and an index structure is established to obtain the data to be desensitized; Desensitizing the data to be desensitized using a pre-set desensitization strategy to obtain desensitized data; Distributed storage is performed on the desensitized data to obtain stored desensitized data; The stored desensitized data is queried and analyzed using query and analysis tools to obtain query and analysis results for business decision-making.

2. The full-flow data desensitization method for sensitive data according to claim 1 is characterized in that: The step of obtaining all the full flow data entering and leaving the system and performing preliminary data cleaning to obtain the cleaned full flow data includes: Using a data capture tool, the network traffic data entering and leaving the system is monitored and collected to obtain the full traffic data; The full flow data is cleaned to obtain the cleaned full flow data.

3. The full-flow data desensitization method for sensitive data according to claim 1 is characterized in that: The step of using a regular expression or a machine learning model to identify the sensitivity of the cleaned full flow data to obtain full flow sensitive data includes: Get sensitive data types; Using regular expressions or machine learning models, for defined sensitive data types, pattern matching is performed in the cleaned full flow data to obtain the labeled full flow data; The full flow sensitive data is obtained according to the marked full flow data.

4. The full-flow data desensitization method for sensitive data according to claim 1 is characterized in that: The step of converting the full flow sensitive data into a serialized format and establishing an index structure to obtain the data to be desensitized includes: Serializing the full-flow sensitive data and converting it into a standardized format suitable for storage and retrieval to obtain structured data; For the structured data, an index structure is established to obtain the data to be desensitized.

5. The full-flow data desensitization method for sensitive data according to claim 1 is characterized in that: The step of desensitizing the data to be desensitized by using a preset desensitization strategy to obtain desensitized data includes: Obtaining the type and sensitivity of the data to be desensitized; In the desensitization strategy library, a desensitization strategy is selected according to the type and sensitivity of the data to be desensitized to obtain a target desensitization strategy; The target desensitization strategy is used to desensitize the data to be desensitized to obtain the desensitized data.

6. The full-flow data desensitization method for sensitive data according to claim 1 is characterized in that: The step of distributing and storing the desensitized data to obtain the stored desensitized data includes: Based on the hash value or primary key of the data, the desensitized data is distributedly stored to obtain multiple sharded desensitized data; The multiple sharded desensitized data are dispersedly stored on different nodes of the distributed storage system to obtain stored desensitized data.

7. The full-flow data desensitization method for sensitive data according to claim 1 is characterized in that: The step of using a query and analysis tool to query and analyze the stored desensitized data to obtain query and analysis results for business decision-making includes: Build queries and analyze permissions; According to the query and analysis authority, query and analyze the stored desensitized data using query and analysis tools to obtain query and analysis results for business decision-making; Based on the query and analysis results used for business decision-making, industry extension associations are performed to obtain extended results.

8. A full-flow data desensitization system for sensitive data, characterized in that: The full-flow data desensitization method for sensitive data described in any one of claims 1 to 7 is applied, and the system comprises: An acquisition unit, used to acquire all the full flow data entering and leaving the system, and perform preliminary data cleaning to obtain the cleaned full flow data; An identification unit, used to perform sensitivity identification on the cleaned full flow data using a regular expression or a machine learning model to obtain full flow sensitive data; A processing unit, used to convert the full-flow sensitive data into a serialized format and establish an index structure to obtain the data to be desensitized; A desensitizing unit, used to desensitize the data to be desensitized by using a pre-set desensitization strategy to obtain desensitized data; A storage unit, used for performing distributed storage on the desensitized data to obtain stored desensitized data; The query and analysis unit is used to query and analyze the stored desensitized data using query and analysis tools to obtain query and analysis results for business decision-making.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the full-flow data desensitization method for sensitive data described in any one of claims 1 to 7 are implemented.

10. A computer program, characterized in that When the computer program is executed by a processor, the steps of the full-flow data desensitization method for sensitive data described in any one of claims 1 to 7 are implemented.

Images