Data desensitization method
By defining and customizing the desensitization algorithm in the data warehouse and combining with the logical execution plan transformation of the Spark SQL engine, personalized data desensitization is achieved, solving the problem of lack of personalization and flexibility in traditional technologies, and improving the quality of data analysis .
Patent Information
- Application Number
- CN202411176753.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional data desensitization technology lacks personalization and cannot adapt to the needs of different users. Too strict desensitization rules may lead to information loss, affecting the quality of data analysis and modeling, lack of flexibility, and it is difficult to adjust the desensitization strategy according to different analytical purposes. .
By defining the default desensitization algorithm for each field in the configuration file, and customizing the configuration according to the user's role, permissions and data desensitization needs, a personalized desensitization algorithm file is generated, combined with the logical execution plan transformation strategy of the Spark SQL query engine , dynamically match the desensitization algorithm to achieve data desensitization.
It realizes personalized data desensitization, retains the value of data analysis as much as possible, improves the quality of data analysis and modeling, supports a variety of scenarios and user needs, and provides high flexibility.
Smart Images

Figure CN119089495A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method for desensitizing data. Background Art
[0002] Data desensitization technology is a data processing technology that uses data deformation to process sensitive data to prevent privacy data leakage.
[0003] Traditional data desensitization technology usually adopts unified desensitization rules, such as data replacement, hashing or encryption, which has some limitations, such as: (1) lack of personalization, unified desensitization rules cannot adapt to the needs of different users; (2) information loss, overly strict desensitization rules may lead to the loss of valuable information and affect the quality of data analysis and modeling; (3) lack of flexibility, it is difficult to adjust the desensitization strategy according to different analysis purposes; these limitations will lead to the inability to meet the different data needs of different users and poor practicality.
[0004] To this end, the present invention provides a method for data desensitization. Summary of the invention
[0005] Based on this, it is necessary to provide a data desensitization method to address the above technical issues.
[0006] According to one aspect of the present invention, a method for data desensitization is provided, which is applied to a data warehouse, wherein the data warehouse is built using Hadoop and Hive technologies, and Spark SQL is used as a data query engine for the data warehouse, the method comprising: defining a default desensitization algorithm for each field in a configuration file based on the needs of different users and different scenarios; customizing the desensitization rules of the corresponding default desensitization algorithm in the configuration file according to the user's role, authority and data desensitization needs to obtain a personalized desensitization algorithm; generating a desensitization algorithm file based on the default desensitization algorithm and the personalized desensitization algorithm; obtaining the user's SQL query request, parsing and processing the target data table to be desensitized based on the logic execution plan transformation strategy, capturing the table content information, calling the RestAPI of the desensitization algorithm file according to the table content information, obtaining the corresponding default desensitization algorithm or the personalized desensitization algorithm, desensitizing the table content information according to the default desensitization algorithm or the personalized desensitization algorithm, and obtaining desensitized data.
[0007] In some optional implementation methods of some embodiments, the desensitizing algorithm file is generated based on the default desensitizing algorithm and the personalized desensitizing algorithm, specifically including: writing the implementation logic process of each default desensitizing algorithm and each personalized desensitizing algorithm into an independent Java class file or Scala class file, and compiling all Java class files or Scala class files into a JAR file, and loading the JAR file into the lib directory of the Spark computing engine for use when the Spark SQL query engine is running.
[0008] In some optional implementations of some embodiments, the default desensitization algorithm and the personalized desensitization algorithm both include a rule-based desensitization algorithm or a time-weighted desensitization algorithm.
[0009] In some optional implementations of some embodiments, the logic execution plan transformation strategy specifically includes: compiling the Spark Catalyst code, packaging the compiled Spark Catalyst code, and generating a new Spark Catalyst running package, wherein the new Spark Catalyst running package includes the JAR file required to run the newly added personalized desensitization algorithm, and deploying the new Spark Catalyst running package to the Spark SQL source code environment.
[0010] In some optional implementations of some embodiments, the compiling the Spark Catalyst code specifically includes: setting a rule class for processing a logical plan, which is used to traverse the logical plan tree in its apply method.
[0011] In some optional implementations of some embodiments, the traversal of the logic plan tree specifically includes: during the traversal process, identifying and processing all table nodes and field information thereof contained in the data table to be desensitized.
[0012] In some optional implementations of some embodiments, the method of obtaining the user's SQL query request, parsing the target data table to be desensitized based on the logical execution plan transformation strategy, capturing table content information, calling the RestAPI of the desensitizing algorithm file according to the table content information, obtaining the corresponding default desensitizing algorithm or personalized desensitizing algorithm, and desensitizing the table content information according to the default desensitizing algorithm or personalized desensitizing algorithm to obtain desensitized data, specifically includes: obtaining the user's SQL query request, using the Spark SQL parser to parse the target data table to be desensitized, capturing table content information, generating a logical plan tree object, traversing all table nodes in the plan tree object, screening field information that needs to be desensitized, calling the RestAPI of the desensitizing algorithm file for each field information that needs to be desensitized, obtaining the corresponding default desensitizing algorithm or personalized desensitizing algorithm, and using the default desensitizing algorithm or personalized desensitizing algorithm to desensitize the field information that needs to be desensitized until all field information that needs to be desensitized is desensitized, thereby obtaining the desensitized data.
[0013] In some optional implementations of some embodiments, after obtaining the desensitized data, it also includes: responding to the user's SQL query request based on the desensitized data.
[0014] According to a second aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0015] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the above method when executed by a processor.
[0016] The advantages and beneficial effects of the present invention are:
[0017] 1. Personalized desensitization. The present invention realizes the transformation of the Spark SQL query engine by compiling the Spark Catalyst code. It can dynamically match the corresponding desensitization algorithm according to different scenarios, different users, and different data desensitization requirements, so as to provide customized desensitization effects according to the needs of different users.
[0018] 2. Minimize information loss. The present invention adopts a rule-based desensitization algorithm or a time-weighted desensitization algorithm, which can retain the analytical value of the data as much as possible under the premise of protecting privacy, thereby improving the quality of data analysis and modeling.
[0019] 3. High flexibility. The present invention can adjust the desensitization strategy according to different analysis purposes and data usage scenarios to achieve dynamic desensitization to support a variety of scenarios and user needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 The present invention is a flowchart of a method for data desensitization in an embodiment of the present invention.
[0021] Figure 2 Schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific implementation methods in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0023] Embodiment 1
[0024] Attached Figure 1 A flowchart of a data desensitization method provided for this application.
[0025] Reference Figure 1 A data desensitization method is applied to a data warehouse built by Hadoop and Hive technology, and the data warehouse uses Spark SQL as a data query engine. The method specifically includes the following steps.
[0026] S1. Based on the needs of different users and scenarios, define the default desensitization algorithm for each field in the configuration file.
[0027] In some embodiments, the scenario involved in the present invention is a data usage scenario. The default desensitization algorithm for each field under different users and different data usage scenarios is defined in a configuration file to adapt to the needs of different users and scenarios. The default desensitization algorithm is a rule-based desensitization algorithm or a time-weighted desensitization algorithm.
[0028] S2. According to the user's role, authority and data desensitization requirements, the desensitization rules of the corresponding default desensitization algorithm are customized in the configuration file to obtain a personalized desensitization algorithm.
[0029] In some embodiments, the present invention may also allow users to personalize the relevant default desensitization algorithm in the configuration file according to different roles, permissions and data desensitization requirements. The object of personalized setting is the desensitization rules contained in the default desensitization algorithm to obtain a personalized desensitization algorithm, which also includes a rule-based desensitization algorithm or a time-weighted desensitization algorithm.
[0030] S3. Generate a desensitization algorithm file based on the default desensitization algorithm and the personalized desensitization algorithm.
[0031] In some embodiments, a desensitizing algorithm file is generated based on a default desensitizing algorithm and a personalized desensitizing algorithm, specifically including: writing the implementation logic process of each default desensitizing algorithm and each personalized desensitizing algorithm into an independent Java class file or a Scala class file, and compiling all Java class files or Scala class files into a JAR file, and loading the JAR file into the lib directory of the Spark computing engine for use when the Spark SQL query engine is running.
[0032] In some embodiments, the JAR file includes the implementation logic of a specific default desensitization algorithm or a personalized desensitization algorithm.
[0033] In some embodiments, the present invention adopts a rule-based desensitization algorithm and a time-weighted desensitization algorithm to achieve dynamic adjustment of the desensitization algorithm, wherein the rule-based desensitization algorithm can be used to meet the needs of different scenarios, and the time-weighted desensitization algorithm can completely hide the original time information through weighted conversion, and only retain the difference and relationship information between different data; for example, for marketers, if the data usage scenario is to conduct marketing based on the user's age, a rule-based desensitization algorithm (such as retaining the year, month, and day on the ID card) can be used for desensitization processing to extract desensitized data, so that marketing can be carried out based on the year of birth retained in the desensitized information; if the data usage scenario is to conduct marketing based on the user's region, a rule-based desensitization algorithm (such as retaining the place of origin of the phone number) can be used for desensitization processing. Extract desensitized data, so that marketing can be carried out based on the phone number retained in the desensitized information and the place of attribution. For example, for multi-company federated modeling scenarios, user login data needs to be provided to third-party companies. When third-party companies use user login data for modeling, they need to retain the data difference between the original data, and at the same time cannot accurately reveal the user's actual login time. A time-weighted desensitization algorithm can be used to desensitize the data. Therefore, whether to use a rule-based desensitization algorithm or a time-weighted desensitization algorithm to desensitize the data has the following advantages: it can not only ensure effective data desensitization, but also retain as much data information as possible for necessary non-sensitive operations. The specific desensitization algorithm to be used can be controlled by the administrator according to the data usage scenario.
[0034] In some embodiments, salting is required when using some rule desensitization algorithms so that different users or organizations obtain inconsistent results for the same data, in order to avoid the possibility of database collision. Because the desensitized data retains part of the data information, if the same user applies desensitization of the rules multiple times, it is easy to piece together a relatively complete original data. For this, the administrator needs to perform secondary setting control, that is, first provide rule desensitization algorithms with different rules for desensitization, and different algorithm salts need to be added during the desensitization process.
[0035] In some embodiments, in order to implement rule management by the administrator, the present application may provide a configuration interface for the administrator to configure and manage the rule desensitization algorithm and the time-weighted desensitization algorithm.
[0036] S4. Obtain the user's SQL query request, parse and process the target data table to be desensitized based on the logic execution plan transformation strategy, capture the table content information, call the RestAPI of the desensitization algorithm file according to the table content information, obtain the corresponding default desensitization algorithm or personalized desensitization algorithm, desensitize the table content information according to the default desensitization algorithm or personalized desensitization algorithm, and obtain the desensitized data.
[0037] In some embodiments, in order to achieve dynamic desensitization effect, the present application proposes a logical execution plan transformation strategy for transforming the Spark SQL parser of the Spark SQL query engine so that during the execution of the SQL query request, a rule-based desensitization algorithm or a time-weighted desensitization algorithm can be dynamically applied according to user roles and data usage scenarios.
[0038] In some embodiments, the logic execution plan transformation strategy specifically includes: compiling the Spark Catalyst code, packaging the compiled Spark Catalyst code, and generating a new Spark Catalyst running package, wherein the new Spark Catalyst running package includes the JAR file required to run the newly added personalized desensitization algorithm, and deploying the new Spark Catalyst running package to the Spark SQL source code environment.
[0039] Specifically, recompile and package the modified Spark Catalyst code to generate a new SparkCatalyst runtime package. Ensure that the new Spark Catalyst runtime package contains all necessary dependencies, including the newly added personalized desensitization algorithm implementation. Deploy the new Spark Catalyst runtime package to the production environment or test environment so that desensitization can be performed correctly during actual query execution.
[0040] In some embodiments, compiling the Spark Catalyst code specifically includes: setting a rule class for processing a logical plan, which is used to traverse the logical plan tree in its apply method.
[0041] Specifically, rewrite the Spark Catalyst code and set a custom Rule[LogicalPlan] class to traverse the logical plan tree in its apply method.
[0042] In some embodiments, traversing the logic plan tree specifically includes: during the traversal process, identifying and processing all table nodes and field information contained in the data table to be desensitized.
[0043] In some embodiments, obtaining a default desensitization algorithm or a personalized desensitization algorithm specifically includes: calling the RestAPI of the desensitization algorithm file according to the captured table content information, and returning the corresponding default desensitization algorithm or personalized desensitization algorithm that is hit.
[0044] In some embodiments, the corresponding default desensitization algorithm or personalized desensitization algorithm is dynamically loaded and called through the Java reflection mechanism. It should be noted that these default desensitization algorithms or personalized desensitization algorithms are pre-written and placed in the lib directory.
[0045] In some embodiments, a user's SQL query request is obtained, and based on a logical execution plan transformation strategy, a target data table to be desensitized is parsed and processed, table content information is captured, and according to the table content information, the RestAPI of the desensitizing algorithm file is called to obtain a corresponding default desensitizing algorithm or a personalized desensitizing algorithm, and the table content information is desensitized according to the default desensitizing algorithm or the personalized desensitizing algorithm to obtain desensitized data, specifically including: obtaining a user's SQL query request, using a Spark SQL parser to parse and process the target data table to be desensitized, capturing table content information, generating a logical plan tree object, traversing all table nodes in the plan tree object, screening field information that needs to be desensitized, and for each field information that needs to be desensitized, calling the RestAPI of the desensitizing algorithm file, obtaining a corresponding default desensitizing algorithm or a personalized desensitizing algorithm, and using the default desensitizing algorithm or the personalized desensitizing algorithm to desensitize the field information that needs to be desensitized, until all field information that needs to be desensitized is desensitized, thereby obtaining desensitized data.
[0046] Specifically, obtain the SQL query request submitted by the user, use the Spark SQL parser to parse the target data table to be desensitized, capture the table content information, wherein the table content information includes table name information, field name information, user information and application scenario information, and generate a logical plan tree object, traverse the logical plan tree object, traverse all table nodes in the logical plan tree object, screen the field information that needs to be desensitized, for each field information that needs to be desensitized, call the RestAPI of the desensitizing algorithm file, and obtain the corresponding default desensitizing algorithm or personalized desensitizing algorithm, wherein, for each field information that needs to be desensitized, when calling the corresponding desensitizing algorithm, it is necessary to first determine whether there is a personalized desensitizing algorithm, if there is no personalized desensitizing algorithm, call the default desensitizing algorithm for desensitization, if there is a personalized desensitizing algorithm, call the personalized desensitizing algorithm for desensitization, until all field information that needs to be desensitized is desensitized, and the desensitized data is obtained.
[0047] S5. Respond to the user's SQL query request based on the desensitized data.
[0048] In some embodiments, the desensitized data is returned to the user as a query result to ensure that the query result complies with the desensitization rules, thereby protecting sensitive information.
[0049] Specifically, the data desensitization method provided in this application is further explained.
[0050] 1. Define multiple desensitization algorithms
[0051] (1) Rule-based desensitization algorithm: A rule-based desensitization algorithm is defined for each field in the target data to be desensitized (see Table 1), where the target data to be desensitized is, for example: 41010319*********6, 159********1, No. 5*5, ** Road, Erqi District, Zhengzhou City, Henan Province.
[0052] Table 1 Rule desensitization algorithm
[0053]
[0054]
[0055] Specifically, the operator information, province, city, district information, and whether it is a virtual number in the mobile phone number are extracted and recorded. For example, the operator is China Mobile 01, the province, city, district is Henan Zhengzhou code 410, and whether it is a virtual number, if not, the code is 0.
[0056] Specifically, keyword matching rules are used for address information of whether it is a factory, a school, a company\unit, a business, or a residence. The keyword matching rules for factories and schools are shown in Table 2.
[0057] Table 2 Keyword matching rules for factories and schools
[0058]
[0059] (2) Time-weighted desensitization algorithm:
[0060] The above-mentioned fixed rule desensitization algorithm can provide some flexibility and ensure the desensitization effect of most information. However, for some application scenarios that require the exposure of valid information, such as the need to provide external companies with the time data of customers logging into the App for data analysis and processing, but the accurate time data cannot be leaked, and the data rules and relationships need to be retained to the greatest extent. The time-weighted desensitization algorithm can be used to convert the date and time parameters into timestamps and add random obfuscation codes, where x represents the time and date parameters, such as (2024-07-31, 14:48:00), the timestamp function represents the function of converting the date to a timestamp, and rand represents a random function, the result of which is random data between (0, 1). The formula of the time-weighted desensitization algorithm is as follows: result = timestamp (x) + rand * 1000000, result represents the obfuscated data after desensitization, where rand * 1000000 corresponds to the same set of data and needs to remain the same. Since its data distribution and data fluctuations are consistent with the original data, the relationship between the data is retained when the data is desensitized.
[0061] 2. Modify and optimize the Spark Catalyst code and introduce a desensitization algorithm during the parsing process.
[0062] The desensitization algorithm involved in the present invention will perform dynamic desensitization processing during the execution of the Spark SQL query engine. Therefore, the Spark Catalyst code needs to be recompiled to implement the introduction of the desensitization algorithm into Spark for calculation during the SQL query request parsing process.
[0063] Specifically, the Spark SQL parser parses the target data table to be desensitized to obtain table content information including table name information, field name information, user information and application scenario information. This information is used as an input parameter request, and the RestAPI of the desensitizing algorithm file is called to obtain the desensitizing algorithm of the corresponding information. If the corresponding desensitizing algorithm is not matched, no desensitizing processing is performed. If the corresponding desensitizing algorithm is matched, the field information that needs to be desensitized is desensitized according to the desensitizing algorithm (wherein the default desensitizing algorithm is shown in Table 3 below, and the personalized desensitizing algorithm is shown in Table 4 below) until all the field information that needs to be desensitized is desensitized, and the desensitized data is obtained, and the desensitized data is returned to the user as a query result.
[0064] Table 3 Default desensitization algorithm
[0065]
[0066] Table 4 Personalized desensitization algorithm
[0067]
[0068] Embodiment 2
[0069] This embodiment further provides an electronic device based on the above embodiment 1. Figure 2 , Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0070] like Figure 2 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In RAM 303, various programs and data required for the operation of the electronic device are also stored. The processing device 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0071] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, etc., an output device 307 including, for example, a liquid crystal display (LCD), a speaker, etc., a storage device 308 including, for example, a magnetic tape, a hard disk, etc., and a communication device 309. The communication device 309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 2 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 2 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0072] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.
[0073] Embodiment 3
[0074] Based on the above-mentioned embodiment 1, this embodiment further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned method are implemented.
[0075] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0076] In this embodiment, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperTextTransferProtocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0077] The computer-readable medium may be included in the device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains training data, converts the training data into initial data; determines an initial rule base based on the initial data, and optimizes the parameters of the initial rule base to obtain a target rule base; calculates the rules in the target rule base according to a preset activation weight calculation formula to obtain activation weights; determines abnormal information according to the test data and the activation weights.
[0078] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0079] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0080] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be provided in a processor, for example, may be described as: a processor including a data acquisition unit, a rule determination unit, a weight calculation unit, and an anomaly determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the data acquisition unit may also be described as a "unit for acquiring training data".
[0081] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.
[0082] Obviously, it should be understood by those skilled in the art that the above-mentioned various steps of the present invention can be performed in a manner different from the present invention, and the simulation method and experimental equipment include but are not limited to the above description. The above-mentioned various steps of the present invention can be performed in an order different from that here in some cases, and the steps shown or described above can be performed separately. Therefore, the present invention is not limited to any specific combination of hardware and software.
[0083] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.
Claims
1. A method for data desensitization, characterized in that: Applied to a data warehouse, the data warehouse is built using Hadoop and Hive technologies, and Spark SQL is used as the data query engine of the data warehouse. The method includes: Based on the needs of different users and scenarios, define the default desensitization algorithm for each field in the configuration file; According to the user's role, permissions and data desensitization requirements, the desensitization rules of the corresponding default desensitization algorithm are customized in the configuration file to obtain a personalized desensitization algorithm; Generate a desensitization algorithm file based on the default desensitization algorithm and the personalized desensitization algorithm; Obtain the user's SQL query request, parse and process the target data table to be desensitized based on the logic execution plan transformation strategy, capture the table content information, call the RestAPI of the desensitization algorithm file according to the table content information, obtain the corresponding default desensitization algorithm or personalized desensitization algorithm, desensitize the table content information according to the default desensitization algorithm or personalized desensitization algorithm, and obtain the desensitized data.
2. A data desensitization method according to claim 1, characterized in that: The method of generating a desensitizing algorithm file based on the default desensitizing algorithm and the personalized desensitizing algorithm specifically includes: writing the implementation logic process of each default desensitizing algorithm and each personalized desensitizing algorithm into an independent Java class file or a Scala class file, and compiling all Java class files or Scala class files into a JAR file, and loading the JAR file into the lib directory of the Spark computing engine for use when the Spark SQL query engine is running.
3. A data desensitization method according to claim 2, characterized in that: The default desensitization algorithm and the personalized desensitization algorithm both include a rule-based desensitization algorithm or a time-weighted desensitization algorithm.
4. A data desensitization method according to claim 1, characterized in that: The logic execution plan transformation strategy specifically includes: compiling the Spark Catalyst code, packaging the compiled Spark Catalyst code, and generating a new Spark Catalyst running package, wherein the new Spark Catalyst running package includes the JAR file required to run the newly added personalized desensitization algorithm, and deploying the new Spark Catalyst running package to the Spark SQL source code environment.
5. A data desensitization method according to claim 4, characterized in that: The compiling of the Spark Catalyst code specifically includes: setting a rule class for processing the logical plan, which is used to traverse the logical plan tree in its apply method.
6. A data desensitization method according to claim 5, characterized in that: The traversal logic plan tree specifically includes: during the traversal process, all table nodes and field information contained in the data table to be desensitized are identified and processed.
7. A data desensitization method according to claim 6, characterized in that: The method of obtaining a user's SQL query request, parsing a target data table to be desensitized based on a logical execution plan transformation strategy, capturing table content information, calling a RestAPI of a desensitizing algorithm file according to the table content information, obtaining a corresponding default desensitizing algorithm or a personalized desensitizing algorithm, desensitizing the table content information according to the default desensitizing algorithm or the personalized desensitizing algorithm, and obtaining desensitized data specifically includes: obtaining a user's SQL query request, parsing a target data table to be desensitized using a Spark SQL parser, capturing table content information, generating a logical plan tree object, traversing all table nodes in the plan tree object, screening field information that needs to be desensitized, calling a RestAPI of a desensitizing algorithm file for each field information that needs to be desensitized, obtaining a corresponding default desensitizing algorithm or a personalized desensitizing algorithm, desensitizing the field information that needs to be desensitized using the default desensitizing algorithm or the personalized desensitizing algorithm, until all field information that needs to be desensitized has been desensitized, and obtaining the desensitized data.
8. A data desensitization method according to claim 7, characterized in that: After obtaining the desensitized data, the method further includes: responding to the user's SQL query request based on the desensitized data.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
