Data desensitization method and system

Abstract

The invention provides a data desensitization method and system, and the method comprises the steps: obtaining a to-be-desensitized data table generated by a production environment, and constructing a pre-desensitization query command based on a desensitization parameter and an attribute parameter of the to-be-desensitized data table; acquiring a pre-desensitization result of the to-be-desensitized data table by using the command, then obtaining a target desensitization query command based on the pre-desensitization query command, desensitizing the to-be-desensitized data table to obtain desensitization data, and storing the desensitization data in a desensitization table; the production environment proxy server obtains the desensitization table from the first cluster server and stores the desensitization table in a first shared disk; synchronizing the desensitization table from the first shared disk to a second shared disk through the development environment proxy server; and obtaining the desensitization table from the second shared disk through the second cluster server, and storing the desensitization table to the target data table. According to the method, the desensitized data can be previewed, the to-be-desensitized data can be desensitized under the condition that the desensitization effect is correct, the desensitized data can be automatically transmitted from a production environment to a development environment, and the working efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to a data desensitization method and system. Background Art

[0002] Currently, when performing data desensitization, a desensitization list is synchronized from the production environment to the development environment. Then, a corresponding job for exporting a data file is manually scheduled to perform data desensitization to generate a desensitized file. However, the desensitized file may not meet the expectations. For example, there may be a situation where the desensitization execution fails, there is no data, or there is an error in the desensitization configuration field. In addition, during the process of synchronizing desensitized data from the development environment to the production environment, the person in charge needs to continuously monitor the desensitization situation and manually operate the transmission of the desensitized data, resulting in a long duration for the transmission of the desensitized data.

[0003] In summary, when the current desensitized data is synchronized from the development environment to the production environment, there are problems of long duration and low success rate. Summary of the Invention

[0004] This application provides a data desensitization method and system to at least solve the above technical problems existing in the prior art.

[0005] In the first aspect of this application, a data desensitization method is provided, including: Obtain a data table to be desensitized generated in the production environment, and construct a pre-desensitization query command based on desensitization parameters and attribute parameters of the data table to be desensitized; Use the pre-desensitization query command to obtain a pre-desensitization result of the data table to be desensitized, where the pre-desensitization result is used to display the data effect after desensitization of the data table to be desensitized; According to the pre-desensitization result, obtain a target desensitization query command based on the pre-desensitization query command; Perform desensitization processing on the data table to be desensitized based on the target desensitization query command, obtain desensitized data, and store it in a desensitized table of the first cluster server in the production environment; Obtain the desensitized table from the first cluster server through a production environment proxy server, and store the desensitized table in a first shared disk; Synchronize the desensitized table from the first shared disk to a second shared disk through a development environment proxy server; Obtain the desensitized table from the second shared disk through a second cluster server in the development environment, and store the desensitized data in the desensitized table in a target data table.

[0006] In one possible implementation, the attribute parameters include identification information of a source database to which the data table to be desensitized belongs and identification information of the data table to be desensitized; the desensitizing parameters include preset filtering conditions, sensitive fields, and desensitizing rules; and constructing a pre-desensitizing query command based on the desensitizing parameters and the attribute parameters of the data table to be desensitized includes: Based on the identification information of the source database and the identification information of the data table to be desensitized, a first command is generated to obtain all table structure fields of the data table to be desensitized; Traversing all table structure fields of the data table to be desensitized, and judging whether all table structure fields of the data table to be desensitized are sensitive fields based on the sensitive fields; In response to all the table structure fields of the to-be-desensitized data table being sensitive fields, obtaining corresponding desensitization functions based on desensitization rules corresponding to the sensitive fields; Based on the attribute parameters of the data table to be desensitized, the filtering conditions, the sensitive fields of the data table to be desensitized and the desensitization function, a pre-desensitization query command is constructed.

[0007] In one possible implementation, obtaining a target desensitized query command based on the pre-desensitization query command according to the pre-desensitization result includes: Determine whether the pre-desensitization result meets the preset desensitization condition; In response to the pre-desensitization result satisfying the preset desensitization condition, the desensitization operation restriction condition in the pre-desensitization query command is released to obtain a target desensitization query command.

[0008] In one possible implementation, the desensitization parameter further includes a desensitization application ID; and the desensitizing the data to be desensitized based on the target desensitization query command includes: Constructing a desensitization table in the first cluster server, wherein information of a desensitization database to which the desensitization table belongs is generated according to information of a source database and the desensitization application ID, and information of the desensitization table is generated according to information of the source data table and the desensitization application ID; Based on the information of the desensitization database and the information of the desensitization table, a second command is generated to obtain the table structure fields of the desensitization table; The information of the desensitizing database, the information of the desensitizing table and the table structure fields of the desensitizing table are added to the target desensitizing query command, and the data table to be desensitized is desensitized according to the updated target desensitizing query command.

[0009] In one possible implementation, after storing the desensitization table in the first shared disk, the method further includes: Construct a desensitization flag file corresponding to the desensitization table in the same-level directory of the file where the desensitization table is located in the first shared disk; the desensitization flag file includes the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table.

[0010] In an implementable manner, synchronizing the desensitization table from the first shared disk to the second shared disk through a development environment proxy server includes: Obtain the desensitization table and the desensitization flag file from the first shared disk and store them in the second shared disk; wherein, when storing in the second shared disk, first store the desensitization table in the second shared disk, and then store the desensitization flag file in the second shared disk.

[0011] In an implementable manner, obtaining the desensitization table from the second shared disk by the second cluster server in the development environment and storing the desensitized data in the desensitization table into the target data table includes: In response to the second cluster server receiving a timed polling instruction sent by the distributed task scheduling platform, obtain the desensitization table and the desensitization flag file from the second shared disk through the second cluster server; Based on the desensitization table and the desensitization flag file, load the desensitized data in the desensitization table into the target data table.

[0012] In an implementable manner, the loading the desensitized data in the desensitization table into the target data table based on the desensitization table and the desensitization flag file includes: Parse the desensitization flag file to obtain the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table; Generate a fourth command according to the identification information of the target database and the identification information of the target data table to determine whether the target data table exists in the second cluster server; In response to the target data table existing in the second cluster server, obtain the table structure fields of the target data table; Compare the table structure fields of the target data table with the table structure fields of the desensitization table, and when they are consistent, load the desensitized data into the target data table.

[0013] In a second aspect of the present application, a data desensitization system is provided, including: a first cluster server located in the production environment, a pre-desensitization server, a production environment proxy server, a development environment proxy server, and a second cluster server located in the development environment; wherein, A pre-desensitization server is used to obtain a data table to be desensitized generated in a production environment, construct a pre-desensitization query command based on desensitization parameters and attribute parameters of the data table to be desensitized; use the pre-desensitization query command to obtain a pre-desensitization result of the data table to be desensitized, and the pre-desensitization result is used to display the data effect after desensitization of the data table to be desensitized; based on the pre-desensitization result, obtain a target desensitization query command based on the pre-desensitization query command; perform desensitization processing on the data table to be desensitized based on the target desensitization query command, obtain desensitized data and store it in a desensitized table of a first cluster server in the production environment; A production environment proxy server is used to obtain the desensitized table from the first cluster server and store the desensitized table in a first shared disk; A development environment proxy server is used to synchronize the desensitized table from the first shared disk to a second shared disk; A second cluster server is used to obtain the desensitized table from the second shared disk and store the desensitized data in the desensitized table in a target data table.

[0014] In a third aspect of the present application, an electronic device is provided, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the present application.

[0015] In a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to cause a computer to execute the method described in the present application.

[0016] The data desensitization method and system of the present application can first preview desensitized data, desensitize the data to be desensitized when it is determined that the preview result shows that the desensitized data is correct, and can also realize the automatic transmission of desensitized data from the production environment to the development environment, improving work efficiency.

[0017] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By referring to the accompanying drawings and reading the following detailed description, the above and other objects, features, and advantages of the exemplary embodiments of the present application will become easily understood. In the drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0019] Figure 1 The figure shows a schematic flowchart of the implementation process of the data desensitization method according to the embodiment of the present application; Figure 2 The figure shows a schematic structural diagram of the data desensitization system according to the embodiment of the present application; Figure 3 The figure shows a schematic structural diagram of the composition of an electronic device according to the embodiment of the present application. Detailed implementation manners

[0020] To make the objectives, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0021] The following introduces a data desensitization method provided by the present application with reference to the accompanying drawings, including: S101. Obtain the data table to be desensitized generated in the production environment, and construct a pre-desensitization query command based on the desensitization parameters and the attribute parameters of the data table to be desensitized.

[0022] In the present application, the production environment refers to the source data environment that contains sensitive information (such as personal privacy, business secrets, financial data, etc.) during actual business operations. The production environment includes multiple source databases, and each source database includes multiple data tables to be desensitized. The data table to be desensitized has attribute parameters, such as the database name to which the data table to be desensitized belongs, the name of the data table to be desensitized, etc. The desensitization parameters are configuration items used to control the desensitization rules, scope, and effects during the data desensitization process. The desensitization parameters are the key elements for achieving precise desensitization, and through parameterized configuration, the sensitive data protection requirements of different business scenarios can be flexibly adapted. The desensitization parameters can be sensitive fields and desensitization rules, etc.

[0023] In the present application, a pre-desensitization query command is constructed through the desensitization parameters and the attribute parameters of the data table to be desensitized. Through the pre-desensitization query command, the data table to be desensitized can be determined, and the desensitization operation can be performed on the data table to be desensitized. Among them, the pre-desensitization query command is a SQL query statement that can perform structured desensitization processing on the data table to be desensitized in advance.

[0024] Exemplarily, the data table to be desensitized is desensitized according to the pre-desensitization query command to obtain a pre-desensitization result. For example, personal information such as ID numbers and mobile phone numbers, after desensitization, result in desensitized data with the ID numbers and mobile phone numbers hidden or represented by letters. This desensitized data can be used in scenarios such as development and testing, but is not directly associated with real individuals.

[0025] S102, Use the pre-desensitization query command to obtain the pre-desensitization result of the data table to be desensitized, and the pre-desensitization result is used to display the data effect after desensitization of the data table to be desensitized.

[0026] This application performs a pre-desensitization operation on the data table to be desensitized using the pre-desensitization query command to obtain the pre-desensitization result of the data table to be desensitized. Users can view whether the pre-desensitization result meets the desensitization requirements. Among them, the pre-desensitization result can be displayed on the display interface for the convenience of users to view.

[0027] Among them, the pre-desensitization result can be displayed in the form of a visual table or chart, and different display styles can also be set for fields with different sensitivity levels (such as high-sensitivity fields marked with special colors). It is also possible to show the data changes of the data table to be desensitized before and after the pre-desensitization operation through a desensitization effect comparison chart.

[0028] S103, According to the pre-desensitization result, obtain the target desensitization query command based on the pre-desensitization query command; If the pre-desensitization result meets the desensitization requirements, then this application can determine the target desensitization query command based on the pre-desensitization query command. The target desensitization query command is the command for desensitizing the data table to be desensitized.

[0029] If the pre-desensitization result does not meet the desensitization requirements, that is, the pre-desensitization query command is an incorrect desensitization query command. Then it is necessary to reconstruct the pre-desensitization query command until the pre-desensitization result obtained by desensitizing the data table to be desensitized with the constructed pre-desensitization query command meets the desensitization requirements, and determine the target desensitization query command based on the current pre-desensitization query command.

[0030] S104, Based on the target desensitization query command, desensitize the data table to be desensitized to obtain desensitized data and store it in the desensitized table of the first cluster server in the production environment; This application pre-constructs a desensitized table in the first cluster server of the production environment. This desensitized table is used to store desensitized data. After desensitizing the data table to be desensitized using the target desensitization query command to obtain desensitized data, the desensitized data is written into the desensitized table.

[0031] S105, Obtain the desensitized table from the first cluster server through the production environment proxy server and store the desensitized table in the first shared disk; The production environment proxy server has the same network as the first cluster server in the production environment. A first program is provided in the production environment proxy server. The first program can obtain the desensitized table in the first cluster server and store the desensitized table in the first shared disk. Among them, the first shared disk is mounted under the production environment proxy server. The first program can be a Java program.

[0032] The first shared disk can be a Network Attached Storage (NAS) disk. The NAS disk can provide data storage and sharing services through a network connection with the production environment proxy server.

[0033] S106, synchronize the desensitized table from the first shared disk to the second shared disk through the development environment proxy server; The second shared disk can be a NAS disk. Both the first shared disk and the second shared disk are mounted under the development environment proxy server. A second program is provided in the development environment proxy server. The second program can synchronize the desensitized table in the first shared disk to the second shared disk.

[0034] S107, obtain the desensitized table from the second shared disk through the second cluster server in the development environment, and store the desensitized data in the desensitized table into the target data table.

[0035] The development environment proxy server has the same network as the second cluster server. A third program is provided in the second cluster server. The third program can obtain the desensitized table from the second shared disk mounted by the development environment proxy server and store the desensitized data in the desensitized table into the target data table. The target data table is a table pre-constructed in the second cluster server.

[0036] The data desensitization method provided by this application can construct a pre-desensitization query command based on the desensitization parameters and the attribute parameters of the data table to be desensitized, and use the pre-desensitization query command to obtain the pre-desensitization result of the data table to be desensitized, so as to achieve desensitization preview and judge the data desensitization effect of the pre-desensitization query command. If the pre-desensitization result meets the desensitization requirements, the target desensitization query command is obtained according to the pre-desensitization query command. Desensitize the data table to be desensitized based on the target desensitization query command, obtain the desensitized data and store it in the desensitized table of the first cluster server in the production environment. Obtain the desensitized table from the first cluster server through the production environment proxy server and store the desensitized table in the first shared disk. Then synchronize the desensitized table from the first shared disk to the second shared disk through the development environment proxy server. Finally, obtain the desensitized table from the second shared disk through the second cluster server in the development environment, and store the desensitized data in the desensitized table into the target data table.

[0037] This application first constructs a pre-desensitization query command, previews the desensitization result of the data table to be desensitized according to the pre-desensitization query command, and when the pre-desensitization result meets the desensitization requirements, obtains the target desensitization query command based on the pre-desensitization query command. Then, the data table to be desensitized is desensitized using the target desensitization query command to obtain desensitized data. Next, the synchronization of the desensitized table is achieved through the production environment proxy server, the first shared disk, the development environment proxy server, the second shared disk, and the second cluster server. Finally, the goal of automatically storing the desensitized data into the target data table of the second cluster server in the development environment is realized. This not only improves the accuracy of the desensitized data but also improves the work efficiency of synchronizing the desensitized data from the production environment to the development environment.

[0038] In some embodiments, the attribute parameters include the identification information of the source database to which the data table to be desensitized belongs and the identification information of the data table to be desensitized; the desensitization parameters include a preset filtering condition, sensitive fields, and desensitization rules; the constructing of the pre-desensitization query command based on the desensitization parameters and the attribute parameters of the data table to be desensitized includes: Based on the identification information of the source database and the identification information of the data table to be desensitized, generate a first command to obtain all the table structure fields of the data table to be desensitized; Traverse all the table structure fields of the data table to be desensitized, and based on the sensitive fields, determine whether all the table structure fields of the data table to be desensitized are sensitive fields; In response to all the table structure fields of the data table to be desensitized being sensitive fields, obtain the corresponding desensitization function based on the desensitization rules corresponding to the sensitive fields; Based on the attribute parameters of the data table to be desensitized, the filtering condition, the sensitive fields of the data table to be desensitized, and the desensitization function, construct a pre-desensitization query command.

[0039] In the present application, the attribute parameters include the identification information of the source database to which the data table to be desensitized belongs and the identification information of the data table to be desensitized. Among them, the identification information of the source database may be the library name of the source database, and the identification information of the data table to be desensitized may be the name of the data table to be desensitized. The desensitization parameters include preset filtering conditions, sensitive fields and desensitization rules. In the present application, a first command can be generated according to the library name and table name to which the data table to be desensitized belongs. All table structure fields of the data table to be desensitized can be obtained by executing the first command by the target execution engine. The table structure field is used to characterize the organizational form, field name and storage rules of the data in the data table to be desensitized. Traverse the table structure field, and determine whether all table structure fields of the data table to be desensitized are desensitized fields based on the desensitized field. If all table structure fields of the data table to be desensitized are sensitive fields, determine the desensitization rules corresponding to the sensitive fields, and then obtain the desensitization function corresponding to the desensitization rule. The attribute parameters of the data table to be desensitized, the filtering conditions, the sensitive fields of the data table to be desensitized and the desensitization function are assembled to construct a pre-desensitization query command.

[0040] Among them, the filtering condition can be a condition for filtering the data that needs to be desensitized in the data table to be desensitized. It can be a time condition. For example, you only want to desensitize the data in a fixed time period in the data table to be desensitized. You can set the filtering condition to the time range to filter the data outside the time range and only desensitize the data in the fixed time period. It can also be a product dimension condition, such as financial products. For example, you want to desensitize the relevant data of financial products. Then, according to the filtering condition, filter the non-financial related data in the desensitized data table and desensitize the financial related data.

[0041] Exemplarily, "Description: {data table library name to be desensitized.data table name to be desensitized}" is generated according to the library name and table name of the data table to be desensitized, and "Description: {data table library name to be desensitized.data table name to be desensitized}" is the first command. Call the ETL (Extract, Transform, Load) execution engine and execute "Description: {data table library name to be desensitized.data table name to be desensitized}" to obtain all the table structure fields of the data table to be desensitized. Obtain the corresponding desensitization function based on the desensitization rule, and then assemble the library name, table name, table structure fields, filter conditions, and desensitization function of the data table to be desensitized to obtain the pre-desensitization query command.

[0042] In some embodiments, obtaining a target desensitized query command based on the pre-desensitized query command according to the pre-desensitized query result includes: Determine whether the pre-desensitization result meets the preset desensitization condition; In response to the pre-desensitization result satisfying the preset desensitization condition, the desensitization operation restriction condition in the pre-desensitization query command is released to obtain a target desensitization query command.

[0043] It should be noted that in this application, in order to improve the preview efficiency of pre-desensitization, restrictions are set in the pre-desensitization query command. This restriction causes only the first five desensitized data of the data table to be desensitized to be displayed in the pre-desensitization result obtained by the pre-desensitization query command. Through these five desensitized data, it can be determined whether the pre-desensitization result meets the desensitization requirements. If the desensitization requirements are met, the restriction on the pre-desensitization query command is lifted to obtain the target desensitization query command.

[0044] In some embodiments, the desensitization parameter further includes a desensitization application ID; the desensitizing the data to be desensitized based on the target desensitization query command includes: Constructing a desensitization table in the first cluster server, where the information of the desensitization database to which the desensitization table belongs is generated according to the information of the source database and the desensitization application ID, and the information of the desensitization table is generated according to the information of the source data table and the desensitization application ID; Generating a second command based on the information of the desensitization database and the desensitization table to obtain the table structure fields of the desensitization table; Adding the information of the desensitization database, the information of the desensitization table, and the table structure fields of the desensitization table to the target desensitization query command, and desensitizing the data table to be desensitized according to the updated target desensitization query command.

[0045] In this application, the desensitization parameter further includes a desensitization application ID. In this application, a desensitization table is pre-constructed in the first cluster server. The information of the desensitization database to which the desensitization table belongs is generated according to the information of the source database and the desensitization application ID, and the information of the desensitization table is generated according to the information of the source data table and the desensitization application ID. For example, according to the name of the data table to be desensitized and the table name, the ETL execution engine is called. The ETL execution engine executes "If the desensitization table {desensitization table library name.desensitization table name} already exists, delete the table", "If the desensitization table does not exist, create a desensitization table {desensitization table library name.desensitization table name} based on {data table to be desensitized library name.data table to be desensitized name}", and the desensitization table name is: data table to be desensitized name_application ID. In this application, after constructing the desensitization table, it is necessary to refresh the desensitization table metadata first. Specifically, the ETL execution engine service is called to execute "Refresh metadata {desensitization table library name.desensitization table name}" to refresh the metadata cache of the desensitization table to ensure that the desensitized data is subsequently stored in the desensitization table.

[0046] Then, based on the information of the desensitization database and the desensitization table, a second command is generated to obtain the table structure fields of the desensitization table. The information of the desensitization database, the information of the desensitization table, and the table structure fields of the desensitization table are added to the target desensitization query command, and the data table to be desensitized is desensitized according to the updated target desensitization query command.

[0047] Exemplarily, according to the desensitization table library name and desensitization table name, generate a statement of "Description: desensitization table library name.desensitization table name". Call the ETL execution engine to execute this statement to obtain the table structure fields of the desensitization table. According to the desensitization application ID, obtain the pre-desensitization query command saved locally through the pre-desensitization result interface. Remove the restrictions in the pre-desensitization query command to obtain the target desensitization query command. Concatenate the target desensitization query command into the statement of "Overwrite insert: {desensitization table library name.desensitization table name} ({desensitization table's table structure field names,...}) target desensitization query command [Select {field / field after desensitization function processing,...} from {data table library name to be desensitized.data table name to be desensitized} while {filter condition}]" to achieve the writing of desensitized data. At the same time, ensure that the data written into the desensitization table is consistent with the previewed desensitization result.

[0048] In some embodiments, after storing the desensitization table in the first shared disk, the method further includes: Under the same-level directory of the file where the desensitization table is located in the first shared disk, construct a desensitization flag file corresponding to the desensitization table; the desensitization flag file includes the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table.

[0049] In this application, under the same-level directory of the file where the desensitization table is located, construct a desensitization flag file according to the desensitization table library name, desensitization table name, and desensitization application ID. Write the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table into the desensitization flag file.

[0050] In this application, the desensitization flag file can represent the data that has been desensitized. It can also represent the target data table to which the desensitized data needs to be stored in the second cluster server in the development environment.

[0051] In some embodiments, the synchronization of the desensitization table from the first shared disk to the second shared disk through the development environment proxy server includes: Obtain the desensitization table and the desensitization flag file from the first shared disk and store them in the second shared disk; wherein, when storing in the second shared disk, first store the desensitization table in the second shared disk, and then store the desensitization flag file in the second shared disk.

[0052] In this application, the second program in the development environment proxy server can obtain the desensitization table and the desensitization flag file from the first shared disk and store them in the second shared disk; wherein, the second program first obtains the desensitization table and stores it in the second shared disk, and then obtains the corresponding desensitization flag file and stores it in the second shared disk. If the desensitization flag file exists in the second shared disk, it indicates that the desensitization table has been synchronized from the first shared disk to the second shared disk.

[0053] If the desensitization table is successfully synchronized from the first shared disk to the second shared disk, the desensitization table in the first cluster server in the production environment, the production environment proxy server, and the first shared disk can be deleted to ensure sufficient storage space for the first cluster server, the production environment proxy server, and the first shared disk.

[0054] In some embodiments, obtaining the desensitization table from the second shared disk by the second cluster server in the development environment and storing the desensitized data in the desensitization table into the target data table includes: In response to the second cluster server receiving a timed polling instruction sent by the distributed task scheduling platform, obtaining the desensitization table and the desensitization flag file from the second shared disk through the second cluster server; Based on the desensitization table and the desensitization flag file, loading the desensitized data in the desensitization table into the target data table.

[0055] In this application, the distributed scheduling platform can send a timed polling instruction to the third program in the second cluster server. After receiving the timed polling instruction, the third program obtains the desensitization table and the desensitization flag file from the second shared disk. According to the identification information of the target database to which the target data table belongs and the identification information of the target data table in the desensitization flag file, the desensitized data in the desensitization table is written into the target data table.

[0056] In this application, through the synchronization of the desensitization table among the first cluster server, the production environment proxy server, the development environment proxy server, and the second cluster server, the automatic synchronization of desensitized data from the production environment to the development environment is achieved. There is no need for manual synchronization processing, which improves the desensitization work efficiency.

[0057] In some embodiments, the loading the desensitized data in the desensitization table into the target data table based on the desensitization table and the desensitization flag file includes: Parsing the desensitization flag file to obtain the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table; According to the identification information of the target database and the identification information of the target data table, generating a fourth command to determine whether the target data table exists in the second cluster server; In response to the target data table existing in the second cluster server, obtaining the table structure fields of the target data table; Comparing the table structure fields of the target data table with the table structure fields of the desensitization table, and when they are consistent, loading the desensitized data into the target data table.

[0058] In this application, after obtaining the desensitization table and the desensitization flag file from the second shared disk, first parse the desensitization flag file to obtain the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table; generate a fourth command according to the identification information of the target database and the identification information of the target data table. Executing the fourth command can determine whether the target data table exists in the second cluster server. If the target data table exists in the second cluster server, obtain the table structure fields of the target data table. Compare the table structure fields of the target data table with the table structure fields of the desensitization table. If they are consistent, load the desensitized data into the target data table.

[0059] Exemplarily, parse the desensitization flag file to obtain the desensitization table library name, desensitization table name, desensitization application ID, target data table library name, and target data table name corresponding to the file, and store the above information in a record table. Generate a statement of "description formatting: target data table library name.target data table name", call the intelligent ETL execution engine, execute this statement in the second cluster server, and determine whether the target data table exists in the second cluster server. If it exists, obtain the table structure fields of the target data table and compare them with the table structure fields of the desensitization table. If the comparison passes, proceed to the next step; otherwise, send a notification of the inconsistency of the target data table information and update the import status of this record in the record table.

[0060] Further, before importing the desensitized data into the target data table, this application can also execute a command of "description format: target data table library name.target data table name" to obtain the type information of the target data table. The types of the target data table include static tables and partitioned tables. If the target data table is a static table, generate a statement of "truncate: target data table library name.target data table name", call the ETL execution engine, and execute this statement to clear the data in the static table. If the target data table is a partitioned table, parse the desensitization table to obtain the desensitization table partition information, generate a statement of "add, modify or delete table: target data table library name.target data table name delete the specified partition in the table (partition field = specified partition)", call the ETL execution engine, and execute this statement to clear the data of the specified partition in the partitioned table. The clearing operation ensures the accuracy of the data after the desensitized data is imported into the target data table.

[0061] As a specific implementation manner, taking the example of exporting 100G of data per day on average, compare the existing desensitization results with the desensitization results of this application: Existing desensitization results: For 100G of data, the average execution time of the data file export job is 1.5 hours, the average time for the desensitized table synchronization and loading jobs is 0.5 hours, and the total execution time of the job scheduling through the jump instruction (JMP) is 2 hours. During this period, the data responsible person needs to continuously monitor the execution progress of 3 JMP jobs, and manually start the next job task after one job is completed. The desensitization result can be viewed only after the entire process is completed, and the desensitization success rate is only 50%-60%.

[0062] Desensitization results of this application: Just upload the desensitized data list, parse the desensitization parameters and the attribute parameters of the data table to be desensitized, and the desensitization result can be previewed in real time. After the desensitization starts, the system can automatically implement data desensitization and data file synchronization operations. The data responsible person waits for the notification and does not need to continuously monitor the execution situation. The development environment automatically imports the desensitized data into the corresponding target data table. After the desensitized data process is optimized, the average input time for the data responsible person to desensitize a data table of 100G size is about 20 seconds, and it is estimated that at least 20 minutes of input time can be saved every day. The desensitization success rate is 90%-100%, with a 40%-50% increase.

[0063] The data desensitization method provided by this application can preview the desensitized data, desensitize the data to be desensitized when the desensitization effect is correct, and can also realize the automatic transmission of the desensitized data from the production environment to the development environment, improving work efficiency.

[0064] As Figure 2 shown, this application provides a data desensitization system, including: The first cluster server 201, pre-desensitization server 202, production environment proxy server 203, development environment proxy server 204 located in the production environment, and the second cluster server 205 located in the development environment. Among them, The pre-desensitization server 202 obtains the data table to be desensitized generated in the production environment, constructs a pre-desensitization query command based on the desensitization parameters and the attribute parameters of the data table to be desensitized; uses the pre-desensitization query command to obtain the pre-desensitization result of the data table to be desensitized, and the pre-desensitization result is used to display the data effect after desensitization of the data table to be desensitized; based on the pre-desensitization result, obtain the target desensitization query command based on the pre-desensitization query command; desensitize the data table to be desensitized based on the target desensitization query command to obtain desensitized data and store it in the desensitized table of the first cluster server in the production environment; The production environment proxy server 203 is used to obtain the desensitized table from the first cluster server and store the desensitized table in the first shared disk; The development environment proxy server 204 is used to synchronize the desensitized table from the first shared disk to the second shared disk; The second cluster server 205 is configured to obtain the desensitized table from the second shared disk and store the desensitized data in the desensitized table into a target data table.

[0065] The data desensitization system provided in this application includes a first cluster server 201, a pre-desensitization server 202, a production environment proxy server 203, a development environment proxy server 204 in the production environment, and a second cluster server 205 in the development environment. Among them, the pre-desensitization server 202 obtains the data table to be desensitized generated in the production environment, constructs a pre-desensitization query command based on the desensitization parameters and the attribute parameters of the data table to be desensitized; uses the pre-desensitization query command to obtain the pre-desensitization result of the data table to be desensitized, and the pre-desensitization result is used to display the data effect after desensitization of the data table to be desensitized; according to the pre-desensitization result, obtain a target desensitization query command based on the pre-desensitization query command; perform desensitization processing on the data table to be desensitized based on the target desensitization query command to obtain desensitized data and store it in the desensitized table of the first cluster server in the production environment; the production environment proxy server 203 obtains the desensitized table from the first cluster server and stores the desensitized table in the first shared disk; the development environment proxy server 204 synchronizes the desensitized table from the first shared disk to the second shared disk; the second cluster server 205 obtains the desensitized table from the second shared disk and stores the desensitized data in the desensitized table into a target data table.

[0066] In some embodiments, the attribute parameters include the identification information of the source database to which the data table to be desensitized belongs and the identification information of the data table to be desensitized; the desensitization parameters include a preset filtering condition, sensitive fields, and desensitization rules; the pre-desensitization server is specifically configured to generate a first command based on the identification information of the source database and the identification information of the data table to be desensitized to obtain all the table structure fields of the data table to be desensitized; traverse all the table structure fields of the data table to be desensitized, and determine whether all the table structure fields of the data table to be desensitized are sensitive fields based on the sensitive fields; in response to all the table structure fields of the data table to be desensitized being sensitive fields, obtain the corresponding desensitization function based on the desensitization rule corresponding to the sensitive fields; construct a pre-desensitization query command based on the attribute parameters of the data table to be desensitized, the filtering condition, the sensitive fields of the data table to be desensitized, and the desensitization function.

[0067] An embodiment of this application provides an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any of the above embodiments.

[0068] An embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to cause a computer to execute the method described in any of the above embodiments.

[0069] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0070] Figure 3 A schematic block diagram of an exemplary electronic device 800 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present application described and / or claimed herein.

[0071] As Figure 3 shown, the device 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0072] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0073] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes the various methods and processes described above, such as the data masking method. For example, in some embodiments, the data masking method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data masking method described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute the data masking method by any other suitable means (e.g., by means of firmware).

[0074] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program codes can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0076] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0077] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0078] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0079] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, can also be a server of a distributed system, or a server incorporating a blockchain.

[0080] It should be understood that the various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution of this application can be achieved, and no limitations are imposed herein.

[0081] In addition, the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "a plurality" means two or more, unless otherwise specifically defined.

[0082] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

1. A data desensitization method, characterized in that, include: Obtain a data table to be desensitized generated in a production environment, and construct a pre-desensitization query command based on desensitization parameters and attribute parameters of the data table to be desensitized; The pre-desensitization query command is used to obtain the pre-desensitization result of the data table to be desensitized, and the pre-desensitization result is used to display the data effect after desensitization of the data table to be desensitized; According to the pre-desensitization result, a target desensitization query command is obtained based on the pre-desensitization query command; Performing desensitization processing on the to-be-desensitized data table based on the target desensitization query command, obtaining desensitized data and storing it in a desensitization table of the first cluster server of the production environment; Obtaining the desensitization table from the first cluster server through a production environment proxy server, and storing the desensitization table in a first shared disk; Synchronize the desensitization table from the first shared disk to the second shared disk through a development environment proxy server; The desensitizing table is obtained from the second shared disk through the second cluster server in the development environment, and the desensitizing data in the desensitizing table is stored in the target data table.

2. The method according to claim 1, wherein, The attribute parameters include identification information of the source database to which the data table to be desensitized belongs and identification information of the data table to be desensitized; The desensitization parameters include preset filtering conditions, sensitive fields and desensitization rules; The constructing of a pre-desensitization query command based on the desensitization parameter and the attribute parameter of the data table to be desensitized includes: Based on the identification information of the source database and the identification information of the data table to be desensitized, a first command is generated to obtain all table structure fields of the data table to be desensitized; Traversing all table structure fields of the data table to be desensitized, and judging whether all table structure fields of the data table to be desensitized are sensitive fields based on the sensitive fields; In response to all the table structure fields of the to-be-desensitized data table being sensitive fields, obtaining corresponding desensitization functions based on desensitization rules corresponding to the sensitive fields; Based on the attribute parameters of the data table to be desensitized, the filtering conditions, the sensitive fields of the data table to be desensitized and the desensitization function, a pre-desensitization query command is constructed.

3. The method according to claim 1, characterized in that The step of obtaining a target desensitized query command based on the pre-desensitized query command according to the pre-desensitized query result includes: Determine whether the pre-desensitization result meets the preset desensitization condition; In response to the pre-desensitization result satisfying the preset desensitization condition, the desensitization operation restriction condition in the pre-desensitization query command is released to obtain a target desensitization query command.

4. The method according to claim 2, wherein The desensitization parameter also includes a desensitization application ID; and performing desensitization processing on the data to be desensitized based on the target desensitization query command includes: Constructing a desensitization table in the first cluster server, wherein information of a desensitization database to which the desensitization table belongs is generated according to information of a source database and the desensitization application ID, and information of the desensitization table is generated according to information of the source data table and the desensitization application ID; Based on the information of the desensitization database and the information of the desensitization table, a second command is generated to obtain the table structure fields of the desensitization table; The information of the desensitizing database, the information of the desensitizing table and the table structure fields of the desensitizing table are added to the target desensitizing query command, and the data table to be desensitized is desensitized according to the updated target desensitizing query command.

5. The method according to claim 4, wherein After storing the desensitization table in the first shared disk, the method further includes: Construct a desensitization flag file corresponding to the desensitization table in the same-level directory of the file where the desensitization table is located in the first shared disk; the desensitization flag file includes the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table.

6. The method according to claim 5, wherein The synchronization of the desensitization table from the first shared disk to the second shared disk through the development environment proxy server includes: Obtain the desensitization table and the desensitization flag file from the first shared disk and store them in the second shared disk; when storing in the second shared disk, first store the desensitization table in the second shared disk, and then store the desensitization flag file in the second shared disk.

7. The method according to claim 6, wherein The acquisition of the desensitization table from the second shared disk by the second cluster server in the development environment and the storage of the desensitized data in the desensitization table into the target data table include: In response to the second cluster server receiving a timed polling instruction sent by the distributed task scheduling platform, obtain the desensitization table and the desensitization flag file from the second shared disk through the second cluster server; Based on the desensitization table and the desensitization flag file, load the desensitized data in the desensitization table into the target data table.

8. The method according to claim 7, wherein The loading of the desensitized data in the desensitization table into the target data table based on the desensitization table and the desensitization flag file includes: Parse the desensitization flag file to obtain the table structure fields of the desensitization table, the identification information of the target database to which the target data table belongs, and the identification information of the target data table; Generate a fourth command according to the identification information of the target database and the identification information of the target data table to determine whether the target data table exists in the second cluster server; In response to the existence of the target data table in the second cluster server, obtain the table structure fields of the target data table; Compare the table structure fields of the target data table with the table structure fields of the desensitization table, and when they are consistent, load the desensitized data into the target data table.

9. A data desensitization system, characterized in that, The first cluster server, pre-desensitization server, production environment proxy server, development environment proxy server located in the production environment, and the second cluster server located in the development environment; where The pre-desensitization server is used to obtain the data table to be desensitized generated in the production environment, construct a pre-desensitization query command based on the desensitization parameters and the attribute parameters of the data table to be desensitized; use the pre-desensitization query command to obtain the pre-desensitization result of the data table to be desensitized, and the pre-desensitization result is used to display the data effect after desensitization of the data table to be desensitized; according to the pre-desensitization result, obtain a target desensitization query command based on the pre-desensitization query command; perform desensitization processing on the data table to be desensitized based on the target desensitization query command to obtain desensitized data and store it in the desensitization table of the first cluster server in the production environment; The production environment proxy server is used to obtain the desensitization table from the first cluster server and store the desensitization table in the first shared disk; The development environment proxy server is used to synchronize the desensitization table from the first shared disk to the second shared disk; The second cluster server is used to obtain the desensitized table from the second shared disk and store the desensitized data in the desensitized table into the target data table.

10. The system according to claim 9, wherein The attribute parameters include the identification information of the source database to which the data table to be desensitized belongs and the identification information of the data table to be desensitized; the desensitization parameters include a preset filtering condition, sensitive fields, and desensitization rules. The pre-desensitization server is specifically used for generating a first command based on the identification information of the source database and the identification information of the data table to be desensitized to obtain all the table structure fields of the data table to be desensitized. traversing all the table structure fields of the data table to be desensitized and determining whether all the table structure fields of the data table to be desensitized are sensitive fields based on the sensitive fields. In response to all the table structure fields of the data table to be desensitized being sensitive fields, obtaining the corresponding desensitization function based on the desensitization rules corresponding to the sensitive fields. Constructing a pre-desensitization query command based on the attribute parameters of the data table to be desensitized, the filtering condition, the sensitive fields of the data table to be desensitized, and the desensitization function.

Images