Database data desensitization method and system and medium

Abstract

The invention relates to a data desensitization method and system of a database and a medium, and the method comprises the steps that a desensitization function is created through a script language of a preset database, and the desensitization function is used for desensitizing sensitive data columns in the preset database; based on an SQL query statement input by a user, determining a plurality of data fragments where target query data is located in a preset database; on different data nodes where the data fragments are located, distributed desensitization operation is conducted on sensitive data columns involved in the target query data through the desensitization function. Through the data desensitization method and device, the desensitization function is directly created in the database, the learning cost of data desensitization is reduced, meanwhile, the desensitization task is distributed to the multiple corresponding data nodes to be executed in parallel, the efficiency of large-scale data desensitization is remarkably improved, the desensitization task is completed in a database kernel, and the data desensitization efficiency is improved. The risk that the data is leaked in the transmission process is reduced, and the problem of how to desensitize the data in the database is solved.

Description

Technical Field

[0001] This application relates to the field of database technology, and particularly to a method, system, and medium for data desensitization of a database. Background Art

[0002] In the era of big data, database products are facing increasingly severe data security challenges. Data desensitization technology is an important data security technology that can effectively protect sensitive data and prevent data leakage. However, traditional data desensitization has the following deficiencies:

[0003] First, the coupling degree with database products is low. Traditional data desensitization usually exists as an independent tool or plugin, with a low coupling degree with the database product itself, making it difficult to fully utilize the characteristics of the database for efficient desensitization. In addition, there is a risk of data leakage when sensitive data is transmitted between the database and the independent tool or plugin. Second, the desensitization rules provided by traditional data desensitization are limited, making it difficult to meet the personalized needs of the database in different scenarios, such as dynamically adjusting the desensitization strategy according to data changes and business requirements.

[0004] Currently, no effective solution has been proposed for the problem of how to achieve data desensitization in a database in related technologies. Summary of the Invention

[0005] Embodiments of this application provide a method, system, and medium for data desensitization of a database to at least solve the problem of how to achieve data desensitization in a database in related technologies.

[0006] In a first aspect, embodiments of this application provide a method for data desensitization of a database, the method including:

[0007] Create a desensitization function through the script language of a preset database, where the desensitization function is used to desensitize sensitive data columns in the preset database;

[0008] Based on an SQL query statement input by a user, determine several data shards where target query data is located in the preset database;

[0009] On different data nodes where the several data shards are located, respectively perform distributed desensitization operations on the sensitive data columns involved in the target query data through the desensitization function.

[0010] In some embodiments, creating a desensitization function through the script language of a preset database includes:

[0011] Customize the desensitization logic of the desensitization function through the script language of the preset database, and then create the desensitization function, where the desensitization logic includes regular expression replacement, encryption algorithm, data masking, and hash algorithm.

[0012] In some of these embodiments, after creating a desensitization function through the script language of a preset database, the method includes:

[0013] Then bind the desensitization function to the sensitive data column through the script language of the preset database, where the binding is used to call the desensitization function to desensitize the sensitive data column when a user performs a data query.

[0014] In some of these embodiments, after determining several data shards where target query data is located in the preset database based on an SQL query statement input by a user, the method includes:

[0015] Detect and identify the SQL query statement input by the user, and when it is recognized that the SQL query statement involves a sensitive data column, determine whether the user has the permission to read the plaintext of the sensitive data column.

[0016] In some of these embodiments, on different data nodes where the several data shards are located, respectively performing a distributed desensitization operation on the sensitive data columns involved in the target query data through the desensitization function includes:

[0017] If the user does not have the permission to read the plaintext of the sensitive data column, then call the desensitization function bound to the sensitive data column on different data nodes where the several data shards are located, and respectively perform a distributed desensitization operation on the sensitive data columns involved in the target query data.

[0018] In some of these embodiments, the method further includes:

[0019] If the user does not have the permission to read the plaintext of the sensitive data column and needs to perform a calculation on the sensitive data column, then first perform the calculation of the sensitive data column on different data nodes where the several data shards are located, and then respectively perform a distributed desensitization operation on the sensitive data columns involved in the target query data through the desensitization function.

[0020] In some of these embodiments, the method includes:

[0021] Modify the desensitization logic and / or desensitization object of the created desensitization function through the script language of the preset database, thereby obtaining a modified desensitization function;

[0022] Start or disable the created desensitization function through the script language of the preset database.

[0023] In some of these embodiments, the method includes:

[0024] The data sharding is obtained based on the distributed table built in the preset database. Among them, the sharding rules defined by the distributed table include vertical sharding, horizontal sharding, hash sharding, range sharding, and hybrid sharding.

[0025] In a second aspect, an embodiment of the present application provides a data desensitization system for a database. The system includes a creation module, a positioning module, and a distributed desensitization module.

[0026] The creation module is used to create a desensitization function through the script language of the preset database, where the desensitization function is used to desensitize sensitive data columns in the preset database.

[0027] The positioning module is used to determine several data shards where the target query data is located in the preset database according to the SQL query statement input by the user.

[0028] The distributed desensitization module is used to perform distributed desensitization operations on the sensitive data columns involved in the target query data through the desensitization function on different data nodes where the several data shards are located.

[0029] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the method described in the first aspect above.

[0030] Compared with the related art, an embodiment of the present application provides a data desensitization method, system, and medium for a database. Among them, the method creates a desensitization function through the script language of the preset database, where the desensitization function is used to desensitize sensitive data columns in the preset database; based on the SQL query statement input by the user, it determines several data shards where the target query data is located in the preset database; on different data nodes where the several data shards are located, it performs distributed desensitization operations on the sensitive data columns involved in the target query data through the desensitization function, realizing the creation of a desensitization function directly in the database, reducing the learning cost of data desensitization in the database, and at the same time using the data sharding characteristics of the database to distribute the desensitization tasks to multiple corresponding data nodes for parallel execution, significantly improving the efficiency of large-scale data desensitization, and the desensitization tasks are completed in the database kernel, and the sensitive data columns are desensitized before being transmitted to the middleware or application program, reducing the risk of data leakage during the transmission process, and solving the problem of how to implement data desensitization in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0032] Figure 1 It is a flowchart of the steps of the database data desensitization method according to an embodiment of the present application;

[0033] Figure 2 It is a schematic internal structure diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be described and explained below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0035] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, without creative efforts, the present application can also be applied to other similar scenarios based on these drawings. In addition, it can also be understood that although the efforts made in this development process may be complex and time-consuming, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacturing or production changes based on the technical content disclosed in the present application are only conventional technical means and should not be understood as the content disclosed in the present application being insufficient.

[0036] Referring to "embodiment" in the present application means that the specific features, structures or characteristics described in combination with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0037] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the ordinary meanings understood by those with ordinary skills in the technical field to which this application belongs. The words such as "a", "an", "one", "the" and the like involved in this application do not indicate a quantity limitation and may represent a singular or plural number. The terms "include", "comprise", "have" and any variations thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may further include steps or units not listed, or may further include other steps or units inherent to these processes, methods, products or devices. The words such as "connect", "be connected", "couple" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. The terms "first", "second", "third" and the like involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0038] An embodiment of this application provides a method for data desensitization of a database. Figure 1 It is a flowchart of the steps of the method for data desensitization of a database according to an embodiment of this application, as Figure 1 shown, and the method includes the following steps:

[0039] Step S102, create a desensitization function through the script language of a preset database, where the desensitization function is used to desensitize sensitive data columns in the preset database;

[0040] Specifically in step S102, customize the desensitization logic of the desensitization function through the script language of the preset database, and then create the desensitization function, where the desensitization logic includes regular expression replacement, encryption algorithm, data masking and hash algorithm.

[0041] It should be noted by way of example that the desensitization logic of the desensitization function can be customized through the script language of the preset database (such as replacing the numbers in the string with #). The preset database is preferably the DolphinDB database, that is, the script language is DolphinScript, and the script language statement can be:

[0042] def redaction(str){

[0043] return regexReplace(str, "[0-9]+", "#"); --

[0044] }

[0045] Furthermore, it should be noted that DolphinDB database has built-in distributed tables, and data sharding is obtained based on the built-in distributed tables of the preset database. Among them, the sharding rules specified for distributed tables include vertical sharding, horizontal sharding, hash sharding, range sharding, and hybrid sharding. For example, data can be horizontally sharded and stored on multiple nodes, and the data desensitization task is directly executed in parallel on the data nodes of each shard of the distributed table, without the need to centralize the data to a single node for processing, thus significantly improving the desensitization efficiency.

[0046] After step S102, the method further includes step S103, and then binds the desensitization function to the sensitive data column through the script language of the preset database, where the binding is used to call the desensitization function to desensitize the sensitive data column when the user performs a data query.

[0047] It should be noted by way of example that through the script language of the preset database, the user can mark the sensitive data columns in the table to bind the corresponding desensitization function. The preset database is preferably the DolphinDB database, that is, the script language is DolphinScript, and the script language statement can be:

[0048] setTableSensitiveColumn('pt', 'col1', true, redaction)

[0049] Among them, pt is the target table name of the sensitive data column, col1 is the target column name of the sensitive data column, true indicates enabling desensitization, and redaction is the bound desensitization function. After the marking and binding, the sensitive data column will automatically apply the bound desensitization function during query.

[0050] Step S104, based on the SQL query statement input by the user (such as select redaction(col1) from pt where col2 = 'xxx'), determine several data shards where the target query data is located in the preset database;

[0051] It should be noted that when the user executes a query containing desensitized data, the query parser of the database will identify the desensitized column and determine whether the current user has the permission to read the plaintext. If not, the query of the desensitized data will be rewritten and replaced with a query containing the desensitization function. When the query is executed, it will be combined with the distribution rules of the distributed table. The query task is split into multiple subtasks and distributed to the nodes storing the relevant data shards.

[0052] Step S106: On different data nodes where several data shards are located, perform distributed desensitization operations on the sensitive data columns involved in the target query data through desensitization functions respectively.

[0053] Step S106 specifically includes the following steps:

[0054] Step S1061: Detect and identify the SQL query statement input by the user. When it is identified that the SQL query statement involves sensitive data columns, determine whether the user has the permission to read the plaintext of the sensitive data columns.

[0055] Step S1062: If the user does not have the permission to read the plaintext of the sensitive data columns, call the desensitization functions bound to the sensitive data columns on different data nodes where several data shards are located, and perform distributed desensitization operations on the sensitive data columns involved in the target query data respectively.

[0056] Step S1063: If the user does not have the permission to read the plaintext of the sensitive data columns and calculations need to be performed on the sensitive data columns, first perform the calculations on the sensitive data columns on different data nodes where several data shards are located, and then perform distributed desensitization operations on the sensitive data columns involved in the target query data through desensitization functions respectively.

[0057] It should be noted that if the query needs to perform calculations on sensitive data columns (such as aggregation, sorting, etc.), desensitization is not performed temporarily, and the original data is retained for calculation. If the query only needs to return desensitized data (such as presenting to the user), the desensitization function is executed on the calculation nodes. Each node independently executes the desensitization function to desensitize the local stored data shards. After each node completes the desensitization process, the results are returned to the coordination node, and the coordination node summarizes the results and returns them to the user.

[0058] Through the above steps in the embodiments of the present application, the desensitization function is directly created in the database, reducing the learning cost of data desensitization in the database. At the same time, by utilizing the data sharding characteristics of the database, the desensitization tasks are distributed to multiple corresponding data nodes for parallel execution, significantly improving the efficiency of large-scale data desensitization. Moreover, the desensitization tasks are completed in the database kernel, and the sensitive data columns are desensitized before being transmitted to the middleware or application program, reducing the risk of data leakage during the transmission process, and solving the problem of how to implement data desensitization in the database.

[0059] In some of the embodiments, the method further includes:

[0060] Modify the desensitization logic and / or desensitization object of the created desensitization function through the script language of the preset database, so as to obtain the modified desensitization function;

[0061] Start or disable the created data masking function through the scripting language of the preset database.

[0062] It should be noted that, according to business requirements, the data masking strategy can be dynamically adjusted through the scripting language of the preset database. For example, the preset database is preferably the DolphinDB database, and the data masking function of the sensitive data column is modified through the scripting language statement of DolphinDB:

[0063] setTableSensitiveColumn('pt','col1',true,new_redaction)

[0064] Dynamically enable or disable the data masking function through the scripting language statement of DolphinDB:

[0065] setTableSensitiveColumn('pt','col1',false)

[0066] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0067] The embodiment of the present application provides a data masking system for a database. The system includes a creation module, a positioning module, and a distributed data masking module;

[0068] The creation module is used to create a data masking function through the scripting language of the preset database, where the data masking function is used to perform data masking processing on the sensitive data columns in the preset database;

[0069] The positioning module is used to determine several data shards where the target query data is located in the preset database according to the SQL query statement input by the user;

[0070] The distributed data masking module is used to perform distributed data masking operations on the sensitive data columns involved in the target query data through the data masking function on different data nodes where several data shards are located.

[0071] Through the creation module, positioning module, and distributed desensitization module in the embodiments of the present application, the desensitization function can be directly created in the database, reducing the learning cost of data desensitization in the database. At the same time, by utilizing the data sharding feature of the database, the desensitization tasks are distributed to multiple corresponding data nodes for parallel execution, significantly improving the efficiency of large-scale data desensitization. Moreover, the desensitization tasks are completed in the database kernel, and the sensitive data columns are desensitized before being transmitted to the middleware or application, reducing the risk of data leakage during the transmission process, and solving the problem of how to implement data desensitization in the database.

[0072] It should be noted that the above-mentioned various modules can be functional modules or program modules, which can be implemented either by software or by hardware. For the modules implemented by hardware, the above-mentioned various modules can be located in the same processor; or the above-mentioned various modules can also be located in different processors in any combined form.

[0073] This embodiment also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0074] Optionally, the above-mentioned electronic device may further include a transmission device and input / output devices. Among them, the transmission device is connected to the above-mentioned processor, and the input / output devices are connected to the above-mentioned processor.

[0075] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated herein.

[0076] In addition, in combination with the data desensitization method of the database in the above embodiments, the embodiments of the present application can be implemented by providing a storage medium. A computer program is stored on the storage medium; when the computer program is executed by a processor, it implements any one of the above-mentioned data desensitization methods of the database.

[0077] In one embodiment, a computer device is provided, and the computer device may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for data desensitization of a database. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0078] In one embodiment, Figure 2 is a schematic internal structure diagram of an electronic device according to an embodiment of the present application, as Figure 2 shown, an electronic device is provided, and the electronic device may be a server, and its internal structure diagram may be as Figure 2 shown. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected through an internal bus. Among them, the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a method for data desensitization of a database, and the database is used to store data.

[0079] Those skilled in the art can understand that Figure 2 the structure shown in

[0080] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. This computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0081] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0082] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for desensitizing data in a database, characterized in that: The method comprises: Creating a desensitization function by using a scripting language of a preset database, wherein the desensitization function is used to perform desensitization processing on sensitive data columns in the preset database; Based on the SQL query statement input by the user, determine the number of data shards where the target query data in the preset database is located; On different data nodes where the several data shards are located, distributed desensitization operations are performed on the sensitive data columns involved in the target query data through the desensitization function.

2. The method according to claim 1, characterized in that Creating desensitization functions using the scripting language of the preset database includes: The desensitization logic of the desensitization function is customized by using a scripting language of a preset database, thereby creating the desensitization function, wherein the desensitization logic includes regular expression replacement, encryption algorithm, data mask and hash algorithm.

3. The method according to claim 1, characterized in that After creating the desensitization function by using a scripting language of a preset database, the method includes: The desensitizing function is then bound to the sensitive data column through a scripting language of a preset database, wherein the binding is used to call the desensitizing function to perform desensitization processing on the sensitive data column when a user performs a data query.

4. The method according to claim 1, characterized in that: After determining the number of data shards where the target query data is located in the preset database based on the SQL query statement input by the user, the method includes: The SQL query statement input by the user is detected and identified, and when it is identified that the SQL query statement involves a sensitive data column, it is determined whether the user has the authority to read the plain text of the sensitive data column.

5. The method according to claim 4, characterized in that On different data nodes where the plurality of data shards are located, performing distributed desensitization operations on sensitive data columns involved in target query data by using the desensitization function respectively includes: If the user does not have permission to read the plaintext of the sensitive data column, the desensitizing function bound to the sensitive data column is called on different data nodes where the several data shards are located, and distributed desensitizing operations are performed on the sensitive data columns involved in the target query data.

6. The method according to claim 5, characterized in that The method further comprises: If the user does not have permission to read the plaintext of the sensitive data column and needs to perform calculations on the sensitive data column, the calculations on the sensitive data column are first performed on different data nodes where the several data shards are located, and then distributed desensitization operations are performed on the sensitive data columns involved in the target query data through desensitization functions.

7. The method according to claim 1, characterized in that The method comprises: Modify the desensitization logic and / or desensitization object of the created desensitization function by using the scripting language of the preset database, thereby obtaining the modified desensitization function; Enable or disable the created desensitization function through the scripting language of the preset database.

8. The method according to claim 1, characterized in that The method comprises: The data sharding is obtained based on the distributed table built into the preset database, wherein the sharding rules specified by the distributed table include vertical sharding, horizontal sharding, hash sharding, range sharding and mixed sharding.

9. A database data desensitization system, characterized in that: The system includes a creation module, a positioning module and a distributed desensitization module; The creation module is used to create a desensitization function by using a scripting language of a preset database, wherein the desensitization function is used to perform desensitization processing on sensitive data columns in the preset database; The positioning module is used to determine the number of data shards where the target query data in the preset database is located according to the SQL query statement input by the user; The distributed desensitization module is used to perform distributed desensitization operations on the sensitive data columns involved in the target query data through the desensitization function on different data nodes where the multiple data shards are located.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images