A controllable concurrent data desensitization method and device based on Chinese-English sensitive dictionary

Abstract

The present invention proposes a controllable concurrent data desensitization method and device based on a Chinese-English sensitive dictionary. The method comprises the following steps: obtaining the data to be desensitized from the metadata library, and synchronizing the table structure of the data to be desensitized to the metadata table in the parameter library; configuring the Chinese-English sensitive dictionary table; associating the metadata table with the Chinese-English sensitive dictionary table, obtaining the table name of the metadata table with sensitive fields and storing it in file A; looping through the table names in file A, creating an isomorphic table corresponding to file A, obtaining the table structure of the isomorphic table and storing it in file B; obtaining the fields in file B and comparing them with the Chinese-English sensitive dictionary table, performing desensitization conversion according to the sensitive word type, and generating a desensitization script; inputting concurrent parameters, and executing the desensitization script concurrently to complete data desensitization. The present invention changes the value of the data while preserving its original characteristics, so that the data can still be used normally by the business while avoiding the risk of data leakage.

Description

Technical Field

[0001] The present invention relates to the technical field of data desensitization, and in particular to a controllable concurrent data desensitization method and device based on a Chinese-English sensitive dictionary. Background Art

[0002] With the rapid development of banking services, banks' production databases store a vast amount of sensitive information generated by production systems, such as depositors' personal identification information, mobile phone numbers, ID cards, bank account information, and financial information. This data is used in many banking scenarios, including business analysis, development and testing, audit and oversight, and even in some outsourced services, all utilizing authentic business data and information. Leakage or corruption of this data would not only result in financial losses for the bank but, more importantly, significantly undermine customer trust in the bank. Therefore, ensuring the security of this data while maintaining its use has become a critical issue for banks.

[0003] Currently, to protect the real data used during development and testing, banks manually fabricate data to provide it to non-production environments. This method often fails to guarantee data integrity, authenticity, or data relationships. Manual processing incurs additional labor costs and creates difficulties in standardizing processes. Furthermore, banks distribute some desensitized data to development environments for testing and development. However, as business expands, new applications in more development environments are no longer suitable for testing with this outdated data, necessitating the urgent need to update core data in production environments. Summary of the Invention

[0004] In view of the above problems, the present invention provides a controllable concurrent data desensitization method and device based on a Chinese-English sensitive dictionary, which completes the desensitization of key production system data through an efficient and automated desensitization program, while preserving the original characteristics of the data while changing its value, so that the data can still be used normally by the business while avoiding the risk of data leakage.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is: a controllable concurrent data desensitization method based on a Chinese-English sensitive dictionary, comprising the following steps: obtaining the data to be desensitized from the metadata library, and synchronizing the table structure of the data to be desensitized to the metadata table in the parameter library, wherein the field columns of the metadata table include a table name column, a table description column, a field order column, a field name column, a field type column, and a field description column; configuring a Chinese-English sensitive dictionary table in the parameter library, wherein the field columns of the Chinese-English sensitive dictionary table include a sensitive type column, a sensitive word column, and a sensitive word type column; associating the metadata table with the Chinese-English sensitive dictionary table, obtaining the table name of the metadata table where the sensitive field exists and storing it in file A; looping through the table names in file A, creating an isomorphic table corresponding to file A in the sandbox library, obtaining the table structure of the isomorphic table and storing it in file B; obtaining the fields in file B and comparing them with the Chinese-English sensitive dictionary table, performing desensitization conversion according to the sensitive word type, and generating a desensitization script; inputting concurrent parameters, and executing the desensitization script concurrently to complete data desensitization.

[0006] As a preferred solution, it also includes: configuring an incremental parameter table in the parameter library, the field columns of the incremental parameter table include a table name column, a table description column and a desensitization days column, then after generating the desensitization script, it also includes: traversing the table names in the file A, associating and matching with the incremental parameter table; if no match is found, generating a single-partition desensitization script in combination with the desensitization script; if a match is found, obtaining the desensitization days of the incremental parameter table, and generating a multi-partition desensitization script in combination with the desensitization script.

[0007] As a preferred solution, the sensitive type column of the Chinese-English sensitive dictionary table includes Chinese and English categories, and the table name of the metadata table with the sensitive field is obtained and stored in file A, including: using structured query language to associate the metadata table and the Chinese-English sensitive dictionary table, selecting the Chinese category for sensitive type, and judging whether the field name column of the metadata table contains sensitive words in the sensitive word column of the Chinese-English sensitive dictionary table; if so, exporting the table name of the metadata table to file A; otherwise, proceeding to the next step; using structured query language to associate the metadata table and the Chinese-English sensitive dictionary table, selecting the English category for sensitive type, and judging whether the field name column of the metadata table exists in the sensitive word column of the Chinese-English sensitive dictionary table; if so, exporting the table name of the metadata table to file A; otherwise, not desensitizing the metadata table.

[0008] As a preferred solution, the method of obtaining the table structure of the isomorphic table and storing it in file B further includes: pre-processing the table structure of the isomorphic table by using the regular expression "sed's / [][]* / ; / g'" and then storing it in file B.

[0009] As a preferred solution, the fields in file B are obtained and compared with the Chinese and English sensitive dictionary tables, including: looping through file B to obtain the original fields of each row; obtaining the first element separated by a semicolon in the original field to obtain a field name column; obtaining the third element separated by a semicolon in the original field to obtain a field description column; accurately matching the field name column with the English sensitive word column in the Chinese and English sensitive dictionary tables, or fuzzy matching the field description column with the Chinese sensitive word column in the Chinese and English sensitive dictionary tables to determine whether the field name column or the field description column is a sensitive field; if so, synchronously obtaining the matching sensitive word type; otherwise, the original field is directly mapped without desensitization.

[0010] As a preferred solution, the desensitization conversion according to the sensitive word type includes: if the sensitive word type is a name type, the field column is desensitized and rewritten according to the name rule, the name rule is to truncate the first digit of the field column and splice "field length - 1" "*"; if the sensitive word type is a certificate type, the field column is desensitized and rewritten according to the certificate rule, the certificate rule is to truncate the first 6 digits of the field column, splice 8 "*" in the middle, and then truncate the last 4 digits and splice. If the sensitive word type is a custom type, the field column is desensitized and rewritten according to the custom rule.

[0011] As a preferred solution, the multiple concurrent executions of the desensitizing script complete data desensitization, including: creating and naming a pipe file, the pipe file name is the process number; creating a file descriptor, and associating the pipe file in a readable and writable manner; putting a token mark into the pipe file; collecting the token mark from the pipe file, entering the desensitization module, calling and executing the desensitization script, and after execution, putting the token mark back into the pipe file for new tasks to collect the token mark, and continuously maintaining the concurrency; after all desensitization script loops are executed, closing the reading and writing of the file descriptor.

[0012] A controllable concurrent data desensitization device based on a Chinese-English sensitive dictionary comprises: an acquisition synchronization module for acquiring data to be desensitized from a metadata database, and synchronizing the table structure of the data to be desensitized to a metadata table in a parameter library, wherein the field columns of the metadata table include a table name column, a table description column, a field order column, a field name column, a field type column, and a field description column; a first configuration module for configuring a Chinese-English sensitive dictionary table in the parameter library, wherein the field columns of the Chinese-English sensitive dictionary table include a sensitive type column, a sensitive word column, and a sensitive word type column; an association acquisition module for associating the metadata table with the sensitive word column; Data table and Chinese-English sensitive dictionary table, obtain the table name of the metadata table with sensitive fields and store it in file A; traversal creation module, used to loop through the table names in file A, create an isomorphic table corresponding to file A in the sandbox library, obtain the table structure of the isomorphic table and store it in file B; script generation module, used to obtain the fields in file B and compare them with the Chinese-English sensitive dictionary table, perform desensitization conversion according to the sensitive word type, and generate a desensitization script; concurrent desensitization module, input concurrency parameters, and execute the desensitization script concurrently to complete data desensitization.

[0013] As a preferred solution, it also includes a second configuration module and a partitioning module. The second configuration module is used to configure an incremental parameter table in the parameter library. The field columns of the incremental parameter table include a table name column, a table description column, and a desensitization days column; the partitioning module is used to traverse the table names in the file A and associate and match them with the incremental parameter table; if no match is found, a single-partition desensitization script is generated in combination with the desensitization script; if a match is found, the desensitization days of the incremental parameter table are obtained, and a multi-partition desensitization script is generated in combination with the desensitization script.

[0014] As a preferred solution, the sensitive type column of the Chinese-English sensitive dictionary table includes Chinese and English categories, and the association acquisition module includes a Chinese export unit and an English export unit; the Chinese export unit is used to associate the metadata table and the Chinese-English sensitive dictionary table through a structured query language, select the Chinese category as the sensitive type, and judge whether the field name column of the metadata table contains sensitive words in the sensitive word column of the Chinese-English sensitive dictionary table; the English export unit is used to associate the metadata table and the Chinese-English sensitive dictionary table through a structured query language, select the English category as the sensitive type, and judge whether the field name column of the metadata table exists in the sensitive word column of the Chinese-English sensitive dictionary table.

[0015] Compared with the prior art, the beneficial effects of the present invention include: through the Chinese and English sensitive dictionary, the field information required for desensitization is classified into Chinese and English, and the similarity characteristics of the Chinese and English descriptions are creatively used to automatically perform data desensitization conversion of sensitive fields, and support the continuous replenishment of the vocabulary and sensitive types; the numerous tables produced are classified, the desensitization day range is set as needed, and desensitization is performed according to the required range, which greatly improves work efficiency and data security. The present invention completes the desensitization of key production system data through an efficient and automated desensitization program, changes its value while preserving the original characteristics of the data, so that the data can still be used normally by the business while avoiding the risk of data leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The disclosure of the present invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the accompanying drawings, the same reference numerals are used to refer to the same components. Among them:

[0017] Figure 1 Schematic diagram of the flow of a controllable concurrent data desensitization method based on a Chinese-English sensitive dictionary according to an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a process for associating and matching with an incremental parameter table according to an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of a process for generating a desensitization script according to an embodiment of the present invention;

[0020] Figure 4 This is a flowchart of multiple concurrent executions of desensitization scripts according to an embodiment of the present invention;

[0021] Figure 5 Schematic diagram of the structure of a controllable concurrent data desensitization device based on a Chinese-English sensitive dictionary according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention and should not be regarded as the entire invention or as a limitation or restriction of the technical solution of the present invention.

[0023] According to one embodiment of the present invention, Figure 1 A controllable concurrent data desensitization method based on a Chinese-English sensitive dictionary includes the following steps:

[0024] S100: Obtain the data to be desensitized from the metadata library and synchronize the table structure of the data to be desensitized to the metadata table in the parameter library. The field columns of the metadata table include table name column, table description column, field order column, field name column, field type column and field description column.

[0025] S200: Configure a Chinese and English sensitive dictionary table in the parameter library. The field columns of the Chinese and English sensitive dictionary table include a sensitive type column, a sensitive word column, and a sensitive word type column.

[0026] Furthermore, an incremental parameter table is configured in the parameter library, and the field columns of the incremental parameter table include a table name column, a table description column, and a desensitization days column.

[0027] In this embodiment of the present invention, the sensitive type columns of the Chinese-English sensitive dictionary table include two categories: English (EN) and Chinese (CH). The English category is used to perform an exact match between the sensitive word column and the field name column. An exact match means that the contents of the sensitive word column and the field name column are completely consistent. The Chinese category is used to perform a fuzzy match between the sensitive word column and the field description column. A fuzzy match means that the contents of the field description column contain the contents of the sensitive word column.

[0028] Sensitive word lists include detailed sensitive words for English and Chinese categories, such as "CUST_NAME" for English and "Customer Name" for Chinese. Sensitive word types are categorized based on actual sensitive scenarios. For example, the English sensitive word "CUST_NAME" falls under the "name" category, "tele_no" falls under the "phone" category, and the Chinese sensitive word "Customer Name" also falls under the "name" category. The Chinese and English sensitive dictionary tables are continuously updated, and sensitive types can be added as needed.

[0029] The above incremental parameter table configures the table information that needs to be desensitized for multiple partitions according to the required desensitization days. The table name column is the name of the table to be desensitized and is used for association conditions. The table description is an optional reference field used to describe the meaning of the table. The desensitization days column defines the number of partitions that need to be desensitized for the table.

[0030] S300: Associating the metadata table with the Chinese and English sensitive dictionary tables, obtaining the table name of the metadata table containing sensitive fields and storing it in file A.

[0031] Specifically, step S300 includes the following steps:

[0032] (1) Using structured query language (SQL), the metadata table and the Chinese-English sensitive dictionary table are linked. The Chinese category is selected as the sensitive type. A determination is made as to whether the field name column of the metadata table contains sensitive words in the sensitive word column of the Chinese-English sensitive dictionary table. If so, the table name of the metadata table is exported to file A. Otherwise, the next step is performed. The structured query language is an SQL statement.

[0033] (2) Using structured query language, the metadata table and the Chinese and English sensitive dictionary tables are associated, and the English class is selected as the sensitive type. It is determined whether the field name column of the metadata table exists in the sensitive word column of the Chinese and English sensitive dictionary table. If so, the table name of the metadata table is exported to file A. Otherwise, the metadata table is not desensitized.

[0034] S400 , looping through the table names in file A, creating an isomorphic table corresponding to file A in the sandbox library, obtaining the table structure of the isomorphic table and storing it in file B.

[0035] The method further includes: pre-processing the table structure of the isomorphic table by using a regular expression "sed's / [][]* / ; / g'" and then storing the pre-processed table structure in file B.

[0036] For example, obtain the name of the table to be masked, tbl_name, from file A. Create a homogeneous table in the masking database sandbox library. Run "use sandbox;" to switch to the library and execute "create table tbl_name_mask like tbl_name;" to create the homogeneous table. Execute "desc sandbox.tbl_name;" to obtain the table structure and store it in the file tbl_name_1.txt.

[0037] The sed regular expression "cat tbl_name_1.txt|sed's / [][]* / ; / g'>tbl_name_2.txt" replaces multiple spaces in the tbl_name_1.txt file with semicolons, preprocessing the format of the table field file. For example, if the content of a field line is "cust_name varchar(30)'Customer Name'", it will be replaced with "cust_name;varchar(30);'Customer Name';". After the replacement, the semicolon ";" is used as a delimiter to identify each separated content.

[0038] S500: Obtain the fields in file B and compare them with the Chinese and English sensitive dictionary tables, perform desensitization conversion according to the sensitive word type, and generate a desensitization script.

[0039] See also Figure 2After generating the desensitization script, the process also includes: traversing the table names in file A and matching them with the incremental parameter table. If no match is found, the desensitization script is combined with the desensitization script to generate a single-partition desensitization script. If a match is found, the desensitization days for the incremental parameter table are obtained and combined with the desensitization script to generate a multi-partition desensitization script.

[0040] See also Figure 3 , the above step S500 includes the following steps:

[0041] (1) Loop through file B and obtain the original fields of each row. Get the first element in the original field separated by a semicolon to obtain the field name column. Get the third element in the original field separated by a semicolon to obtain the field description column.

[0042] For example, a "while read src_col_all" loop reads file B to obtain the original field information for each line, src_col_all. Using the awk syntax "$(echo${src_col_all}|awk–F\;'{print$1}')" retrieves the first element in the row, separated by a semicolon, to obtain the field name column, src_col_en. "$(echo${src_col_all}|awk–F\;'{print$3}')" retrieves the third element, separated by a semicolon, to obtain the field description column, src_col_ch.

[0043] (2) Accurately match the field name column with the English sensitive word column in the Chinese and English sensitive dictionary table.

[0044] For example, "if[${src_col_en} == ${mask_col_en}]" indicates an exact match between the field name column src_col_en and the English sensitive word column mask_col_en in the Chinese and English sensitive dictionary files. Exact matching determines whether the contents of the sensitive word column and the field name column are completely consistent.

[0045] (3) Perform fuzzy matching on the field description column and the Chinese sensitive word column in the Chinese and English sensitive dictionary table.

[0046] For example, "if[${src_col_ch}==*${mask_col_ch}*]" indicates that the field description column is used for fuzzy matching with Chinese sensitive words in the Chinese and English sensitive field files. Fuzzy matching determines whether the content of the field description column contains the content of the sensitive word column.

[0047] (4) Determine whether the field name column or field description column is a sensitive field. If so, obtain the matching sensitive word type mask_type synchronously. Otherwise, the field column does not need to be desensitized and is directly mapped.

[0048] (5) If the sensitive word type is a name type, the field column is desensitized and rewritten according to the name rule. The name rule is to cut off the first digit of the field column and splice "field length - 1" "*".

[0049] For example, the "if[[${mask_type}=="name"]]" if statement combined with a regular expression determines whether the sensitive type is "name." "rpad(substr(${src_col_en},1,1),length(${src_col_en}),'*')as${src_col_en},"" truncates the first digit of the source field column and appends (field length - 1) "*" characters. For example, if the original data value of the field column is "12345," the converted value is "1****."

[0050] (6) If the sensitive word type is a certificate type, the field column is desensitized and rewritten according to the certificate rule. The certificate rule is to cut off the first 6 digits of the field column, splice 8 "*" in the middle, and then cut off the last 4 digits and splice them together.

[0051] For example, the elif statement, combined with a regular expression, checks whether the sensitive type is "cert." The substr(substr(${src_col_en},1,6),'********',substr(${src_col_en},15,4))as${src_col_en},” command truncates the first six digits of the ID field, appends eight asterisks (*) in the middle, and then truncates the last four digits to create a desensitized field. For example, if the original value of a field column is "320101199001011234," the converted value is "320101********1234."

[0052] (7) If the sensitive word type is a custom class, the field column will be desensitized and rewritten according to the custom rules.

[0053] For example: "elif[[${mask_type}=="XXX"]]", the elif statement is combined with a regular expression to determine whether the sensitive type is "XXX".

[0054] (8) Generate desensitization script.

[0055] At step S600, enter the concurrency parameter and execute the desensitization script concurrently to complete the data desensitization. The number of concurrent operations is obtained by entering the parameter variable.

[0056] See also Figure 4 , the above step S600 includes the following steps:

[0057] S601: Create and name a pipe file, with the pipe file name being the process number. For example, according to the pipe definition statement "fifo_file=$$.fifo; mkfifo${fifo_file}", a named pipe file is created and named according to the process number ($$.fifo).

[0058] S602: Create a file descriptor and associate it with the pipe file in both readable and writable mode. For example, using "exec4<>${fifo_file}" creates a file descriptor and associates it with the pipe file in both readable (<) and writable (>) mode. File descriptor 4 now has all the properties of a named pipe file. "rm -f${fifo_file}" removes the pipe file, leaving the file descriptor unused.

[0059] S603: Add a token to the pipeline file. For example, "{count = ${parallel}; while [${count} – gt 0]; do echo; let count = ${count} - 1; done;}>&4" is used. &4 refers to file descriptor 4. This command adds a token to the pipeline.

[0060] S604: Receive a token from the pipe file, enter the desensitization module, and call and execute the desensitization script. For example, "read<&4" reads (<) a token from the pipe of the file descriptor (&4) and enters the desensitization module; "echo>&4" indicates calling and executing the desensitization script.

[0061] S605, after the execution is completed, the token mark is put back into the pipeline file for new tasks to collect the token mark and continuously maintain the concurrency.

[0062] S606: After all the desensitization scripts have finished executing in a loop, close the read and write functions of the file descriptor. Execute "exec4<&-" to close the read function of the file descriptor, and "exec 4>&-" to close the write function of the file descriptor.

[0063] See also Figure 5 The present invention also discloses a controllable concurrent data desensitization device based on a Chinese-English sensitive dictionary, comprising:

[0064] Acquisition synchronization module 100 is used to obtain the data to be desensitized in the metadata database and synchronize the table structure of the data to be desensitized to the metadata table in the parameter library. The field columns of the metadata table include table name column, table description column, field order column, field name column, field type column and field description column.

[0065] The first configuration module 200 is used to configure a Chinese-English sensitive dictionary table in the parameter library. The field columns of the Chinese-English sensitive dictionary table include a sensitive type column, a sensitive word column, and a sensitive word type column.

[0066] The association acquisition module 300 is used to associate the metadata table with the Chinese and English sensitive dictionary table, obtain the table name of the metadata table containing sensitive fields, and store it in file A.

[0067] The traversal creation module 400 is used to loop through the table names in file A, create an isomorphic table corresponding to file A in the sandbox library, obtain the table structure of the isomorphic table and store it in file B.

[0068] The script generation module 500 is used to obtain the fields in file B and compare them with the Chinese and English sensitive dictionary tables, perform desensitization conversion according to the sensitive word type, and generate a desensitization script.

[0069] The concurrent desensitization module 600 inputs concurrent parameters and executes the desensitization scripts concurrently to complete data desensitization.

[0070] Furthermore, the device also includes a second configuration module and a partition module. The second configuration module is used to configure an incremental parameter table in the parameter library. The field columns of the incremental parameter table include a table name column, a table description column, and a desensitization days column.

[0071] The partitioning module traverses the table names in File A and associates them with the incremental parameter table. If no match is found, the module generates a single-partition desensitization script based on the desensitization script. If a match is found, the module obtains the desensitization days for the incremental parameter table and generates a multi-partition desensitization script based on the desensitization script.

[0072] In an embodiment of the present invention, the sensitive type column of the Chinese-English sensitive dictionary table includes both Chinese and English categories, and the association acquisition module 300 includes a Chinese export unit and an English export unit. The Chinese export unit is configured to associate the metadata table with the Chinese-English sensitive dictionary table using structured query language, select the Chinese category for the sensitive type, and determine whether the field name column of the metadata table contains sensitive words in the sensitive word column of the Chinese-English sensitive dictionary table. The English export unit is configured to associate the metadata table with the Chinese-English sensitive dictionary table using structured query language, select the English category for the sensitive type, and determine whether the field name column of the metadata table exists in the sensitive word column of the Chinese-English sensitive dictionary table.

[0073] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0074] In summary, the beneficial effects of the present invention include: through the Chinese and English sensitive dictionary, the field information that needs to be desensitized is classified into Chinese and English, and the similarity characteristics of the Chinese and English descriptions are creatively used to automatically perform data desensitization conversion of sensitive fields, and support the continuous replenishment of the vocabulary and sensitive types. The numerous tables produced are classified, the desensitization day range is set as needed, and desensitization is performed according to the required range, which greatly improves work efficiency and data security. The present invention completes the desensitization of key production system data through an efficient and automated desensitization program, changes its value while preserving the original characteristics of the data, so that the data can still be used normally by the business, while avoiding the risk of data leakage.

[0075] It should be understood that if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0076] The technical scope of the present invention is not limited to the contents of the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A controllable concurrent data desensitization method based on Chinese-English sensitive dictionary, characterized in that: The steps include: Obtain the data to be desensitized from the metadata database, and synchronize the table structure of the data to be desensitized to the metadata table in the parameter library, where the field columns of the metadata table include a table name column, a table description column, a field order column, a field name column, a field type column, and a field description column; A Chinese-English sensitive dictionary table is configured in the parameter library, wherein the field columns of the Chinese-English sensitive dictionary table include a sensitive type column, a sensitive word column, and a sensitive word type column; Associating the metadata table with the Chinese and English sensitive dictionary tables, obtaining the table name of the metadata table containing sensitive fields and storing it in file A; Loop through the table names in file A, create an isomorphic table corresponding to file A in the sandbox library, obtain the table structure of the isomorphic table and store it in file B; Obtain the fields in the file B and compare them with the Chinese and English sensitive dictionary tables, perform desensitization conversion according to the sensitive word type, and generate a desensitization script; Enter the concurrency parameters and execute the desensitization script multiple times concurrently to complete data desensitization.

2. The controllable concurrent data desensitization method based on the Chinese-English sensitive dictionary according to claim 1 is characterized in that: Also includes: An incremental parameter table is configured in the parameter library. The field columns of the incremental parameter table include a table name column, a table description column, and a desensitization days column. The desensitization days column defines the number of partitions that need to be desensitized for the table. After generating the desensitization script, the following is also included: Traverse the table names in the file A and associate and match them with the incremental parameter table; If no match is found, a single partition desensitization script is generated in combination with the desensitization script; If a match is found, the desensitization days of the incremental parameter table are obtained, and a multi-partition desensitization script is generated in combination with the desensitization script.

3. The controllable concurrent data desensitization method based on the Chinese-English sensitive dictionary according to claim 1 is characterized in that: The sensitive type column of the Chinese-English sensitive dictionary table includes Chinese and English categories, so the table name of the metadata table with sensitive fields is obtained and stored in file A, including: Using structured query language, the metadata table is associated with the Chinese and English sensitive dictionary tables, with the Chinese category selected as the sensitive type. A determination is made as to whether the field name column of the metadata table contains sensitive words in the sensitive word column of the Chinese and English sensitive dictionary tables. If so, the table name of the metadata table is exported to file A. Otherwise, the process proceeds to the next step. Through structured query language, the metadata table and the Chinese-English sensitive dictionary table are associated, the English class is selected as the sensitive type, and it is determined whether the field name column of the metadata table exists in the sensitive word column of the Chinese-English sensitive dictionary table. If so, the table name of the metadata table is exported to file A. Otherwise, the metadata table is not desensitized.

4. The controllable concurrent data desensitization method based on the Chinese-English sensitive dictionary according to claim 1 is characterized in that: The obtaining of the table structure of the isomorphic table and storing it in file B further includes: pre-processing the table structure of the isomorphic table by using a regular expression "sed 's / [ ][ ]* / ; / g'" and then storing it in file B.

5. The controllable concurrent data desensitization method based on the Chinese-English sensitive dictionary according to claim 1 is characterized in that: Obtaining the fields in file B and comparing them with the Chinese and English sensitive dictionary tables includes: Read the file B in a loop to obtain the original fields of each line; Get the first element separated by a semicolon in the original field to get the field name column; Get the third element separated by semicolons in the original field to get the field description column; The field name column is precisely matched with the English sensitive word column in the Chinese-English sensitive dictionary table, or the field description column is fuzzy matched with the Chinese sensitive word column in the Chinese-English sensitive dictionary table to determine whether the field name column or field description column is a sensitive field. If so, the matching sensitive word type is obtained synchronously. Otherwise, the original field is directly mapped without desensitization.

6. The controllable concurrent data desensitization method based on the Chinese-English sensitive dictionary according to claim 1 is characterized in that: The desensitization conversion according to the sensitive word type includes: If the sensitive word type is name type, the field column is desensitized and rewritten according to the name rule. The name rule is to truncate the first digit of the field column and concatenate "field length - 1" and "*"; If the sensitive word type is certificate type, the field column is desensitized and rewritten according to the certificate rule. The certificate rule is to truncate the first 6 digits of the field column, splice 8 "*" in the middle, and then truncate the last 4 digits; If the sensitive word type is a custom type, the field column is desensitized and rewritten according to the custom rules.

7. The controllable concurrent data desensitization method based on the Chinese-English sensitive dictionary according to claim 1 is characterized in that: The multiple concurrent executions of the desensitization scripts to complete data desensitization include: Create and name a pipeline file, where the pipeline file name is the process number; Create a file descriptor to associate the pipe file in a readable and writable manner; Putting a token into the pipeline file; Get the token from the pipeline file, enter the desensitization module, call and execute the desensitization script; After execution is completed, the token mark is put back into the pipeline file for new tasks to collect the token mark and maintain the concurrency; After all desensitization scripts are executed in a loop, close the reading and writing of the file descriptor.

8. A controllable concurrent data desensitization device based on a Chinese-English sensitive dictionary, characterized in that: include: An acquisition synchronization module is used to obtain the data to be desensitized from the metadata library and synchronize the table structure of the data to be desensitized to the metadata table in the parameter library. The field columns of the metadata table include a table name column, a table description column, a field order column, a field name column, a field type column, and a field description column. A first configuration module is configured to configure a Chinese-English sensitive dictionary table in the parameter library, wherein the field columns of the Chinese-English sensitive dictionary table include a sensitive type column, a sensitive word column, and a sensitive word type column; An association acquisition module, configured to associate the metadata table with the Chinese and English sensitive dictionary tables, obtain the table name of the metadata table containing sensitive fields, and store the table name in file A; A traversal creation module is used to loop through the table names in the file A, create an isomorphic table corresponding to the file A in the sandbox library, obtain the table structure of the isomorphic table and store it in the file B; A script generation module is used to obtain the fields in the file B and compare them with the Chinese and English sensitive dictionary tables, perform desensitization conversion according to the sensitive word type, and generate a desensitization script; The concurrent desensitization module inputs concurrent parameters and executes the desensitization script multiple times concurrently to complete data desensitization.

9. The controllable concurrent data desensitization device based on the Chinese-English sensitive dictionary according to claim 8 is characterized in that: It also includes a second configuration module and a partition module, the second configuration module is used to configure an incremental parameter table in the parameter library, the field columns of the incremental parameter table include a table name column, a table description column and a desensitization days column, and the desensitization days column is used to define the number of partitions that need to be desensitized for the table; The partitioning module is used to traverse the table names in the file A and associate and match them with the incremental parameter table; if no match is found, a single partition desensitization script is generated in combination with the desensitization script; If a match is found, the desensitization days of the incremental parameter table are obtained, and a multi-partition desensitization script is generated in combination with the desensitization script.

10. The controllable concurrent data desensitization device based on the Chinese-English sensitive dictionary according to claim 9 is characterized in that: The sensitive type column of the Chinese-English sensitive dictionary table includes Chinese and English categories, and the association acquisition module includes a Chinese export unit and an English export unit; The Chinese export unit is used to associate the metadata table with the Chinese-English sensitive dictionary table through structured query language, select the Chinese category as the sensitive type, and determine whether the field name column of the metadata table contains sensitive words in the sensitive word column of the Chinese-English sensitive dictionary table; The English export unit is used to associate the metadata table with the Chinese-English sensitive dictionary table through structured query language, select the English class as the sensitive type, and determine whether the field name column of the metadata table exists in the sensitive word column of the Chinese-English sensitive dictionary table.

Images