A sensitive data encryption method based on ODPS offline data warehouse

Abstract

The present invention relates to the technical field of data security encryption, and specifically to a sensitive data encryption method based on an ODPS offline data warehouse. A DataWorks project space is created separately to store Jar package resources required by UDF, and only the highest authority person in the project space can access it. Through the form of management and control of resource Jar packages, users only need to know the function name, and can use the encryption and decryption function by calling the function. By modifying the encryption and decryption function, it is first determined whether it is plain text or ciphertext before encryption and decryption, and then it is decided whether to perform encryption and decryption. A new ODPS space is created to store encryption and decryption function resources, and the ODPS space authority control only allows the authority approver to access and grant authority. The present invention can create a separate project space to store encryption and decryption function resources, so as to conveniently and effectively isolate the user's operation on the original encryption and decryption logic and key, and control the function usage authority.

Description

Technical Field

[0001] The present invention relates to the technical field of data security encryption, and specifically to a sensitive data encryption method based on ODPS offline data warehouse. Background Art

[0002] At present, small and medium-sized enterprises do not have a deep understanding of data security or do not attach great importance to data security. In the early stage of development, small and medium-sized enterprises focus on business development and take business development as the primary goal. Currently, there are many kinds of encryption algorithms in the industry, and the encryption methods are also different.

[0003] The common technical solution in the current industry situation is to add an encrypted ciphertext field, then process the plaintext field downstream, and subsequently switch the logical processing using the ciphertext field. In fact, the data is transitioned from plaintext to ciphertext by making a field redundant, and finally the plaintext field is left blank. Summary of the invention

[0004] In view of this, the embodiment of the present invention hopes to provide a sensitive data encryption method based on ODPS offline data warehouse to solve or alleviate the following technical problems:

[0005] (1) The existing technology uses a method of adding a new ciphertext field (such as mobile_enc) to perform field backup transition, exchanging storage redundancy for computation. The problem is that the field cannot be deleted and needs to be retained all the time. A mobile field needs to be represented by two fields, which increases maintenance costs.

[0006] (2) There is a risk of secondary encryption and decryption (encryption based on the ciphertext) in the data processing logic of the existing technology. For downstream use, it is difficult to identify which data needs to be decrypted several times.

[0007] (3) The existing technology adds encrypted fields. First, each field that needs to be encrypted must be processed by adding a ciphertext field. Then, the logic of each output task is sorted out to determine whether there is any association, filtering, or other operations. After that, the original plaintext field is set to empty. This processing method has a complex process, high work cost, and requires frequent changes to online tables and online tasks, which is highly risky.

[0008] (4) The prior art directly uses the UDF function for encryption and decryption operations, which easily leaks the logic and key of the encryption and decryption functions (such as decompiling after obtaining the Jar package), and the security is not high.

[0009] The technical problems existing in the prior art at least provide a beneficial option for the above-mentioned technical problems.

[0010] To achieve the above object, the present invention provides the following technical solutions:

[0011] A sensitive data encryption method based on ODPS offline data warehouse, the method includes the following steps:

[0012] S1. Create a separate DataWorks project space to store the Jar package resources required by UDF. Only the person with the highest authority can access the project space.

[0013] S2. Through the management and control of resource Jar packages, users only need to know the function name and can use the encryption and decryption functions by calling the function;

[0014] S3, by modifying the encryption and decryption functions, before encryption and decryption, first determine whether it is plaintext or ciphertext, and then decide whether to encrypt and decrypt;

[0015] S4. Create a new ODPS space to store encryption and decryption function resources. The ODPS space permission control only allows the permission approver to access and grant permissions.

[0016] S5. For the target DataWorks space of the data that needs to be encrypted, sort out the sensitive information fields that need to be encrypted in the task and encrypt the source layer data;

[0017] S6. Check whether there is any association with sensitive information fields in online tasks and perform filtering operations;

[0018] S7. Pay attention to whether there are table field deletion operations on the upstream business side. If so, inform them in advance so that the downstream data warehouse can perform field accommodation switching in time.

[0019] As a further solution of the present invention: the encryption and decryption function resources in S4 include function logic and keys.

[0020] As a further solution of the present invention: if the source layer in S5 cannot be encrypted, encryption can be performed inside the middle layer of the next layer of nodes.

[0021] As a further solution of the present invention: in the filtering operation in S6, the plain text enumeration value is encrypted.

[0022] As a further solution of the present invention: in the above S6, for the operations such as association, it is determined whether both sides are ciphertexts, and the non-ciphertext table is encrypted.

[0023] As a further embodiment of the present invention, the method further comprises the following steps:

[0024] Step 1: Selectively encrypt sensitive data fields using a master key, including sensitive data, a master key, and a secondary key. The sensitive data is stored in the form of ciphertext, and the sensitive data is encrypted hierarchically at the field level. The encryption algorithm and key strength of the sensitive data column are set at multiple levels of keys. The sensitive data is encrypted in-process, and the access and transmission permissions of the sensitive data are controlled by the master key.

[0025] Step 2: Determine whether the sensitive data is plaintext sensitive data or ciphertext sensitive data;

[0026] Step 3: For plaintext sensitive data, determine the service type corresponding to the plaintext sensitive data, determine the encryption level corresponding to the service type, and obtain the encryption method and encryption key corresponding to the encryption level;

[0027] Step 4: Based on the encryption method and encryption key, encrypt the plaintext sensitive data to generate encrypted data;

[0028] Step 5: Control the user's retrieval indexing authority for sensitive data, and set different encryption settings for index items of the same record content. Perform multi-factor field access control on the encrypted index of the sensitive data. The primary key of the sensitive data and the secondary key of the encrypted index of the sensitive data are both stored in the data dictionary of the database.

[0029] Step six: the secondary key of the encrypted index of the sensitive data is managed by the administrator, and the master key for accessing and transmitting the sensitive data is managed by the first person in charge. When the user performs encrypted indexing of sensitive data, the administrator is requested to obtain the indexing and decryption authority of the sensitive data. After the request is approved, the indexing authority and decryption of the sensitive data are obtained. When the user accesses and transmits sensitive data, the first person in charge is requested to obtain the corresponding access and transmission authority. After the request is approved, the database performs autonomous decryption according to the master key corresponding to the sensitive data to be accessed in the data dictionary, and allows the user to complete the access and transmission of sensitive data. The allowed operations of the administrator and the first person in charge are recorded, and the access and transmission content of the backup user are recorded.

[0030] As a further solution of the present invention: when a user accesses data, the database is monitored to identify whether sensitive data is involved. If so, the user needs to apply for access to the administrator or the first person in charge. After the administrator or the first person in charge receives and agrees, the user is allowed to query, access or transmit sensitive data stored in encrypted form. After the administrator or the first person in charge receives and refuses, the user's query, access and transmission permissions are terminated. If not, the user is allowed to query, access or transmit.

[0031] As a further solution of the present invention: decrypting the ciphertext sensitive data, including: obtaining a decryption key of a version corresponding to the ciphertext sensitive data, and decrypting the ciphertext sensitive data based on the decryption key of the version.

[0032] As a further solution of the present invention: decrypting the ciphertext sensitive data, including: obtaining a decryption key of a version corresponding to the ciphertext sensitive data, and decrypting the ciphertext sensitive data based on the decryption key of the version.

[0033] As a further solution of the present invention: using the primary key and the secondary key to complete the decryption of sensitive data, the specific operation is: using the primary key retrieved from the database to decrypt the sensitive data accessed and transmitted by the user, and using the secondary key retrieved from the database to decrypt the sensitive data retrieved by the user.

[0034] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0035] 1. This solution creates a separate project space to store the resources of encryption and decryption functions, which can easily and effectively isolate the user's operations on the original encryption and decryption logic and keys, and control the function usage permissions.

[0036] 2. This solution saves storage space by encrypting the original fields, reduces the impact of discarded fields caused by encryption and decryption, and avoids the trouble of later maintenance.

[0037] 3. This solution avoids the unpredictable impact caused by multiple encryption and decryption of data by expanding the encryption and decryption functions.

[0038] 4. This solution avoids coupling with the implementation of encryption work on the business side. It can perform encryption and decryption independently without relying on the business side, and is decoupled from the encryption work on the business side.

[0039] 5. Fine-grained hierarchical encryption: This solution performs fine-grained hierarchical encryption on sensitive data at the field level. Different encryption algorithms and key strengths can be used according to business needs and data sensitivity to better protect data security.

[0040] 6. Access control and authority management: This solution controls access and transmission permissions of sensitive data through master keys and secondary keys. Administrators can manage the encryption index of sensitive data, and the first person in charge manages the master keys for access and transmission, thereby achieving strict control over access permissions to sensitive data.

[0041] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 This is a workflow diagram of a sensitive data encryption method based on ODPS offline data warehouse proposed by the present invention;

[0044] Figure 2 This is a workflow diagram of an embodiment of a sensitive data encryption method based on ODPS offline data warehouse proposed by the present invention. DETAILED DESCRIPTION

[0045] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0046] It should be noted that the terms "first", "second", "symmetrical", "array", etc. are only used to distinguish between description and position description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the definition of "first", "symmetrical", etc. can explicitly or implicitly include one or more of these features; similarly, when the quantity of certain features is not limited in the form of words such as "two" or "three", it should be noted that this feature also explicitly or implicitly includes one or more feature quantities;

[0047] like Figure 1 As shown, the present invention provides a sensitive data encryption method based on ODPS offline data warehouse, the method comprising the following steps:

[0048] S1. Create a separate DataWorks project space to store the Jar package resources required by UDF. Only the highest-privileged person can access the project space. In this method, it is recommended to create a separate DataWorks project space to store the Jar package resources required by UDF. By creating a separate project space, resources can be isolated and centrally managed. In addition, only the highest-privileged person can access the project space to ensure the security and controllability of resources.

[0049] S2. Through the management and control of resource Jar packages, users only need to know the function name and call the function to use the encryption and decryption function. Management and control of resource Jar packages: Through the management and control of resource Jar packages, users only need to know the name of the encryption and decryption function and call the corresponding function to complete the encryption and decryption operation. This method simplifies the user's operation process and improves the user experience.

[0050] S3. By modifying the encryption and decryption functions, it is recommended to first determine whether it is plaintext or ciphertext before encryption and decryption, and then decide whether to perform encryption and decryption. This method recommends modifying the logic of the encryption and decryption functions, first determining whether the data is plaintext or ciphertext, and then deciding whether to perform encryption and decryption operations. Such a modification can reduce the number of calls to the encryption and decryption functions and improve system performance;

[0051] S4. Create a new ODPS space to store encryption and decryption function resources. The ODPS space permission control only allows permission approvers to access and grant permissions. Encryption and decryption function resources include function logic and keys. In order to store encryption and decryption function resources, this method recommends creating a new ODPS space. At the same time, the permission control of the ODPS space only allows permission approvers to access and grant permissions, which can ensure the security and controllability of encryption and decryption function resources.

[0052] S5. In the target DataWorks space, sort out the data that needs to be encrypted, find the sensitive information fields that need to be encrypted, and encrypt the source layer data. If the source layer cannot be encrypted, it can be encrypted in the middle layer of the next node. In the target DataWorks space, sort out the data that needs to be encrypted, determine which fields contain sensitive information, and then encrypt these fields in the source layer or middle layer node. If the source layer cannot be encrypted, it can be encrypted in the middle layer of the next node. The encryption algorithm and key management method should be selected according to security requirements.

[0053] S6. Check whether there are any associations with sensitive information fields in online tasks, and perform filtering operations. For filtering operations, encrypt the plaintext enumeration values. For association operations, determine whether both sides are ciphertext, and encrypt the non-ciphertext tables. In online tasks, it is necessary to check whether there are any association operations with sensitive information fields, and perform corresponding filtering. For plaintext enumeration values, encryption can be performed to protect the security of sensitive information. At the same time, for association operations, it should be determined whether the two tables involved in the association are both ciphertext. If there is a non-ciphertext table, it needs to be encrypted;

[0054] S7. Pay attention to whether there are any table field deletion operations on the upstream business side. If so, inform the downstream data warehouse in advance so that the downstream data warehouse can switch the field accommodation in time. Pay attention to whether there are any table field deletion operations on the upstream business side. If so, notify the downstream data warehouse in advance so that the field accommodation switch can be carried out in time. This can ensure that the data structure of the data warehouse is consistent with the changes on the upstream business side to avoid data loss or errors.

[0055] like Figure 2 As shown, in one embodiment, the method further comprises the following steps:

[0056] Step 1: Selectively encrypt sensitive data fields using a master key, including sensitive data, a master key, and a secondary key. The sensitive data is stored in ciphertext form, and the sensitive data is encrypted at the field level in a fine-grained hierarchical manner. The encryption algorithm and key strength of the sensitive data column are set at a multi-level key, and the sensitive data is encrypted in-process. The access and transmission permissions of the sensitive data are controlled by the master key.

[0057] In this step, the master key is used to selectively encrypt sensitive data fields. First, determine which fields contain sensitive data and store the data in these fields in ciphertext. Then, set multi-level keys for the encryption algorithm and key strength of sensitive data columns to increase data security. For encryption operations, symmetric encryption algorithms (such as AES) or asymmetric encryption algorithms (such as RSA) can be used. In addition, sensitive data is encrypted within the process to ensure the security of data during transmission and access. By using the master key, access to sensitive data and transmission permissions can be controlled.

[0058] Step 2: Determine whether the sensitive data is plaintext sensitive data or ciphertext sensitive data.

[0059] In this step, it is necessary to determine whether the sensitive data is plain text or cipher text. For plain text sensitive data, encryption processing is required; for cipher text sensitive data, decryption processing is required. This determination can be completed by checking the format of the data field or a specific identifier.

[0060] Step 3: For plaintext sensitive data, determine the business type corresponding to the plaintext sensitive data, determine the encryption level corresponding to the business type, and obtain the encryption method and encryption key corresponding to the encryption level.

[0061] For plaintext sensitive data, in this step, it is necessary to determine the business type to which the plaintext sensitive data belongs, and determine the corresponding encryption level for the business type. According to the encryption level, obtain the corresponding encryption method and encryption key. The encryption method can include symmetric encryption and asymmetric encryption, and the encryption key can be pre-generated or dynamically generated.

[0062] Step 4: Based on the encryption method and encryption key, encrypt the plaintext sensitive data to generate encrypted data.

[0063] Based on the determined encryption method and encryption key, the plaintext sensitive data is encrypted to generate encrypted data. The encryption process involves data conversion, application of encryption algorithm and use of key. For example, the AES algorithm and symmetric key are used to encrypt the plaintext sensitive data.

[0064] Step 5: Control the user's retrieval indexing authority for sensitive data, and set different encryption settings for index items of the same record content. Perform multi-factor field access control on the encrypted index of the sensitive data. The primary key of the sensitive data and the secondary key of the encrypted index of the sensitive data are both stored in the data dictionary of the database.

[0065] In this step, it is necessary to control the user's retrieval and indexing permissions for sensitive data. To increase security, different encryption settings are used for index items with the same record content. Access control for multi-factor fields can be achieved through the use of access tokens, authentication, and access policies. The primary key and secondary key are stored in the database's data dictionary to ensure their security and manageability.

[0066] Step six: the secondary key of the encrypted index of the sensitive data is managed by the administrator, and the master key for accessing and transmitting the sensitive data is managed by the first person in charge. When the user performs encrypted indexing of sensitive data, the administrator is requested to obtain the indexing and decryption authority of the sensitive data. After the request is approved, the indexing authority and decryption of the sensitive data are obtained. When the user accesses and transmits sensitive data, the first person in charge is requested to obtain the corresponding access and transmission authority. After the request is approved, the database performs autonomous decryption according to the master key corresponding to the sensitive data to be accessed in the data dictionary, and allows the user to complete the access and transmission of sensitive data. The allowed operations of the administrator and the first person in charge are recorded, and the access and transmission content of the backup user are recorded.

[0067] In this step, the administrator is responsible for managing the secondary key of the encrypted index of sensitive data, while the first person in charge is responsible for managing the master key for accessing and transmitting sensitive data. When a user needs to access the encrypted index of sensitive data, he needs to request the administrator to obtain the index and decryption permissions. When a user needs to access and transmit sensitive data, he needs to request the first person in charge to obtain the corresponding access and transmission permissions. The database performs autonomous decryption based on the master key corresponding to the sensitive data to be accessed in the data dictionary, and allows the user to complete the access and transmission of sensitive data. The operations of the administrator and the first person in charge will be recorded, and the user's access and transmission content will be backed up.

[0068] When a user accesses data, the database is monitored to identify whether sensitive data is involved. If so, the user needs to apply for access to the administrator or the first person in charge. After the administrator or the first person in charge receives and agrees, the user is allowed to query, access or transmit sensitive data stored in encrypted form. After the administrator or the first person in charge receives and refuses, the user's query, access and transmission permissions are terminated. If not, the user is allowed to query, access or transmit.

[0069] When a user accesses data, the database will monitor and identify whether sensitive data is involved. If sensitive data is involved, the user needs to apply for access rights from the administrator or the first person in charge. After the administrator or the first person in charge receives and agrees, the user will be given query, access or transmission rights to sensitive data stored in encrypted text. If the administrator or the first person in charge receives and refuses, the user's query, access and transmission rights will be terminated. If sensitive data is not involved, the user will be allowed to query, access or transmit.

[0070] The ciphertext sensitive data is decrypted, including: obtaining a decryption key corresponding to a version of the ciphertext sensitive data, and decrypting the ciphertext sensitive data based on the decryption key of the version. The ciphertext sensitive data is decrypted, including: obtaining a decryption key corresponding to a version of the ciphertext sensitive data, and decrypting the ciphertext sensitive data based on the decryption key of the version, and decrypting the sensitive data using a primary key and a secondary key. The specific operations are: using the primary key retrieved from the database to decrypt the sensitive data accessed and transmitted by the user, and using the secondary key retrieved from the database to decrypt the sensitive data retrieved by the user.

[0071] For ciphertext sensitive data, decryption processing is required. This process includes obtaining the decryption key corresponding to the version of the ciphertext sensitive data and decrypting the ciphertext sensitive data based on the key. The decryption key can be managed according to the version and controlled according to the access rights.

[0072] The above is the overall workflow of the present invention, and you can repeat this step next time you use it.

[0073] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0074] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A sensitive data encryption method based on ODPS offline data warehouse, characterized in that: The method comprises the following steps: S1. Create a separate DataWorks project space to store the Jar package resources required by UDF. Only the person with the highest authority can access the project space. S2. Through the management and control of resource Jar packages, users only need to know the function name and can use the encryption and decryption functions by calling the function; S3, by modifying the encryption and decryption functions, before encryption and decryption, first determine whether it is plaintext or ciphertext, and then decide whether to encrypt and decrypt; S4. Create a new ODPS space to store encryption and decryption function resources. The ODPS space permission control only allows the permission approver to access and grant permissions. S5. For the target DataWorks space of the data that needs to be encrypted, sort out the sensitive information fields that need to be encrypted in the task and encrypt the source layer data; S6. Check whether there is any association with sensitive information fields in online tasks and perform filtering operations; S7. Pay attention to whether there are table field deletion operations on the upstream business side. If so, inform them in advance so that the downstream data warehouse can perform field accommodation switching in time.

2. According to claim 1, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: The encryption and decryption function resources in S4 include function logic and keys.

3. According to claim 2, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: In S5, if the source layer cannot be encrypted, encryption can be performed inside the middle layer of the next layer node.

4. According to claim 3, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: In the filtering operation in S6, the plain text enumeration value is encrypted.

5. According to claim 4, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: In the above S6, for the association operation, it is determined whether both sides are ciphertexts, and the non-ciphertext table is encrypted.

6. According to claim 5, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: The method further comprises the following steps: Step 1: Selectively encrypt sensitive data fields using a master key, including sensitive data, a master key, and a secondary key. The sensitive data is stored in the form of ciphertext, and the sensitive data is encrypted hierarchically at the field level. The encryption algorithm and key strength of the sensitive data column are set at multiple levels of keys. The sensitive data is encrypted in-process, and the access and transmission permissions of the sensitive data are controlled by the master key. Step 2: Determine whether the sensitive data is plaintext sensitive data or ciphertext sensitive data; Step 3: For plaintext sensitive data, determine the service type corresponding to the plaintext sensitive data, determine the encryption level corresponding to the service type, and obtain the encryption method and encryption key corresponding to the encryption level; Step 4: Based on the encryption method and encryption key, encrypt the plaintext sensitive data to generate encrypted data; Step 5: Control the user's retrieval indexing authority for sensitive data, and set different encryption settings for index items of the same record content. Perform multi-factor field access control on the encrypted index of the sensitive data. The primary key of the sensitive data and the secondary key of the encrypted index of the sensitive data are both stored in the data dictionary of the database. Step six: the secondary key of the encrypted index of the sensitive data is managed by the administrator, and the master key for accessing and transmitting the sensitive data is managed by the first person in charge. When the user performs encrypted indexing of sensitive data, the administrator is requested to obtain the indexing and decryption authority of the sensitive data. After the request is approved, the indexing authority and decryption of the sensitive data are obtained. When the user accesses and transmits sensitive data, the first person in charge is requested to obtain the corresponding access and transmission authority. After the request is approved, the database performs autonomous decryption according to the master key corresponding to the sensitive data to be accessed in the data dictionary, and allows the user to complete the access and transmission of sensitive data. The allowed operations of the administrator and the first person in charge are recorded, and the access and transmission content of the backup user are recorded.

7. According to claim 6, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: When a user accesses data, the database is monitored to identify whether sensitive data is involved. If so, the user needs to apply for access to the administrator or the first person in charge. After the administrator or the first person in charge receives and agrees, the user is allowed to query, access or transmit sensitive data stored in encrypted form. After the administrator or the first person in charge receives and refuses, the user's query, access and transmission permissions are terminated. If not, the user is allowed to query, access or transmit.

8. According to claim 6, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: The ciphertext sensitive data is decrypted, including: obtaining a decryption key of a version corresponding to the ciphertext sensitive data, and decrypting the ciphertext sensitive data based on the decryption key of the version.

9. According to claim 6, a sensitive data encryption method based on ODPS offline data warehouse is characterized in that: The specific operation of using the primary key and secondary key to complete the decryption of sensitive data is: using the primary key retrieved from the database to decrypt the sensitive data accessed and transmitted by the user, and using the secondary key retrieved from the database to decrypt the sensitive data retrieved by the user.

Images