Data encryption and decryption method based on big data platform

Abstract

The application discloses a data encryption and decryption method based on a big data platform, and comprises the following steps: data source access; judging whether the data source is sensitive data; if the data source is non-sensitive data, processing the data source according to a normal data specification process; if the data source is sensitive data, entering a sensitive data area; when the data in the sensitive data area is needed, according to user requirements, the sensitive data is trimmed and aggregated according to user requirements, and after processing, an encryption UDF function is called to encrypt the data, and the data is pushed to a non-sensitive data area for subsequent data integration or output; after the data integration processing of the sensitive data is completed, the data is synchronously input into a kylin database or a mysql database according to requirements; when the classified data is queried, an OTP check code is input, and after verification, the sensitive data field is displayed; the encryption method is more flexible, and the encryption mode is safer.

Description

Technical Field

[0001] This invention relates to the field of data encryption and decryption technology, and in particular to a data encryption and decryption method based on a big data platform. Background Technology

[0002] Cryptography is an important means of maintaining confidentiality in communication, where both parties transform information according to agreed-upon rules. Transforming plaintext into ciphertext according to these rules is called encryption; transforming ciphertext into plaintext is called decryption. Early cryptography only performed encryption and decryption transformations on text or numbers. With the development of communication technology, encryption and decryption transformations can be applied to voice, images, and data. However, when dealing with massive amounts of data, performance issues may arise during data decryption. Summary of the Invention

[0003] To address the problems existing in the prior art, the purpose of this invention is to provide a data encryption and decryption method based on a big data platform. This invention achieves zero-contact encryption and decryption keys; convenient key modification; combines OTP access control and key management; and offers more flexible and secure encryption methods.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a data encryption and decryption method based on a big data platform, comprising the following steps:

[0005] Step 1: Data source connection;

[0006] Step 2: Determine if the data source is sensitive: If it is non-sensitive data, process it according to normal data standard procedures; if it is sensitive data, move it to the sensitive data area.

[0007] Step 3: When data from the sensitive data area is needed, the sensitive data is cropped and aggregated according to user requirements. Then, the encrypted UDF function is called to encrypt the data before pushing it to the non-sensitive data area for subsequent data integration or output.

[0008] Step 4: After the data involving sensitive data has been integrated and processed, synchronize it to the Kylin database or the MySQL database according to the requirements.

[0009] Step 5: When querying classified data, enter the OTP verification code. After successful verification, the sensitive data fields will be displayed.

[0010] As a further improvement of the present invention, in step 3, if the data output to the non-sensitive area is used as a one-time data extraction request, the sensitive data extraction UDF function is called to first decrypt the data, then randomly generate a data encryption key, encrypt the data again according to this key, and then output the encrypted data to the data extraction platform. At the same time, this temporary key is sent to the data extraction user. The data extraction user logs into the data extraction platform, enters the temporary key, and then downloads the data extraction request involving sensitive data.

[0011] As a further improvement to the present invention, the encryption and decryption of data are specifically performed as follows:

[0012] Data encryption: Request a key list from the key management server for data encryption; based on the key list and the server date, use the AES encryption algorithm to encrypt the data, and record both the ciphertext and the encryption date; encode the encrypted data using BASE64 to ensure stability during string processing and transmission.

[0013] Data decryption: Request a list of keys from the key management server to decrypt the data; perform BASE64 decoding on the ciphertext fields; select the decryption key corresponding to the date of the data encryption based on the key list and the data encryption date, and call the AES encryption algorithm to decrypt the data;

[0014] Data extraction and encryption / decryption: First, decrypt the data; then, randomly generate a data encryption / decryption key; use the random key to encrypt the data using the AES algorithm; finally, encode the encrypted data using BASE64 to ensure stability during string transmission.

[0015] As a further improvement of the present invention, in step 5, the method for generating the OTP verification code is as follows:

[0016] Users register with the key management server. The administrator reviews the users, assigns them a user type, and sends them an OTP key, which includes a plaintext key and a QR code.

[0017] When making a key call, the encryption / decryption module obtains the user's OTP key based on the user passed in during data processing, then generates the user's OTP verification code, and requests a list of encryption / decryption keys from the key management server through the interface;

[0018] The key management server obtains the user's OTP key based on the user information, and then generates the user's OTP verification code on the server side using the OTP algorithm. When the two OTP verification codes match, the server returns the encryption and decryption key list to the user; otherwise, it displays an error message and interrupts subsequent operations.

[0019] The encryption / decryption module performs data encryption / decryption processing based on the obtained key list.

[0020] The beneficial effects of this invention are:

[0021] 1. This invention addresses the sensitive data area, making sensitive data processing more flexible. After the data leaves the sensitive area, the entire data processing process is encrypted to ensure data security.

[0022] 2. Using the AES symmetric encryption method, data encryption and decryption speed is relatively fast;

[0023] 3. Enhance OTP verification capabilities for users when encrypting and decrypting data, thereby improving data security while ensuring efficiency;

[0024] 4. Users can specify a time period for their OTP permissions; outside of the specified time period, they will not have data decryption permissions.

[0025] 5. OTP authentication and data encryption / decryption keys are processed on the server side, reducing the risk of key cracking and maximizing data security. Attached Figure Description

[0026] Figure 1 This is a flowchart of an embodiment of the present invention;

[0027] Figure 2 This is a structural block diagram of data encryption and decryption in an embodiment of the present invention;

[0028] Figure 3 This is a flowchart of key management in an embodiment of the present invention. Detailed Implementation

[0029] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] Example

[0031] AES: Advanced Encryption Standard

[0032] OTP: One-Time Password

[0033] BASE64: A method for representing binary data using 64 printable characters.

[0034] like Figure 1 As shown, a data encryption / decryption method based on a big data platform includes:

[0035] 1. Source data access.

[0036] 2. Determine if the data is sensitive.

[0037] 2.1 If it is non-sensitive data, it shall be processed in accordance with the normal data standard procedures.

[0038] 2.2. If the data is sensitive, it should be moved to the sensitive data area. This area is only visible to the administrator of that data area. The sensitive data area is maintained to preserve the flexibility and ease of handling sensitive data as much as possible.

[0039] 3. When data from the sensitive data area needs to be used, a designated person will be responsible for processing the sensitive data according to user requirements, such as cropping and aggregating it, before encrypting the data using an encryption UDF function. This data will then be pushed to the non-sensitive data area for subsequent data integration or output.

[0040] 3.1 If the data output to the non-sensitive area is used for a one-time data extraction request, the data extraction UDF function will be called to first decrypt the data, then randomly generate a data encryption key, and encrypt the data again using this key. This encrypted data will then be output to the data extraction platform. Simultaneously, this temporary key will be sent to the data extraction user via email. The data extraction user needs to log in to the data extraction platform within 24 hours, enter the temporary key, and then download the data involving sensitive data. The same temporary key can be downloaded a maximum of 3 times.

[0041] 4. After the data involving sensitive information has been integrated and processed, it should be synchronized to either the Kylin database or the MySQL database as needed.

[0042] 5. When querying classified data in Yonghong BI reports, you need to enter an OTP verification code each time. Sensitive data fields can only be displayed after verification.

[0043] Next, we will provide a detailed explanation of the solution in three aspects: data encryption / decryption, data key management, and report display.

[0044] I. For example Figure 2 As shown, data encryption and decryption:

[0045] 1. Data encryption module, only involving HIVE data encryption functionality. Data encryption processing steps:

[0046] 1.1 Request a list of keys for encrypting data from the key management server (see Key Management for details).

[0047] 1.2. Based on the key list and server date, use the AES encryption algorithm to encrypt the data, and record both the ciphertext and the encryption date.

[0048] 1.3 Encode the encrypted data using BASE64 to ensure stability during string processing and transmission.

[0049] 2. Data decryption module, involving Kylin decryption and MySQL decryption functions. Data processing methods:

[0050] 2.1 Request a list of keys for decrypting data from the key management server (see Key Management for details).

[0051] 2.2 Perform BASE64 decoding on the ciphertext field.

[0052] 2.3. Select the decryption key corresponding to the date of the data encryption based on the key list and call the AES encryption algorithm to decrypt the data.

[0053] 3. Data extraction and encryption / decryption module, involving HIVE encryption and decryption functions:

[0054] 3.1 First, decrypt the data using the method described in step 2.

[0055] 3.2. Randomly generate a data encryption / decryption key.

[0056] 3.3. Use a random key to call the AES algorithm for data encryption.

[0057] 3.4 Encode the encrypted data using BASE64 to ensure stability during string transmission.

[0058] II. Figure 3 As shown, key management:

[0059] The core of a big data platform encryption / decryption scheme lies in key management and authorized use. The implementation process of key service management is as follows:

[0060] 1. Users register with the key management server. Users are divided into two categories: data warehouse users (used for encryption and decryption of data in HIVE, Kylin, and MySQL databases) and report users (used for decryption of data in the report end).

[0061] 2. The administrator reviews the user and assigns the user type, and sends the OTP key (plaintext key and QR code) to the user via email.

[0062] 3. To ensure the security of the key data, the key is stored in a database independent of the big data platform. A separate RESTful data service interface is created based on this data; when the key is needed, it must be retrieved through this interface call.

[0063] 4. Multiple keys are stored according to time periods, and different keys can be used for data encryption and decryption in different time periods.

[0064] 5. When calling the key, the encryption / decryption module obtains the user's OTP key based on the user passed in during data processing, then generates the user's OTP verification code, and requests the encryption / decryption key list from the key management server through the interface.

[0065] 6. The key management server also obtains the user's OTP key based on the user information, and then generates the user's OTP verification code on the server side using the OTP algorithm. When the two OTP verification codes match, the server returns the encryption and decryption key list to the user; otherwise, it displays an error message and interrupts subsequent operations.

[0066] 7. The encryption / decryption module performs data encryption / decryption processing based on the obtained key list.

[0067] III. Report Display:

[0068] 1. Users can configure the OTP key via the mobile OTP application or by scanning a QR code, and an OTP verification code will be automatically generated.

[0069] 2. For reports involving sensitive data queries, a user and OTP verification code are required for each data query. The reporting system then requests a data decryption key from the key management server based on the entered user and OTP verification code.

[0070] 3. If the decryption key can be obtained normally, the report will display the data normally; otherwise, the sensitive data information column will display "No permission to view".

[0071] This embodiment can be specifically applied to sensitive employee information (such as ID number, phone number, salary, etc.) in human resources systems; company financial data (such as product cost, profit, etc.) in financial systems; and customer information (such as customer contact person, contract amount, etc.) in contract systems.

[0072] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A data encryption and decryption method based on a big data platform, characterized in that, Includes the following steps: Step 1: Data source connection; Step 2: Determine if the data source is sensitive: If it is non-sensitive data, process it according to normal data standard procedures; If it is sensitive data, move it to the sensitive data area; Step 3: When data from the sensitive data area is needed, the sensitive data is cropped and aggregated according to user requirements. Then, the encrypted UDF function is called to encrypt the data before pushing it to the non-sensitive data area for subsequent data integration or output. Step 4: After the data involving sensitive data has been integrated and processed, synchronize it to the Kylin database or MySQL database as needed; Step 5: When performing a sensitive data query, generate an OTP verification code. After successful verification, display the sensitive data fields.

2. The data encryption and decryption method based on a big data platform according to claim 1, characterized in that, In step 3, if the data output to the non-sensitive area is used as a one-time data extraction request, the sensitive data extraction UDF function is called to first decrypt the data, then randomly generate a data encryption key, encrypt the data again using this key, and then output the encrypted data to the data extraction platform. At the same time, this temporary key is sent to the data extraction user. The data extraction user logs into the data extraction platform, enters the temporary key, and then downloads the data extraction request involving sensitive data.

3. The data encryption and decryption method based on a big data platform according to claim 2, characterized in that, The specific steps for encrypting and decrypting the data are as follows: Data encryption: Request a key list from the key management server for data encryption; based on the key list and the server date, use the AES encryption algorithm to encrypt the data, and record both the encrypted data and the encryption date; encode the encrypted data using BASE64 to ensure stability during string processing and transmission; Data decryption: Request a list of keys for decrypting data from the key management server; perform BASE64 decoding on the ciphertext fields; select the decryption key corresponding to the date of the data encryption based on the key list and the data encryption date, and call the AES encryption algorithm to decrypt the data; Data extraction and encryption / decryption: First, decrypt the data; then, randomly generate a data encryption / decryption key; use the random key to encrypt the data using the AES algorithm; finally, encode the encrypted data using BASE64 to ensure stability during string transmission.

4. The data encryption and decryption method based on a big data platform according to claim 3, characterized in that, In step 5, the method for generating and verifying the OTP verification code is as follows: Users register with the key management server. The administrator reviews the users, assigns them a user type, and sends them an OTP key, which includes a plaintext key and a QR code. When making a key call, the encryption / decryption module obtains the user's OTP key based on the user passed in during data processing, then generates the user's OTP verification code, and requests a list of encryption / decryption keys from the key management server through the interface; The key management server obtains the user's OTP key based on the user information, and then generates the user's OTP verification code on the server side using the OTP algorithm. When the two OTP verification codes match, the server returns the encryption and decryption key list to the user; otherwise, it displays an error message and interrupts subsequent operations. The encryption / decryption module performs data encryption / decryption processing based on the obtained key list.

Images