Method and system for parallel encryption and decryption of database by using cipher machine
By sharding database data interaction tasks and using multiple processing nodes to encrypt and decrypt them in parallel with the crypto machine, the problem of performance bottleneck in existing encryption and decryption equipment in large-scale data processing is solved, and efficient encryption and decryption performance and security compliance are achieved.
Patent Information
- Application Number
- CN202411716438.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-28
AI Technical Summary
Existing encryption and decryption devices (such as password machines) have serious performance bottlenecks when processing large-scale database data, resulting in the performance of a single query dropping from seconds to hours or even sky level, resulting in few manufacturers relying on password machines to encrypt the encryption and decryption of encrypted databases.
By sharding data interaction tasks and building a session connection with the cipher machine for each task slice, multiple processing nodes are used to encrypt and decrypt the processing in parallel, and the final result is returned to the database query engine.
It realizes efficient encryption and decryption processing of large-scale data, improves overall encryption and decryption performance, solves performance bottlenecks, and enables the encrypted database system to be truly secure and compliant.
Smart Images
Figure CN119720235A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of encryption and decryption, and in particular relates to a method and system for parallel encryption and decryption of a database using a cipher machine. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Information security is generally classified and graded, with different security requirements for information at different security levels. Under this constraint, data protection needs to meet both security and compliance indicators. Security includes a variety of security measures such as data encryption. Compliance requires the use of national secret algorithms and the use of qualified encryption and decryption hardware for data encryption and decryption and key management.
[0004] However, although current encryption and decryption equipment (such as cipher machines) have high encryption and decryption performance, if the data to be encrypted and decrypted is the entire database, and a ciphertext query involves TB-level data, the cipher machine will become a serious performance bottleneck, and the performance of a single query will drop from seconds to hours or even days. Therefore, few manufacturers will really rely on cipher machines to encrypt and decrypt encrypted databases. They either use cipher machines only in scenarios with extremely small amounts of information and access volume, or use soft encryption technology. Summary of the invention
[0005] In order to solve the above problems, the present invention proposes a method and system for parallel encryption and decryption of a database using a cipher machine. The present invention can truly use a cipher machine to encrypt and decrypt data in an encrypted database, can process large-scale data, and has excellent performance.
[0006] According to some embodiments, the present invention adopts the following technical solutions:
[0007] A method for parallel encryption and decryption of a database using a cipher machine comprises the following steps:
[0008] Get the parsed data interaction task;
[0009] Slice the data interaction task to form multiple task slices;
[0010] For each task slice, a session connection with the cipher machine is established, the key of the cipher machine is initialized, and a tuple queue and a message queue are allocated for each session;
[0011] Assign a separate processing node to each task slice, assign the corresponding session, and bind the key handle, as well as the corresponding tuple queue and message queue;
[0012] Each processing node uses the assigned session and key to send the data in the task slice to the cipher machine for encryption / decryption processing, and returns the encrypted / decrypted tuple;
[0013] The tuples returned by each processing node are merged, and the final merged result is returned to the database query engine.
[0014] As an optional implementation, the data interaction task includes a query task and an insert task.
[0015] As an optional implementation, the data interaction task is sliced to form a plurality of task slices, and the process includes slicing the data interaction task according to the estimated number of data rows / amounts of the data interaction task and the maximum number of concurrent sessions of the configured cryptographic machine.
[0016] As a further step, the number of task slices is calculated based on the estimated number of data rows, and the number of task slices is less than or equal to the maximum number of concurrent sessions of the configured cryptographic machine.
[0017] If the estimated number of task slices of the data interaction task is greater than the maximum number of concurrent sessions of the configured cryptographic machine, the maximum number of concurrent sessions of the configured cryptographic machine shall be the total number of task slices.
[0018] As an optional implementation, the process of building a session connection with the cipher machine for each task slice includes obtaining a session connection that matches the number of task slices from a cached session pool. If the number of sessions in the cached session pool does not meet the number of task slices, a new cipher machine session is created, put into the session pool, and then the required number of sessions are taken out.
[0019] As an optional implementation, all sessions use the same key handle / key.
[0020] As an optional implementation, a tuple queue and a message queue are allocated to each session, wherein the tuple queue is used to receive tuples encrypted / decrypted by the cipher machine and returned by the processing node, and the message queue is used to receive error report information from the processing node.
[0021] As an optional implementation, the processing node returns error information if an error occurs during the encryption / decryption process of the data sent to the cipher machine;
[0022] If any processing node returns an error message, the query is terminated and an error message is sent to the query engine.
[0023] As an optional implementation, when the data interaction task is an insert task, the database query engine inserts the merged encrypted tuple into the specified target object;
[0024] When the data interaction task is a query task, the database query engine sends the merged and decrypted tuples to the target user.
[0025] A database encryption and decryption system using a cipher machine in parallel, comprising:
[0026] The merging module is configured to obtain the parsed data interaction task; slice the data interaction task to form multiple task slices; build a session connection with the cipher machine for each task slice, initialize the key of the cipher machine, and allocate a tuple queue and a message queue for each session; allocate a separate processing node for each task slice, allocate a corresponding session, and bind the key handle, as well as the corresponding tuple queue and message queue; merge the tuples returned by each processing node, and return the merged final result to the database query engine.
[0027] The processing node is configured to connect with the merging module and the cipher machine, use the assigned session and key, send the data in the task slice to the cipher machine for encryption / decryption processing, and return the encrypted / decrypted tuple.
[0028] Compared with the prior art, the present invention has the following beneficial effects:
[0029] The present invention makes the encrypted database system truly safe and compliant. It can truly use the cryptographic machine to perform data encryption and decryption and key management in a compliant manner, ensuring the safety and compliance of the encryption and decryption algorithms, ensuring data security, and ensuring that the system meets national security requirements.
[0030] The present invention makes full use of the parallel capability of the cipher machine and solves the encryption / decryption performance of large-scale data by improving the concurrency. It does not rely on special / special cipher equipment, but is based on the existing hardware environment to achieve very good encryption / decryption performance. After testing, if the concurrency capability is 50, the overall encryption / decryption performance can be improved by more than 30 times.
[0031] The present invention solves the technical problem that a cryptographic machine cannot be efficiently called to perform encryption / decryption by increasing parallel encryption / decryption through an extended query engine.
[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0034] Figure 1 is a system architecture diagram of an embodiment;
[0035] Figure 2 is a schematic diagram of a processing flow of a merging module of an embodiment;
[0036] Figure 3 is a schematic diagram of a processing flow of a processing node of an embodiment;
[0037] Figure 4 is a data insertion task flow chart of an embodiment;
[0038] Figure 5 The present invention is a data query task flow chart of an embodiment. DETAILED DESCRIPTION
[0039] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0040] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0042] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.
[0043] Embodiment 1
[0044] A method for parallel encryption and decryption of a database using a cipher machine comprises the following steps:
[0045] Get the parsed data interaction task;
[0046] Slice the data interaction task to form multiple task slices;
[0047] For each task slice, a session connection with the cipher machine is established, the key of the cipher machine is initialized, and a tuple queue and a message queue are allocated for each session;
[0048] Assign a separate processing node to each task slice, assign the corresponding session, and bind the key handle, as well as the corresponding tuple queue and message queue;
[0049] Each processing node uses the assigned session and key to send the data in the task slice to the cipher machine for encryption / decryption processing, and returns the encrypted / decrypted tuple;
[0050] The tuples returned by each processing node are merged, and the final merged result is returned to the database query engine.
[0051] The above data interaction tasks include query tasks and insert tasks.
[0052] In this embodiment, the process of slicing the data interaction task to form multiple task slices includes slicing the data interaction task according to the estimated number of data rows / amounts of the data interaction task and the maximum number of concurrent sessions of the configured cryptographic machine.
[0053] According to the data interaction task, the process of estimating or predicting the number / amount of data rows can adopt existing technologies, which will not be described in detail here.
[0054] In this embodiment, the number of task slices is calculated based on the estimated number of data rows, and the number of task slices is less than or equal to the maximum number of concurrent sessions of the configured cryptographic machine.
[0055] If the estimated number of task slices of the data interaction task is greater than the maximum number of concurrent sessions of the configured cryptographic machine, the maximum number of concurrent sessions of the configured cryptographic machine shall be the total number of task slices.
[0056] For example, when the estimated number of data rows is 1,000 and the maximum number of concurrent sessions of the pre-configured cipher machine is 50, the total number of task slices can be set to 10, and each task processes 100 rows of data.
[0057] If the estimated number of data rows is 100,000 rows, and the maximum number of concurrent sessions of the pre-configured cipher machine is 50 rows, the total number of task slices can be set to 50, and each task slice processes 2,000 rows of data.
[0058] In this embodiment, the process of building a session connection with the cipher machine for each task slice includes obtaining a session connection that matches the number of task slices from the cached session pool. If the number of sessions in the cached session pool does not meet the number of task slices, a new cipher machine session is created, put into the session pool, and then the required number of sessions are taken out.
[0059] In this embodiment, all sessions use the same key handle.
[0060] In this embodiment, a tuple queue and a message queue are allocated to each session. The tuple queue is used to receive tuples encrypted / decrypted by the cipher machine and returned by the processing node, and the message queue is used to receive error report information from the processing node.
[0061] In this embodiment, if an error occurs during the encryption / decryption process of the data sent by the processing node to the cipher machine, an error message is returned;
[0062] If any processing node returns an error message, the query is terminated and an error message is sent to the query engine.
[0063] Embodiment 2
[0064] A database encryption and decryption system using a cipher machine in parallel, comprising:
[0065] The merging module is configured to obtain the parsed data interaction task; slice the data interaction task to form multiple task slices; build a session connection with the cipher machine for each task slice, initialize the key of the cipher machine, and allocate a tuple queue and a message queue for each session; allocate a separate processing node for each task slice, allocate a corresponding session, and bind the key handle, as well as the corresponding tuple queue and message queue; merge the tuples returned by each processing node, and return the merged final result to the database query engine.
[0066] The processing node is configured to connect with the merging module and the cipher machine, use the assigned session and key, send the data in the task slice to the cipher machine for encryption / decryption processing, and return the encrypted / decrypted tuple.
[0067] In this embodiment, the encryption database and the cipher machine are usually connected through a network, and the main time consumption is the network transmission stage. This embodiment makes full use of the parallelism of the cipher machine and adds two new modules to the database query engine, such as Figure 1 As shown, they are the merge module (i.e., the merging module, hereinafter referred to as merge) and the processor module (i.e., the processing node, hereinafter referred to as processor).
[0068] Merge is mainly responsible for dividing data into multiple task pieces, and the processor is responsible for processing the task pieces. One processor corresponds to one task piece. The processor sends the data in the task piece to the cipher machine for encryption / decryption, and sends the encrypted / decrypted data to the merger. The merger summarizes the data returned by each processor and returns it to the database query engine. Messages are also transmitted along with the encrypted / decrypted data, mainly error messages. When the processor encounters an error, it will return the error information to the merger, and the merger will decide the next step based on the error information.
[0069] Taking the data interaction task as the query task, the execution process of the merge is explained.
[0070] like Figure 2 As shown, the merger is responsible for the management of task slices, the connection of cipher machines, the management of processors, and the processing of returned data and error messages.
[0071] The processing steps are:
[0072] Step S1. According to the query plan generated by the database, the estimated data volume (number of rows) is obtained and divided into multiple task slices. The number of task slices is based on the estimated number of rows and the maximum number of concurrent sessions of the cipher machine configured by the user. The number of task slices is determined based on these two values.
[0073] Step S2. Get a session connection with the cryptographic machine. For example, if there are 10 task slices, 10 session connections are required. The session connection is obtained from the cached session pool. If the session pool does not have enough sessions, a new cryptographic machine session is created, put into the session pool, and then the required session is taken out.
[0074] Step S3. Initialize the key handle of the cipher machine. The key handle is the key provided by the cipher machine to the encryption database for encryption and decryption. However, the key is not provided directly to the encryption database, but only a key handle is provided to ensure that the key will not leave the cipher machine. All sessions will use the same key handle.
[0075] Step S4. Initialize the tuple queue and message queue; allocate a tuple queue and a message queue for each session (each session corresponds to a task slice and also corresponds to a processor). The tuple queue is used by the processor to return the tuple encrypted / decrypted by the cipher machine to the merger, and the message queue is mainly used by the processor to report error information.
[0076] Step S5. Initialize processors. According to the number of task slices, initialize the same number of processors, apply for memory, assign tasks, assign sessions, bind key handles, and bind tuple queues and message queues.
[0077] Step S6. Start the processor. The processor will use the assigned session and key handle to send the data in the task slice to the cipher machine for encryption / decryption processing, and return the encrypted / decrypted tuple. If an error occurs, an error message will be sent back.
[0078] Step S7: Receive and merge the tuples returned by the processor. The merger merges the tuples returned by each processor and returns the result to the database query engine. If an error message is received from a processor, the query is terminated and an error message is sent to the query engine.
[0079] like Figure 3 As shown, the processor module is responsible for sending the data in the task slice to the cipher machine for encryption / decryption, and sending the encrypted / decrypted data to the merger. If an error message is encountered, the error message is sent to the merger and the processing ends.
[0080] Embodiment 3
[0081] Take data insertion as a task, such as Figure 4 As shown, the database uses a cryptographic machine parallel encryption and decryption method, which includes:
[0082] Step S1. The database receives an insert statement;
[0083] Step S2: The database generates an execution plan for the insert statement.
[0084] Step S3. The merge module calculates the required task slices according to the estimated number of inserted data rows in the execution plan, initializes the corresponding number of processors, and hands the task slices to the processors for encryption.
[0085] Step S4. The processor uses the assigned session and key to send the data in the task slice to the cipher machine for encryption. The cipher machine returns the encrypted data to the processor, and the processor returns the encrypted tuple to the merger.
[0086] Step S5.merger aggregates all encrypted tuples returned by all processors and returns them to the database query engine.
[0087] Step S6: The database query engine inserts the encrypted tuple into the specified target object.
[0088] In this embodiment, the execution parts that are consistent with those in Embodiment 1 and Embodiment 2 are not described repeatedly here.
[0089] Embodiment 4
[0090] Taking data query as task, such as Figure 5 As shown, the database uses a cryptographic machine parallel encryption and decryption method, which includes:
[0091] Step S1: The database receives a query statement.
[0092] Step S2: The database generates an execution plan for the query statement.
[0093] Step S3. The merge module calculates the required task slices according to the number of ciphertext data rows queried by the execution plan, initializes the corresponding number of processors, and hands the task slices to the processors for decryption processing.
[0094] Step S4. The processor uses the assigned session and key to send the data in the task slice to the cipher machine for decryption. The cipher machine returns the decrypted data to the processor, and the processor returns the decrypted tuple to the merger.
[0095] Step S5.merger aggregates all decrypted tuples returned by all processors and returns them to the database query engine.
[0096] Step S6: The database query engine sends the decrypted tuple to the target user.
[0097] In this embodiment, the execution parts that are consistent with those in Embodiment 1 and Embodiment 2 are not described repeatedly here.
[0098] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.
Claims
1. A method for parallel encryption and decryption of a database using a cipher machine, characterized in that: The following steps are involved: Get the parsed data interaction task; Slice the data interaction task to form multiple task slices; For each task slice, a session connection with the cipher machine is established, the key of the cipher machine is initialized, and a tuple queue and a message queue are allocated for each session; Assign a separate processing node to each task slice, assign the corresponding session, and bind the key handle, as well as the corresponding tuple queue and message queue; Each processing node uses the assigned session and key to send the data in the task slice to the cipher machine for encryption / decryption processing, and returns the encrypted / decrypted tuple; The tuples returned by each processing node are merged, and the final merged result is returned to the database query engine.
2. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 1, characterized in that: The data interaction tasks include query tasks and insert tasks.
3. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 1, characterized in that: The process of slicing the data interaction task to form a plurality of task slices includes slicing the data interaction task according to the estimated number of data rows / amounts of the data interaction task and the maximum number of concurrent sessions of the configured cryptographic machine.
4. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 3, characterized in that: The number of task slices is calculated based on the estimated number of data rows, and the number of task slices is less than or equal to the maximum number of concurrent sessions of the configured cryptographic machine; If the estimated number of task slices of the data interaction task is greater than the maximum number of concurrent sessions of the configured cryptographic machine, the maximum number of concurrent sessions of the configured cryptographic machine shall be the total number of task slices.
5. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 1, characterized in that: The process of building a session connection with the cipher machine for each task slice includes obtaining a session connection that matches the number of task slices from the cached session pool. If the number of sessions in the cached session pool does not meet the number of task slices, a new cipher machine session is created, put into the session pool, and then the required number of sessions is taken out.
6. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 1, characterized in that: All sessions use the same key handle / key.
7. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 1, characterized in that: A tuple queue and a message queue are allocated to each session. The tuple queue is used to receive the tuples encrypted / decrypted by the cipher machine returned by the processing node, and the message queue is used to receive the report error information of the processing node.
8. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 1, characterized in that: When the processing node is sending data to the cipher machine for encryption / decryption, if an error occurs, an error message is returned; If any processing node returns an error message, the query is terminated and an error message is sent to the query engine.
9. A method for parallel encryption and decryption of a database using a cipher machine as claimed in claim 1, characterized in that: When the data interaction task is an insert task, the database query engine inserts the merged encrypted tuple into the specified target object; When the data interaction task is a query task, the database query engine sends the merged and decrypted tuples to the target user.
10. A database encryption and decryption system using a cipher machine, characterized in that: include: The merging module is configured to obtain the parsed data interaction task; The data interaction task is sliced to form a plurality of task slices; a session connection with a cryptographic machine is established for each task slice, the key of the cryptographic machine is initialized, and a tuple queue and a message queue are allocated for each session; a separate processing node is allocated for each task slice, a corresponding session is allocated, and a key handle, as well as a corresponding tuple queue and a message queue are bound; the tuples returned by each processing node are merged, and the merged final result is returned to the database query engine. The processing node is configured to connect with the merging module and the cipher machine, use the assigned session and key, send the data in the task slice to the cipher machine for encryption / decryption processing, and return the encrypted / decrypted tuple.