Data processing method and device

Abstract

Embodiments of the invention provide a data processing method and apparatus. The method comprises the steps of obtaining a data processing request queue of a cloud storage node; wherein the data processing request queue comprises a specified data processing request and a non-specified data processing request, and the specified data processing request is a data processing request which needs to encrypt or decrypt carried data; performing circular processing on the data processing requests in the data processing request queue, including encrypting or decrypting data carried by the specified data processing request, responding to the encrypted or decrypted data, and responding to the data carried by the non-specified data processing request; and when it is detected that the accumulated encrypted or decrypted data volume in the current circular processing process is greater than a preset data volume, stopping processing the specified data processing request in the current circular processing process.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data processing method and device. Background Art

[0002] In some storage systems, such as block storage systems, transparent encryption services can be provided for cloud storage nodes (such as cloud disks). Users can provide their own or use the key management service of the public cloud to maintain the corresponding keys in the storage system. The storage system encrypts the data at the bottom layer, and when users write and read data, it is in plain text. The data needs to be encrypted before storage, and the encrypted stored data needs to be decrypted before replying.

[0003] However, the encryption and decryption process takes a lot of time. When the amount of data to be encrypted and decrypted is large, it will cause a large delay, affecting the writing and reading process of data that does not need to be encrypted and decrypted. Summary of the Invention

[0004] In view of the above problems, a data processing method and apparatus are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A data processing method, comprising:

[0006] Obtaining a data processing request queue of a cloud storage node; wherein the data processing request queue includes designated data processing requests and non-designated data processing requests, wherein the designated data processing request is a data processing request requiring encryption or decryption of the carried data;

[0007] cyclically processing the data processing requests in the data processing request queue, including: encrypting or decrypting the data carried by the designated data processing request and responding with the encrypted or decrypted data, and responding with the data carried by the non-designated data processing request;

[0008] When it is detected that the accumulated amount of data encrypted or decrypted during the current cycle processing is greater than the preset amount of data, the processing of the designated data processing request is stopped during the current cycle processing.

[0009] Optionally, the data processing request queue is a data write request queue for a cloud storage node, the designated data processing request is a designated data write request, the non-designated data processing request is a non-designated data write request, and the designated data write request is a data write request requiring encryption processing of the carried data;

[0010] The cyclic processing of the data processing requests in the data processing request queue includes: encrypting the data carried by the designated data write request and sending the encrypted data to the cloud storage node for storage, and sending the data carried by the non-designated data write request to the cloud storage node for storage.

[0011] Optionally, the data processing request queue is a data reply request queue for feedback from the cloud storage node in response to a data read request, the designated data processing request is a designated data reply request, the non-designated data processing request is a non-designated data reply request, and the designated data reply request is a data reply request requiring decryption processing of the carried data;

[0012] The cyclic processing of the data processing requests in the data processing request queue includes: decrypting the encrypted data carried by the designated data reply request and replying with the decrypted data, and replying with the data carried by the non-designated data reply request.

[0013] Optionally, responding using the data carried in the non-specified data processing request includes:

[0014] The non-designated data processing request is moved into a first queue, and when processing the first queue, the data carried by the non-designated data processing request is used to respond.

[0015] Optionally, encrypting or decrypting the data carried in the designated data processing request and responding with the encrypted or decrypted data includes:

[0016] moving the designated data processing request into a second queue;

[0017] encrypting or decrypting data carried in the specified data processing request in the second queue;

[0018] The designated data processing request carrying the encrypted or decrypted data is moved into the first queue, and when processing the first queue, the encrypted or decrypted data is used to respond.

[0019] Optionally, the processing the first queue includes:

[0020] During processing of the first queue, the non-designated data processing requests and the designated data processing requests in the first queue are processed according to the request time sequence of the non-designated data processing requests and the designated data processing requests in the first queue.

[0021] Optionally, the encrypting or decrypting the data carried by the specified data processing request and responding with the encrypted or decrypted data further comprises:

[0022] In the current cycle processing, the amount of data accumulated by the encrypting or decrypting is determined, and the remaining data amount is determined according to the amount of data accumulated by the encrypting or decrypting and the preset data amount.

[0023] In the case that the data amount of the data carried by the specified data processing request is greater than the remaining data amount, the encrypting or decrypting is performed on part of the data carried by the specified data processing request according to the remaining data amount, and the response is performed with the encrypted or decrypted data.

[0024] Optionally, after the processing of the specified data processing request is stopped in the current cycle processing, the method further comprises:

[0025] In the case that part of the data carried by the specified data processing request is not encrypted or decrypted, the encrypted or decrypted part of the data in the specified data processing request and the non-encrypted or decrypted part of the data are recorded.

[0026] In the next cycle processing, the non-encrypted or decrypted part of the data is encrypted or decrypted, and the response is performed with all the encrypted or decrypted data.

[0027] Optionally, the method further comprises:

[0028] The preset data amount is obtained through the external interface.

[0029] A data processing system, which is used to implement the method described above.

[0030] A data processing device, which is used to implement the method described above.

[0031] An electronic device, characterized in that it comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program is executed by the processor to implement the method described above.

[0032] A computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the method described above.

[0033] A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method described above.

[0034] The embodiments of the present application have the following advantages:

[0035] In an embodiment of the present invention, a data processing request queue of a cloud storage node is obtained, the data processing request queue includes designated data processing requests and non-designated data processing requests, the designated data processing request is a data processing request that requires encryption or decryption of the carried data, and the data processing requests in the data processing request queue are cyclically processed, including: encrypting or decrypting the data carried by the designated data processing request and responding with the encrypted or decrypted data, and responding with the data carried by the non-designated data processing request. When it is detected that the cumulative amount of data encrypted or decrypted during the current cyclic processing is greater than the preset data amount, the designated data processing request is stopped from being processed during the current cyclic processing, thereby limiting the amount of data that needs to be encrypted or decrypted during a single cyclic processing process, avoiding excessive encryption or decryption of data during a single cyclic processing process, reducing the delay caused by data encryption or decryption, and thereby improving the processing speed of requests that do not require encryption or decryption, and reducing the impact on the data processing process that does not require encryption or decryption. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is a flowchart of data processing steps provided by one embodiment of the present invention;

[0038] Figure 2 is a flowchart of another data processing method provided by one embodiment of the present invention;

[0039] Figure 3 This is a flowchart of another data processing method provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0040] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0041] In some storage systems, such as block storage systems, transparent encryption services can be provided for cloud storage nodes (such as cloud disks). Users can provide their own or use the key management service of the public cloud to maintain the corresponding keys in the storage system. The storage system encrypts the data at the bottom layer, and when users write and read data, it is in plain text. The data needs to be encrypted before storage, and the encrypted stored data needs to be decrypted before replying.

[0042] In practical applications, storage clients can be deployed on compute nodes, such as by installing block storage terminal software on the compute nodes. The storage client provides data read and write capabilities to the compute nodes. The storage client acts as the storage system's interface to users, and its input and output data are all in plain text. Specifically, the storage client can implement the forwarding module function, forwarding read and write requests issued by the upper-level virtual machine or virtualization platform to the storage system's backend service module for processing according to the corresponding address rules, thereby enabling reading and writing to the cloud storage node.

[0043] Among them, the computing node can be a physical server, on which virtualization software can be allowed to create virtual machines or virtualization platforms for users to use. When the upper-level virtual machines or virtualization platforms need to use cloud storage nodes, they can be processed through the storage client deployed in the computing node.

[0044] In the storage client, a processing thread loop is used to execute the operations of sending requests to the backend and receiving requests to the upper layer.

[0045] The process of sending a request to the backend service module includes:

[0046] a. Obtain a list of pending requests (i.e., a data write request queue for a cloud storage node) from the request queue of each cloud disk (i.e., cloud storage node) from the upper-level virtual machine or virtualization platform.

[0047] b. Process each request in the request list one by one. The processing flow is as follows:

[0048] 1. Perform other basic processing on the request, such as unpacking and filling the header according to the requirements of the network protocol.

[0049] 2. If the cloud disk to be written is an encrypted cloud disk (that is, the data carried needs to be encrypted and stored), the data part of the request is encrypted to ensure that the data sent down is ciphertext.

[0050] c. Deliver the processed request to the corresponding backend service module.

[0051] For the process of receiving a request and replying to the upper virtual machine or virtualization platform:

[0052] a. Get the reply list (i.e., data reply request queue) processed by the backend service module from the network card queue. b. Process each reply request in the reply list one by one. The processing flow is as follows:

[0053] 1. Check whether the data message has errors (such as using the crc32 algorithm for detection).

[0054] 2. If the cloud disk from which the data is obtained is an encrypted cloud disk (i.e. the reply request carries encrypted data), the data portion of the request is decrypted to ensure that the data being replied to is in plain text.

[0055] c. Deliver the processed request to the corresponding upper-level virtual machine or virtualization platform.

[0056] As can be seen from the above, the storage client needs to perform the corresponding steps a, b, and c for each message processing process of sending a request to the backend service module and receiving the request reply to the upper-level virtual machine or virtualization platform before it can process the next request in the queue.

[0057] Among them, steps a and c take little time (such as a few microseconds) and have little impact on the delay of other requests in the queue. However, for step b, when encryption or decryption is required, the computing time required for the encryption and decryption process is positively correlated with the amount of data. When the amount of data is large, it takes more time and has a greater impact on the delay of other messages in the queue.

[0058] For example, assuming a regular message that does not require encrypted storage (i.e., a data write request), steps a, b, and c take a total of 5 microseconds. If there are 100 requests in the queue at the same time, the last request at the end of the queue needs to wait 500 microseconds to complete processing, that is, the delay caused by the request queuing in the queue is 500 microseconds.

[0059] For a message that requires encrypted storage, such as a 64KB message, the encryption process in step b takes 100 microseconds, and steps a, b, and c take a total of 105 microseconds. If there are 100 messages that require encryption in the queue, the last request at the end of the queue will be affected by a delay of 100*105=10,500 microseconds, which is 20 times longer than the 500 microseconds affected by conventional messages that do not require encrypted storage under the same conditions.

[0060] It can be seen that if the number of requests requiring encryption or decryption processing in the client is large, the delay of other messages in the queue will have a huge impact. Based on this, in the embodiment of the application, the request requiring encryption or decryption is first delivered to a specific queue, and in each processing cycle of the thread, the amount of data requiring encryption or decryption in the queue is limited, and only the preset amount of data is encrypted or decrypted, and the time period other than this is used to process other requests that do not require encryption or decryption, thereby improving the processing speed of the request that does not require encryption or decryption processing and reducing the impact on the request that does not require encryption or decryption processing.

[0061] For example, assuming that 100 requests requiring encryption processing enter the queue at a time (assuming that the data amount of each request is 100 KB), if the previous improved scheme is used, the 100 requests need to be encrypted and forwarded before the 101st request (assuming that the 101st request does not require encryption processing) can be processed, then the queuing delay of the request will be 100*100 KB=10 MB, which is the data amount of the previous 100 requests requiring encryption processing, data encryption time-consuming, and the impact is very large.

[0062] According to the scheme of the embodiment of the application, the 100 requests requiring encryption processing are delivered to a specific queue, and the specific queue only encrypts a certain amount of data each time, assuming that the value is 200 KB, then after 200 KB of data is encrypted, other requests that do not require encryption processing, i.e., the 101st request, are processed, and the queuing time of the 101st request is about the encryption time of 200 KB of data, which is greatly optimized compared with the 10 MB encryption time.

[0063] The application will be further described below in conjunction with the drawings:

[0064] Referring to Figure 1 , a step flowchart of a data processing method provided by an embodiment of the application is shown, which can be applied to a storage client, which can be deployed on a computing node, such as a storage client of a block storage system, and the storage client is deployed by installing a block storage terminal software on the computing node, and the storage client provides read and write functions of data to the computing node, and the storage client is an interface of the storage system to the user, and the input and output data of the storage client are all plaintexts. Specifically, the storage client can implement the function of a forwarding module, and forwards the read and write requests issued by an upper virtual machine or a virtualization platform to a back-end service module of a storage system according to a corresponding address rule for processing, thereby realizing reading and writing of a cloud storage node.

[0065] Among them, the computing node can be a physical server, on which virtualization software can be allowed to create virtual machines or virtualization platforms for users to use. When the upper-level virtual machines or virtualization platforms need to use cloud storage nodes, they can be processed through the storage client deployed in the computing node.

[0066] Specifically, the following steps may be included:

[0067] Step 101, obtaining a data processing request queue of a cloud storage node; wherein the data processing request queue includes designated data processing requests and non-designated data processing requests, and the designated data processing request is a data processing request that requires encryption or decryption of the carried data.

[0068] In some embodiments of the present invention, the data processing request queue is a data write request queue for the cloud storage node, the designated data processing request is a designated data write request, and the non-designated data processing request is a non-designated data write request; the designated data write request is a data write request that requires encryption of the carried data.

[0069] In the data writing scenario, when the upper-level virtual machine or virtualization platform needs to write data to a cloud storage node (such as a cloud disk), a data writing request can be sent to the storage client deployed in the computing node where the upper-level virtual machine or virtualization platform is located, thereby forming a data writing request queue containing multiple data writing requests.

[0070] In actual applications, some cloud storage nodes provide transparent encryption services, that is, the data in the cloud storage nodes is encrypted, and some data write requests in the data write request queue need to encrypt and store the carried data, that is, designated data write requests. Of course, there are also some cloud storage nodes that do not provide transparent encryption services, that is, the data in the cloud storage nodes is not encrypted, and some data write requests in the data write request queue do not need to encrypt and store the data, that is, non-designated data write requests. In some examples, it is possible to determine whether a data write request is a designated data write request by an identifier carried in the data write request, such as an identifier of a cloud storage node, that is, whether the cloud storage node provides transparent encryption services to determine whether it is a designated data write request.

[0071] In some embodiments of the present invention, the data processing request queue is a data reply request queue for feedback from the cloud storage node in response to a data reading request, the designated data processing request is a designated data reply request, the non-designated data processing request may be a non-designated data reply request, and the designated data reply request may be a data reply request that requires decryption processing of the carried data.

[0072] In the data reading scenario, the upper-level virtual machine or virtualization platform can send a data reading request to the cloud storage node (such as a cloud disk). The cloud storage node can respond to the data reading request with a data reply request fed back to the storage client deployed in the computing node where the upper-level virtual machine or virtualization platform is located, thereby forming a data reply request queue containing multiple data reply requests.

[0073] In actual applications, some cloud storage nodes provide transparent encryption services, that is, the data in the cloud storage nodes is encrypted, and the data carried by some data reply requests in the data reply request queue is encrypted data, that is, designated data reply requests. Of course, there are also some cloud storage nodes that do not provide transparent encryption services, that is, the data in the cloud storage nodes is not encrypted, and the data carried by some data reply requests in the data reply request queue is unencrypted data, that is, non-designated data reply requests. In some examples, it is possible to determine whether a data reply request is a designated data reply request by an identifier carried in the data reply request, such as an identifier of a cloud storage node, that is, whether the cloud storage node provides transparent encryption services to determine whether it is a designated data reply request.

[0074] Step 102, cyclically processing the data processing requests in the data processing request queue, including: encrypting or decrypting the data carried by the designated data processing request and responding with the encrypted or decrypted data, and responding with the data carried by the non-designated data processing request.

[0075] In the data writing scenario, the data processing requests in the data processing request queue are cyclically processed, including: encrypting the data carried by the specified data writing request and sending the encrypted data to the cloud storage node for storage, and sending the data carried by the non-specified data writing request to the cloud storage node for storage.

[0076] In practical applications, a processing thread may be used to cyclically process the data write requests in the data write request queue, that is, the processing process may be divided into multiple cyclic processing stages.

[0077] In some examples, a polling mode can be used to continuously execute a loop. Each loop cycle checks all queues for requests to be processed. If there are still requests to be processed, they are processed. If there are no more requests to be processed, the loop ends. Of course, other loop execution methods can also be used, such as using an interrupt trigger mode to execute the loop, that is, waking up the thread to process when there is a request, or dividing the loop by time, such as each loop processing time is 1 second.

[0078] The process of processing data write requests can include processing specified data write requests. Since the specified data write request is to write data to a cloud storage node that uses transparent encryption services, that is, the data needs to be encrypted and stored, and the data obtained from the upper virtual machine or virtualization platform is not encrypted, the data can be encrypted first, and then the encrypted data can be sent to the cloud storage node for storage.

[0079] The process of processing data write requests can also include processing non-specified data write requests. Since the cloud storage node to which the non-specified data write request is to be written does not use transparent encryption services, that is, the data does not need to be encrypted and stored, the non-encrypted data obtained from the upper virtual machine or virtualization platform can be directly sent to the cloud storage node for storage without encryption processing.

[0080] In the data reading scenario, the data processing requests in the data processing request queue are cyclically processed, including: decrypting the encrypted data carried by the specified data reply request and replying with the decrypted data, and replying with the data carried by the non-specified data reply request.

[0081] In practical applications, a processing thread may be used to cyclically process the data reply requests in the data reply request queue, that is, the processing process may be divided into multiple cyclic processing stages.

[0082] In some examples, a polling mode can be used to continuously execute a loop. Each loop cycle checks all queues for requests to be processed. If there are still requests to be processed, they are processed. If there are no more requests to be processed, the loop ends. Of course, other loop execution methods can also be used, such as using an interrupt trigger mode to execute the loop, that is, waking up the thread to process when there is a request, or dividing the loop by time, such as each loop processing time is 1 second.

[0083] The process of processing the data reply request can include the processing of the specified data reply request. Since the specified data reply request carries encrypted data, and the data that needs to be replied to the upper virtual machine or virtualization platform is unencrypted data, the data can be decrypted first, and then the decrypted data can be replied to the upper virtual machine or virtualization platform.

[0084] The process of processing data reply requests can also include processing non-specified data reply requests. Since non-specified data reply requests carry non-encrypted data, the non-encrypted data can be directly replied to the upper-level virtual machine or virtualization platform without the need for decryption processing.

[0085] In some embodiments of the present invention, responding using the data carried by the non-specified data processing request includes: moving the non-specified data processing request into a first queue, and responding using the data carried by the non-specified data processing request when processing the first queue.

[0086] In the data writing scenario, a first queue can be established. The first queue can be a regular queue, that is, a queue that does not require encryption processing. Non-specified data write requests can be moved into the first queue. Then, when processing the data write requests in the first queue, the data carried by the non-specified data write requests can be directly sent to the cloud storage node for storage.

[0087] In some examples, the first queue may also reuse the data write request queue, that is, the first queue and the data write request queue are the same queue. Of course, they may also be different queues.

[0088] In the data reading scenario, a first queue can be established. The first queue can be a regular queue, that is, a queue that does not require decryption processing. The non-specified data reply request can be moved into the first queue, and then when the data reply request in the first queue is processed, the data carried by the non-specified data reply request can be directly replied to the upper virtual machine or virtualization platform.

[0089] In some examples, the first queue may also reuse the data reply request queue, that is, the first queue and the data reply request queue are the same queue. Of course, they may also be different queues.

[0090] In some embodiments of the present invention, encrypting or decrypting the data carried by the specified data processing request and responding with the encrypted or decrypted data includes: moving the specified data processing request into a second queue; encrypting or decrypting the data carried by the specified data processing request in the second queue; moving the specified data processing request carrying the encrypted or decrypted data into the first queue, and responding with the encrypted or decrypted data when processing the first queue.

[0091] In the data writing scenario, for specified data writing requests, a second queue, namely an encryption queue, can be established, that is, a queue that needs to be encrypted. The specified data writing request can be moved into the second queue, and then when the data writing request in the second queue is processed, the data carried by the specified data writing request is encrypted.

[0092] After encryption, the specified data write request carrying the encrypted data can be moved into the first queue, and when the data write request in the first queue is processed, the encrypted data is sent to the cloud storage node for storage, that is, the second queue is only used to encrypt data, and the first queue is used to send data to the cloud storage node.

[0093] In the data reading scenario, for specified data reply requests, a second queue, namely a decryption queue, can be established, that is, a queue that needs to be decrypted. The specified data reply request can be moved into the second queue, and then when the data reply request in the second queue is processed, the data carried by the specified data reply request can be decrypted.

[0094] After decryption, the specified data reply request carrying the decrypted data can be moved into the first queue, and when the data reply request in the first queue is processed, the decrypted data is sent to the upper virtual machine or virtualization platform, that is, the second queue is only used to decrypt the data, and the first queue is used to reply data to the upper virtual machine or virtualization platform.

[0095] In some embodiments of the present invention, the processing of the first queue includes: in the process of processing the first queue, processing the non-specified data processing requests and the specified data processing requests in the first queue according to the request time sequence of the non-specified data processing requests and the specified data processing requests in the first queue.

[0096] When processing the data processing requests in the first queue, the non-specified data processing requests and the specified data processing requests in the first queue can be sorted according to the request time sequence of the non-specified data processing requests and the specified data processing requests, and then the data processing requests can be processed in sequence according to the request time sequence.

[0097] Step 103 : When it is detected that the accumulated amount of data encrypted or decrypted during the current cyclic processing is greater than a preset amount of data, the processing of the designated data processing request is stopped during the current cyclic processing.

[0098] In the data writing scenario, during the current loop processing, the cumulative amount of data processed by encryption can be calculated, and it can be determined whether the amount of data processed by encryption is greater than a first preset data amount, such as 16KB, 32KB, or 200KB. When the cumulative amount of data processed by encryption is less than or equal to the first preset data amount, the designated data write request is allowed to continue to be processed during the current loop processing, i.e., encryption processing is performed. When the cumulative amount of data processed by encryption is greater than the first preset data amount, the designated data write request is stopped during the current loop processing, i.e., encryption processing is not performed, and only non-designated data write requests are processed until the next loop processing, at which time processing of the designated data write request is resumed.

[0099] In the data reading scenario, during the current loop processing, the cumulative amount of data processed by decryption can be calculated, and it can be determined whether the amount of data processed by decryption is greater than a second preset data amount, such as 16KB, 32KB, or 200KB. If the cumulative amount of data processed by decryption is less than or equal to the second preset data amount, processing of designated data reply requests in the current loop processing is allowed to continue, i.e., decryption processing is performed. If the cumulative amount of data processed by decryption is greater than the second preset data amount, processing of designated data reply requests in the current loop processing is stopped, i.e., decryption processing is not performed, and only processing of non-designated data reply requests is executed until the next loop processing process, at which time processing of designated data reply requests is resumed.

[0100] In some embodiments of the present invention, the method further includes: obtaining a set preset data volume through an external interface.

[0101] In actual applications, the preset data volume may be pre-set or may be set by the user through an external interface.

[0102] In some embodiments of the present invention, the encrypting or decrypting the data carried by the specified data processing request and responding with the encrypted or decrypted data also includes: during the current cycle processing, determining the cumulative amount of data encrypted or decrypted, and determining the remaining data amount based on the cumulative amount of data encrypted or decrypted and the preset data amount; when the amount of data carried by the specified data processing request is greater than the remaining data amount, encrypting or decrypting part of the data carried by the specified data processing request according to the remaining data amount, and responding with the encrypted or decrypted data.

[0103] In the data writing scenario, during the current loop processing, the current cumulative amount of encrypted data can be determined, and then the preset amount of data used as the data amount limit can be subtracted from the current cumulative amount of encrypted data to obtain the remaining data amount. After determining the remaining data amount, it can be determined whether the amount of data carried by the next designated data write request to be processed is greater than the remaining data amount. If the amount of data carried by the next designated data write request to be processed is less than or equal to the remaining data amount, it can be processed directly. If the amount of data carried by the next designated data write request to be processed is greater than the remaining data amount, only the portion of data equal to the remaining data amount can be encrypted, and the unencrypted portion of data can be left for processing in the next loop.

[0104] In the data reading scenario, during the current loop processing, the current cumulative decrypted data volume can be determined, and then the preset data volume as the data volume limit can be subtracted from the current cumulative decrypted data volume to obtain the remaining data volume. After determining the remaining data volume, it can be determined whether the data volume carried by the next designated data read request to be processed is greater than the remaining data volume. If the data volume carried by the next designated data read request to be processed is less than or equal to the remaining data volume, it can be processed directly. If the data volume carried by the next designated data read request to be processed is greater than the remaining data volume, only the data of the remaining data volume can be decrypted, and the undecrypted data can be left for processing in the next loop.

[0105] In some embodiments of the present invention, after stopping processing the specified data processing request in the current loop processing process, it also includes: when there is part of the data carried by the specified data processing request that has not been encrypted or decrypted, recording the part of the data that has been encrypted or decrypted and the part of the data that has not been encrypted or decrypted in the specified data processing request; in the next loop processing process, encrypting or decrypting the part of the data that has not been encrypted or decrypted, and responding with all the encrypted or decrypted data.

[0106] In the data writing scenario, during the current loop processing, when the cumulative amount of encrypted data is greater than the first preset data amount, it is necessary to stop processing the specified data write request, that is, stop the encryption processing. There may be a situation where some data carried by the specified data write request is not encrypted, that is, some data has been encrypted in a certain request, and some data is not encrypted. In this case, the encrypted part of the data and the unencrypted part of the data in the specified data write request can be recorded. In the next loop processing, the unencrypted part of the data is encrypted, and then all the encrypted data in the request is sent to the cloud storage node for storage.

[0107] In the data reading scenario, during the current loop processing, when the cumulative decrypted data volume is greater than the second preset data volume, it is necessary to stop processing the specified data reply request, that is, stop decryption processing. There may be a situation where some data carried by the specified data reply request has not been decrypted, that is, some data has been decrypted in a certain request, and some data has not been decrypted. In this case, the decrypted part of the data and the undecrypted part of the data in the specified data reply request can be recorded. In the next loop processing, the undecrypted part of the data is decrypted, and then all the decrypted data in the request is replied to the upper virtual machine or virtualization platform.

[0108] In an embodiment of the present invention, a data processing request queue of a cloud storage node is obtained, the data processing request queue includes designated data processing requests and non-designated data processing requests, the designated data processing request is a data processing request that requires encryption or decryption of the carried data, and the data processing requests in the data processing request queue are cyclically processed, including: encrypting or decrypting the data carried by the designated data processing request and responding with the encrypted or decrypted data, and responding with the data carried by the non-designated data processing request. When it is detected that the cumulative amount of data encrypted or decrypted during the current cyclic processing is greater than the preset data amount, the designated data processing request is stopped from being processed during the current cyclic processing, thereby limiting the amount of data that needs to be encrypted or decrypted during a single cyclic processing process, avoiding excessive encryption or decryption of data during a single cyclic processing process, reducing the delay caused by data encryption or decryption, and thereby improving the processing speed of requests that do not require encryption or decryption, and reducing the impact on the data processing process that does not require encryption or decryption.

[0109] Reference Figure 2 , which shows a flowchart of a data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0110] Step 201: Obtain a data write request queue for a cloud storage node, wherein the data write request queue includes a specified data write request, and the specified data write request is a data write request requiring encrypted storage of carried data.

[0111] In the data writing scenario, when the upper-level virtual machine or virtualization platform needs to write data to a cloud storage node (such as a cloud disk), a data writing request can be sent to the storage client deployed in the computing node where the upper-level virtual machine or virtualization platform is located, thereby forming a data writing request queue containing multiple data writing requests.

[0112] In actual applications, some cloud storage nodes provide transparent encryption services, that is, the data in the cloud storage nodes is encrypted, and some data write requests in the data write request queue need to encrypt and store the carried data, that is, designated data write requests. Of course, there are also some cloud storage nodes that do not provide transparent encryption services, that is, the data in the cloud storage nodes is not encrypted, and some data write requests in the data write request queue do not need to encrypt and store the data, that is, non-designated data write requests. In some examples, it is possible to determine whether a data write request is a designated data write request by an identifier carried in the data write request, such as an identifier of a cloud storage node, that is, whether the cloud storage node provides transparent encryption services to determine whether it is a designated data write request.

[0113] Step 202, cyclically processing the data write requests in the data write request queue, including: encrypting the data carried by the designated data write request and sending the encrypted data to the cloud storage node for storage, and sending the data carried by the non-designated data write request to the cloud storage node for storage.

[0114] In practical applications, a processing thread may be used to cyclically process the data write requests in the data write request queue, that is, the processing process may be divided into multiple cyclic processing stages.

[0115] In some examples, a polling mode can be used to continuously execute a loop. Each loop cycle checks all queues for requests to be processed. If there are still requests to be processed, they are processed. If there are no more requests to be processed, the loop ends. Of course, other loop execution methods can also be used, such as using an interrupt trigger mode to execute the loop, that is, waking up the thread to process when there is a request, or dividing the loop by time, such as each loop processing time is 1 second.

[0116] The process of processing data write requests can include processing specified data write requests. Since the specified data write request is to write data to a cloud storage node that uses transparent encryption services, that is, the data needs to be encrypted and stored, and the data obtained from the upper virtual machine or virtualization platform is not encrypted, the data can be encrypted first, and then the encrypted data can be sent to the cloud storage node for storage.

[0117] The process of processing data write requests can also include processing non-specified data write requests. Since the cloud storage node to which the non-specified data write request is to be written does not use transparent encryption services, that is, the data does not need to be encrypted and stored, the non-encrypted data obtained from the upper virtual machine or virtualization platform can be directly sent to the cloud storage node for storage without encryption processing.

[0118] Step 203 : When it is detected that the accumulated encrypted data volume in the current cyclic processing process is greater than the preset data volume, the processing of the designated data write request is stopped in the current cyclic processing process.

[0119] During the current cycle processing, the cumulative encrypted data volume can be calculated, and it can be determined that the encrypted data volume is greater than a first preset data volume, such as 16KB, 32KB, or 200KB.

[0120] When the cumulative amount of data processed by encryption is less than or equal to the first preset data amount, it is allowed to continue to process the designated data write request in the current cycle processing, that is, to perform encryption processing. When the cumulative amount of data processed by encryption is greater than the first preset data amount, the processing of the designated data write request is stopped in the current cycle processing, that is, encryption processing is not performed, and only the processing of non-designated data write requests is executed until the next cycle processing process, at which time the processing of the designated data write request is restarted.

[0121] The following is an example to illustrate the data writing process in the present invention:

[0122] 1. Collect data and write it into the request queue

[0123] From the request queues of the upper-level virtual machine or virtualization platform for each cloud disk, the system obtains a list of pending requests (i.e., the data write request queue) and performs other basic processing on the requests, such as unpacking and filling in packet headers according to network protocol requirements. If the request is a message that requires encryption (i.e., a request for specifying data writes), it is placed in the encryption queue (i.e., the second queue). If the request does not require encryption, it is placed in the regular queue (i.e., the first queue).

[0124] 2. Processing encryption queue

[0125] Get the first preset data size. Assume that the first preset data size limited to be processed in one cycle is totalSize, and the accumulated encrypted data size in the current cycle is SnowSize (initial value is 0). Determine whether SnowSize is less than totalSize. If SnowSize is greater than totalSize, no longer process the encryption queue. <totalSize的情况下,则从加密队列取出请求,对报文进行加密处理。

[0126] During encryption processing, if SnowSize + message size to be processed<totalSize,则对整个报文的数据进行加密处理,如果nowSize+报文待处理大小> totalSize, only the data amount of totalSize-snowSize is encrypted, and the remaining data is left for processing in the next loop.

[0127] 3. Processing regular queues

[0128] During the current cycle processing, each request in the regular queue is directly delivered to the corresponding back-end module and then sent to the cloud storage node for storage.

[0129] In an embodiment of the present invention, a data write request queue for a cloud storage node is obtained, the data write request queue includes specified data write requests, the specified data write requests are data write requests that require encrypted storage of the carried data, and the data write requests in the data write request queue are cyclically processed, including: encrypting the data carried by the specified data write request and sending the encrypted data to the cloud storage node for storage, and when it is detected that the cumulative amount of encrypted data in the current cycle processing is greater than the first preset data amount, the processing of the specified data write request is stopped in the current cycle processing, thereby limiting the amount of data that needs to be encrypted in a single cycle processing, avoiding excessive amount of encrypted data in a single cycle processing, reducing the delay caused by data encryption, and thus improving the processing speed of requests that do not require encryption processing, and reducing the impact on the data writing process that does not require encryption.

[0130] Reference Figure 3 , shows a flowchart of another data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0131] Step 301: Obtain a data reply request queue fed back by a cloud storage node in response to a data read request, wherein the data reply request queue includes a specified data reply request, and the specified data reply request is a data reply request that requires decryption processing of the carried data.

[0132] In the data reading scenario, the upper-level virtual machine or virtualization platform can send a data reading request to the cloud storage node (such as a cloud disk). The cloud storage node can respond to the data reading request with a data reply request fed back to the storage client deployed in the computing node where the upper-level virtual machine or virtualization platform is located, thereby forming a data reply request queue containing multiple data reply requests.

[0133] In actual applications, some cloud storage nodes provide transparent encryption services, that is, the data in the cloud storage nodes is encrypted, and the data carried by some data reply requests in the data reply request queue is encrypted data, that is, designated data reply requests. Of course, there are also some cloud storage nodes that do not provide transparent encryption services, that is, the data in the cloud storage nodes is not encrypted, and the data carried by some data reply requests in the data reply request queue is unencrypted data, that is, non-designated data reply requests. In some examples, it is possible to determine whether a data reply request is a designated data reply request by an identifier carried in the data reply request, such as an identifier of a cloud storage node, that is, whether the cloud storage node provides transparent encryption services to determine whether it is a designated data reply request.

[0134] Step 302, cyclically processing the data reply requests in the data reply request queue, including: decrypting the encrypted data carried by the designated data reply request and replying with the decrypted data, and replying with the data carried by the non-designated data reply request.

[0135] In practical applications, a processing thread may be used to cyclically process the data reply requests in the data reply request queue, that is, the processing process may be divided into multiple cyclic processing stages.

[0136] In some examples, a polling mode can be used to continuously execute a loop. Each loop cycle checks all queues for requests to be processed. If there are still requests to be processed, they are processed. If there are no more requests to be processed, the loop ends. Of course, other loop execution methods can also be used, such as using an interrupt trigger mode to execute the loop, that is, waking up the thread to process when there is a request, or dividing the loop by time, such as each loop processing time is 1 second.

[0137] The process of processing the data reply request can include the processing of the specified data reply request. Since the specified data reply request carries encrypted data, and the data that needs to be replied to the upper virtual machine or virtualization platform is unencrypted data, the data can be decrypted first, and then the decrypted data can be replied to the upper virtual machine or virtualization platform.

[0138] The process of processing data reply requests can also include processing non-specified data reply requests. Since non-specified data reply requests carry non-encrypted data, the non-encrypted data can be directly replied to the upper-level virtual machine or virtualization platform without the need for decryption processing.

[0139] Step 303: When it is detected that the accumulated decrypted data volume in the current cyclic processing process is greater than the second preset data volume, the processing of the designated data reply request is stopped in the current cyclic processing process.

[0140] During the current loop processing, the cumulative decrypted data volume can be calculated, and it can be determined that the decrypted data volume is greater than a second preset data volume, such as 16KB, 32KB, or 200KB.

[0141] When the cumulative amount of data processed by decryption is less than or equal to the second preset data amount, it is allowed to continue to process the specified data reply request in the current cycle processing process, that is, to perform decryption processing. When the cumulative amount of data processed by decryption is greater than the second preset data amount, the processing of the specified data reply request is stopped in the current cycle processing process, that is, decryption processing is not performed, and only processing of non-specified data reply requests is executed until the next cycle processing process, at which time the processing of the specified data reply request is restarted.

[0142] The following is an example to illustrate the data reading process in the present invention:

[0143] 1. Receive data reply request queue

[0144] From the reply request queues for data retrieval requests on each cloud disk, a list of pending reply requests (i.e., the data reply request queue) is obtained. Other basic processing is performed on the requests, such as unpacking and filling in packet headers according to network protocol requirements. If the request is a message that requires decryption (i.e., a specified data reply request), it is placed in the decryption queue (i.e., the fourth queue). If the request does not require decryption, it is placed in the regular queue (i.e., the third queue).

[0145] 2. Processing decryption queue

[0146] Get the second preset data size. Assume that the second preset data size limited to be processed in one cycle is totalSize, and the accumulated decrypted data size in the current cycle is SnowSize (initial value is 0). Determine whether SnowSize is less than totalSize. If SnowSize is greater than totalSize, no longer process the decryption queue. <totalSize的情况下,则从解密队列取出请求,对报文进行解密处理。

[0147] During decryption, if SnowSize + message size to be processed<totalSize,则对整个报文的数据进行解密处理,如果nowSize+报文待处理大小> If totalSize is decrypted, only the data amount of totalSize-snowSize is decrypted, and the remaining data is left for processing in the next loop.

[0148] 3. Processing regular queues

[0149] During the current loop processing, each request in the regular queue is directly delivered to the corresponding backend module, and then replied to the upper virtual machine or virtualization platform.

[0150] In an embodiment of the present invention, a data reply request queue for data read request feedback from a cloud storage node is obtained, the data reply request queue includes a specified data reply request, the specified data reply request carries encrypted data, and the data reply requests in the data reply request queue are cyclically processed, including: decrypting the encrypted data carried by the specified data reply request and replying with the decrypted data, and when it is detected that the cumulative amount of decrypted data in the current cyclic processing process is greater than a second preset data amount, the processing of the specified data reply request is stopped in the current cyclic processing process, thereby limiting the amount of data that needs to be decrypted in a single cyclic processing process, avoiding excessive decryption of data in a single cyclic processing process, reducing the delay caused by data decryption, and thereby improving the processing speed of requests that do not require decryption processing, and reducing the impact of the reply process on data read requests that do not require decryption.

[0151] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0152] An embodiment of the present invention further provides a data processing system, which is used to implement the above method.

[0153] An embodiment of the present invention further provides a data processing device, which is used to implement the above method.

[0154] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above method is implemented.

[0155] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0156] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0157] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0159] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0160] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, compact disc read-only memory, optical storage, etc.) containing computer-usable program code.

[0161] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0165] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0166] The above is a detailed introduction to the provided data processing method and device. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a data processing request queue of a cloud storage node; wherein the data processing request queue includes designated data processing requests and non-designated data processing requests, wherein the designated data processing request is a data processing request requiring encryption or decryption of the carried data; cyclically processing the data processing requests in the data processing request queue, including: encrypting or decrypting the data carried by the designated data processing request and responding with the encrypted or decrypted data, and responding with the data carried by the non-designated data processing request; When it is detected that the accumulated amount of data encrypted or decrypted during the current cycle processing is greater than the preset amount of data, the processing of the designated data processing request is stopped during the current cycle processing.

2. The method according to claim 1, characterized in that The data processing request queue is a data write request queue for the cloud storage node, the designated data processing request is a designated data write request, the non-designated data processing request is a non-designated data write request, and the designated data write request is a data write request that requires encryption processing of the carried data; The cyclic processing of the data processing requests in the data processing request queue includes: encrypting the data carried by the designated data write request and sending the encrypted data to the cloud storage node for storage, and sending the data carried by the non-designated data write request to the cloud storage node for storage.

3. The method according to claim 1, characterized in that The data processing request queue is a data reply request queue for data read request feedback from the cloud storage node, the designated data processing request is a designated data reply request, the non-designated data processing request is a non-designated data reply request, and the designated data reply request is a data reply request that requires decryption processing of the carried data; The cyclic processing of the data processing requests in the data processing request queue includes: decrypting the encrypted data carried by the designated data reply request and replying with the decrypted data, and replying with the data carried by the non-designated data reply request.

4. The method according to any one of claims 1 to 3, characterized in that The responding with the data carried in the non-specified data processing request includes: The non-designated data processing request is moved into a first queue, and when processing the first queue, the data carried by the non-designated data processing request is used to respond.

5. The method according to claim 4, characterized in that The encrypting or decrypting the data carried in the designated data processing request and responding with the encrypted or decrypted data includes: moving the designated data processing request into a second queue; encrypting or decrypting data carried in the specified data processing request in the second queue; The designated data processing request carrying the encrypted or decrypted data is moved into the first queue, and when processing the first queue, the encrypted or decrypted data is used to respond.

6. The method according to claim 5, characterized in that The processing of the first queue includes: During processing of the first queue, the non-designated data processing requests and the designated data processing requests in the first queue are processed according to the request time sequence of the non-designated data processing requests and the designated data processing requests in the first queue.

7. The method according to claim 1, characterized in that The encrypting or decrypting the data carried in the designated data processing request and responding with the encrypted or decrypted data further includes: During the current cycle processing, determining the cumulative amount of data encrypted or decrypted, and determining the remaining amount of data based on the cumulative amount of data encrypted or decrypted and the preset amount of data; When the amount of data carried by the specified data processing request is greater than the remaining data amount, part of the data carried by the specified data processing request is encrypted or decrypted according to the remaining data amount, and a response is made using the encrypted or decrypted data.

8. The method according to claim 1, characterized in that After stopping processing the designated data processing request in the current cycle processing, the method further includes: If part of the data carried in the designated data processing request is not encrypted or decrypted, record the part of the data that has been encrypted or decrypted and the part of the data that has not been encrypted or decrypted in the designated data processing request; In the next cycle of processing, the unencrypted or undecrypted part of the data is encrypted or decrypted, and all the encrypted or decrypted data is used to respond.

9. The method according to claim 1, characterized in that Also includes: Obtain the preset data volume through the external interface.

10. A data processing system, characterized in that: The system is used to implement the method according to any one of claims 1 to 9.

11. A data processing device, characterized in that: The device is used to implement the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by the processor.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images