Data processing method, medium, computing device and program product

Abstract

The invention discloses a data processing method, a medium, computing equipment and a program product. The data processing method comprises the following steps: acquiring original data from an external storage unit of a processor; writing the original data into an internal storage unit of a processor; performing a plurality of data processing operations on the original data stored in the internal storage unit to obtain a plurality of processing results of the original data; and writing a plurality of processing results of the original data into the external storage unit from the internal storage unit.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to data processing methods, media, computing devices, and program products. Background Technology

[0002] In related technologies, data is typically stored in the processor's external storage. When the processor needs to process the data, it first writes the data from the processor's external storage to its internal storage, and then the processor retrieves the data from the internal storage and processes it. In some cases, multiple data processing operations need to be performed on the same data. Each data processing operation involves writing the data from the processor's external storage to its internal storage. Therefore, when performing multiple data processing operations, the same data is written to the processor's internal storage multiple times, resulting in duplicate data reads and writes. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a data processing method, the method comprising: acquiring raw data from an external storage unit of a processor; writing the raw data into an internal storage unit of the processor; performing multiple data processing operations on the raw data stored in the internal storage unit to obtain multiple processing results of the raw data; and writing the multiple processing results of the raw data from the internal storage unit into the external storage unit.

[0004] Secondly, this disclosure provides a data processing method applied to a cloud server, the cloud server including a cloud disk and a cloud processor, the registers of the cloud disk and the cloud processor being connected via an I / O stack; the method includes: obtaining raw data from the cloud disk; writing the raw data into the register via the I / O stack; performing multiple data processing operations on the raw data stored in the register to obtain multiple processing results of the raw data; and writing the multiple processing results of the raw data from the register into the cloud disk via the I / O stack.

[0005] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0006] Fourthly, embodiments of this disclosure provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any embodiment of this disclosure.

[0007] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0008] In this embodiment, after obtaining raw data from the processor's external storage unit, the raw data is written to the processor's internal storage unit and subjected to multiple data processing steps to obtain multiple processing results. Then, the multiple processing results are written from the processor's internal storage unit to the processor's external storage unit. This process only requires performing the operation of writing the raw data from the processor's external storage unit to the processor's internal storage unit once to obtain multiple processing results. It eliminates the need to repeatedly write the same raw data from the processor's external storage unit to the processor's internal storage unit, thus reducing redundant data read / write operations.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to illustrate the technical solutions of this disclosure.

[0011] Figure 1A This is a schematic diagram of the structure of a computing device according to an embodiment of the present disclosure.

[0012] Figure 1B This is a schematic diagram of the structure of a computing device according to another embodiment of the present disclosure.

[0013] Figure 2 This is a schematic diagram of the data reading and writing process in related technologies.

[0014] Figure 3 This is a flowchart of a data processing method according to an embodiment of the present disclosure.

[0015] Figure 4 This is a schematic diagram illustrating the erasure coding, CRC check, and data fingerprint calculation process of an embodiment of this disclosure.

[0016] Figure 5 This is a flowchart of a data processing method according to another embodiment of the present disclosure. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0018] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0019] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0021] Figure 1A A schematic diagram of the structure of a computing device 10 is shown. The computing device 10 includes a processor 100, a memory 102, and a computer program stored in the memory 102 and executable on the processor.

[0022] Figure 1B The diagram illustrates a more specific hardware structure of a computing device 10 provided in this embodiment of the present disclosure. The computing device 10 may include a processor 100, a memory 102, an input / output interface 104, a communication interface 106, and a bus 108. The processor 100, memory 102, input / output interface 104, communication interface 106, and bus 108 establish communication connections between them within the computing device 10.

[0023] The processor 100 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute computer programs. The processor 100 may include a memory unit 100a, which may include, but is not limited to, registers or cache. The memory unit 100a can be directly integrated into the processor 100; since the memory unit 100a is an internal memory unit of the processor 100, it can also be referred to as the internal memory unit of the processor 100. The processor 100 may also include a graphics card, which may be an Nvidia Titan X graphics card or a 1080Ti graphics card, etc.

[0024] The memory 102 can be RAM, which can be implemented as a read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. Alternatively, the memory 102 can also be a disk, optical disk, or memory card. The memory 102 is an external storage medium of the processor 100, not directly integrated into the processor 100, but connected to the processor 100 via the system bus 108. Therefore, the memory 102 can also be referred to as an external storage unit of the processor 100. The memory 102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this disclosure are implemented through software or firmware, the relevant program code is stored in the memory 102 and called and executed by the processor 100.

[0025] Input / output interface 104 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input modules may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output modules may include displays, speakers, vibrators, indicator lights, etc.

[0026] The communication interface 106 is used to connect a communication module (not shown in the figure) to enable communication and interaction between the computing device 10 and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0027] Bus 108 includes a pathway for transmitting information between various components of computing device 10 (e.g., processor 100, memory 102, input / output interface 104, and communication interface 106).

[0028] It should be noted that although the computing device 10 described above only shows the processor 100, memory 102, input / output interface 104, communication interface 106, and bus 108, in specific implementations, the computing device 10 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the computing device 10 may only include the components necessary for implementing the embodiments of this disclosure, and not necessarily all the components shown in the figures.

[0029] Since the internal storage space of processor 100 is limited, data is typically stored first in the external storage of processor 100. When processor 100 needs to process the data, the data is first written from the external storage to the internal storage of processor 100, and then processor 100 retrieves the data from the internal storage and processes it. Sometimes, processor 100 may need to perform multiple data processing operations on the same data. Since each data processing operation is generally considered independent, data is written from the external storage to the internal storage of processor 100 for each data processing operation. When multiple data processing operations are performed on the same data, the above data read and write process causes the same data to be written from the external storage to the internal storage of processor 100 multiple times, resulting in duplicate data reads and writes.

[0030] Figure 2 A schematic diagram of the data read / write process in the relevant technology is shown in a specific application scenario. In this scenario, multiple data processing operations include data encoding, generating verification data, and generating data fingerprints. The data is divided into multiple data blocks, denoted as {D1, D2, ..., D...}. k}, where k is a positive integer greater than 1, and the above multiple data blocks {D1, D2, ..., D} k} is stored in memory. In the data block {D1, D2, ..., D} k During processing, the data block {D1, D2, ..., D} is first read from memory. k}, data blocks {D1, D2, ..., D} are transferred through memory channels. k The data is written to the internal storage space (such as registers or cache) of processor 100 for data encoding operations, resulting in one or more encoded blocks, denoted as {P1, ..., P2}. r}, and encode the block {P1, ..., P} rWrite the data to memory. Then, read the data block {D1, D2, ..., D} from memory again. k Simultaneously, it can read encoded blocks {P1, ..., Pr} from memory and transfer data blocks {D1, D2, ..., Dr} through memory channels. k} and coded blocks {P1, ..., P r The operation of writing data into the internal storage space of processor 100 to generate verification data is performed, resulting in the data being written to each data block {D1, D2, ..., D...}. k} and each coded block {P1, ..., P r The corresponding check data blocks are denoted as {CRC(D1), CRC(D2), ..., CRC(D...}. k ), CRC(P1), ..., CRC(P r Then, read the data block {D1, D2, ..., D} from memory again. k Simultaneously, it can read encoded blocks {P1, ..., Pr} from memory and transfer data blocks {D1, D2, ..., Dr} through memory channels. k} and coded blocks {P1, ..., P r The data fingerprint is generated by writing data into the internal storage space of the processor 100, resulting in a fingerprint of each data block {D1, D2, ..., D}. k} and each coded block {P1, ..., P r The corresponding data fingerprints are denoted as {FP(D1), FP(D2), ..., FP(D... k ), FP(P1), ..., FP(P r )}.

[0031] It can be seen that in the above process, the data encoding operation, the verification data generation operation, and the data fingerprint generation operation all read the data block {D1, D2, ..., D}. k This results in the problem of repeated data reading and writing, which increases the consumption of memory bandwidth and processor storage and processing resources.

[0032] It should be noted that the above application scenarios are not intended to limit this disclosure. The solutions of this disclosure can also be applied to other application scenarios. In addition to the data encoding operation, the operation of generating verification data, and the operation of generating data fingerprints in the above application scenarios, the multiple data processing operations in the embodiments of this disclosure can also include other types of data processing operations, such as data encryption operation, data interleaving processing, data desensitization processing, etc., which will not be listed here. In addition to data processing at the data block level, the embodiments of this disclosure can also perform data processing at the level of a single original data item or at the level of a sub-block within a data block. For ease of explanation, the following mainly uses the three operations of data encoding operation, data generation operation, and data fingerprint generation operation as examples to describe the solutions of the embodiments of this disclosure.

[0033] Based on this, the present disclosure provides a data processing scheme that only requires one operation to write the original data from the external storage unit of the processor 100 to the internal storage unit of the processor 100, thereby enabling multiple data processing operations on the original data and obtaining multiple processing results. This eliminates the need to repeatedly write the same original data from the external storage unit of the processor 100 to the internal storage unit of the processor 100, reducing redundant data reads and writes. The specific implementation of the embodiments of the present disclosure will be illustrated below with reference to the accompanying drawings.

[0034] See Figure 3 This disclosure provides a data processing method, the method comprising:

[0035] Step S12: Obtain raw data from the external storage unit of processor 100;

[0036] Step S14: Write the raw data into the internal storage unit of the processor 100;

[0037] Step S16: Perform multiple data processing operations on the raw data stored in the internal storage unit of the processor 100 to obtain multiple processing results of the raw data;

[0038] Step S18: Write multiple processing results of the original data from the internal storage unit of the processor 100 to the external storage unit of the processor 100.

[0039] In step S12, the external storage unit of the processor 100 can be memory, disk, optical disk, or memory card, etc. For ease of explanation, the following description uses memory as an example of the external storage unit of the processor 100. Memory can store raw data. In some embodiments, this raw data includes multiple data blocks, which are respectively denoted as {D1, D2, ..., D...} k Furthermore, each data block can include multiple sub-blocks, where the i-th data block Di Multiple sub-blocks in (1≤i≤k, where i is an integer) are denoted as {S} i1 S i2 S iq}, where q is a positive integer greater than 1.

[0040] In step S14, data can be written to the internal storage unit of the processor 100 at the granularity of the original data. For example, at least one piece of original data can be written to the internal storage unit of the processor 100 at once. Alternatively, data can be written to the internal storage unit of the processor 100 at a smaller granularity. For example, if the original data includes multiple data blocks, and each data block includes multiple sub-blocks, at least one sub-block included in each data block can be written to the internal storage unit of the processor 100 at once. By dividing the original data into smaller granularities, the latency of data reading and writing can be reduced, thereby improving the efficiency of data processing and allowing for more flexible data operations.

[0041] Specifically, it can be derived from multiple data blocks {D1, D2, ..., D...} k At least one data block is determined from the data block {D1, D2, ..., D}, and the current sub-block is determined from each of the at least one data block. For example, if the number of the at least one data block is 1, it can be determined from multiple data blocks {D1, D2, ..., D}. k Data block D1 is determined from multiple data blocks {D1, D2, ..., D}. k In the process of determining at least one data block, data block D2 is identified, and so on. When at least one data block is identified as data block D1, the first sub-block S in data block D1 can be... 11 This is identified as the current sub-block, and further processing is performed. The first sub-block S in data block D1... 11 After processing is complete, the second sub-block S in data block D1 will be processed. 12 Once a data block is identified as the current sub-block, further processing is performed, and so on. When at least one data block is identified as another data block, the method for determining the current sub-block is similar to the above process, and will not be repeated here.

[0042] For example, if the number of at least one data block is 2, it can be obtained from multiple data blocks {D1, D2, ..., D...} k Data blocks D1 and D2 are determined from the data blocks {D1, D2, ..., D}. k In the process of determining data blocks D3 and D4, and so on, when at least one determined data block includes data block D1 and data block D2, the first sub-block S in data block D1 can be... 11 and the first sub-block S in data block D2 21 This is identified as the current sub-block, and further processing is performed. The first sub-block S in data block D1...11 and the first sub-block S in data block D2 21 After processing is complete, the second sub-block S in data block D1 will be processed. 12 and the second sub-block S in data block D2 22 The current sub-block is determined and further processing is performed, and so on. When at least one data block is determined to be another data block, such as data block D3 and data block D4, or data block D5 and data block D6, the method for determining the current sub-block is similar to the above process, and will not be repeated here.

[0043] It is understood that the above is merely an illustrative example. In other examples, the number of at least one data block may also be greater than two, which will not be listed here.

[0044] After identifying multiple data blocks {D1, D2, ..., D...} k After each of the data blocks in the process includes a current sub-block, the current sub-blocks can be written to the internal memory of the processor 100. When the number of current sub-blocks is greater than one, multiple current sub-blocks can be written to the internal memory of the processor 100 simultaneously in a single write operation, so that multiple current sub-blocks can be processed in parallel in the internal memory of the processor 100; alternatively, the current sub-blocks in one data block can be written one at a time, and after the current sub-blocks in the current data block are written and processed, the current sub-blocks in the next data block are written. For example, in a current sub-block including the first sub-block S in data block D1... 11 and the first sub-block S in data block D2 21 At that time, sub-block S can be... 11 and sub-block S 21 Simultaneously writing to the internal storage unit of processor 100, or first writing to sub-block S 11 Write to the internal storage unit of processor 100 for processing, in sub-block S 11 After processing is complete, then sub-block S 21 The data is written to the internal storage unit of the processor 100 for processing.

[0045] In step S16, multiple data processing operations can be performed on the original data to obtain multiple processing results. In some embodiments, the multiple data processing operations may include data encoding operations, operations to generate verification data, and operations to generate data fingerprints. The multiple processing results of the original data may include the processing results corresponding to the data encoding operations, the processing results corresponding to the verification data generation operations, and the processing results corresponding to the data fingerprint generation operations.

[0046] From multiple data blocks {D1, D2, ..., D... kIn an embodiment where at least one data block is determined in the at least one data block, and a current sub-block is determined from each of the at least one data block, multiple current processing results of the original data stored in the internal storage unit of the processor 100 can be incrementally updated based on the current sub-blocks included in each of the at least one data block. The multiple current processing results correspond to multiple data processing operations, and any one current processing result is obtained by performing the corresponding data processing operation on the preceding sub-block of the current sub-block included in each of the at least one data block. This embodiment of the present disclosure obtains multiple processing results of the original data through incremental updates. Thus, each time only one set of current sub-blocks needs to be written to the internal storage unit of the processor 100, and multiple data processing operations are performed on the currently written current sub-blocks. This reduces the occupation of the internal storage unit of the processor 100 during data processing and avoids the situation where the internal storage unit of the processor 100 is insufficient to accommodate the original data when the amount of original data is large, leading to data processing failure.

[0047] In some embodiments, the multiple data processing operations may include data processing operations for processing sub-blocks within a single data block, and data processing operations for processing sub-blocks across multiple data blocks. For example, when the multiple data processing operations include a data encoding operation, a checksum generation operation, and a data fingerprint generation operation, both the checksum generation operation and the data fingerprint generation operation can be data processing operations for processing sub-blocks within a single data block, while the data encoding operation can be a data processing operation for processing sub-blocks across multiple data blocks. The multiple current processing results of the original data may include the current processing result corresponding to the data encoding operation, the current processing result corresponding to the checksum generation operation, and the current processing result corresponding to the data fingerprint generation operation.

[0048] Multiple current processing results of the original data can be stored in the internal storage units of the processor 100. In some embodiments, the internal storage units of the memory can be divided into multiple storage slices corresponding to multiple data processing operations, each storage slice being used to store the current processing result of the corresponding data processing operation. A slice for storing the current sub-block can also be divided in the internal storage units of the processor 100. In the example where the internal storage units of the processor 100 are registers, each of the above-mentioned multiple storage slices may include a set of registers, and the number of registers in each set is greater than or equal to one.

[0049] Any current processing result of the original data can be obtained by performing corresponding data processing operations on the preceding sub-blocks of the current sub-blocks included in the at least one data block. For example, by performing data encoding operations on the preceding sub-blocks of the current sub-blocks included in the at least one data block, the current processing result corresponding to the data encoding operation of the original data can be obtained; by performing the operation of generating check data on the preceding sub-blocks of the current sub-blocks included in the at least one data block, the current processing result corresponding to the operation of generating check data of the original data can be obtained.

[0050] In any data block, the preceding sub-blocks of the current sub-block include all sub-blocks in that data block that precede the current sub-block. For example, in the case where the current sub-block includes the second sub-block S in data block D1... 12 At that time, the current sub-block S 12 The preceding sub-block is the first sub-block S in data block D1. 11 ;The third sub-block S in the current sub-block includes data block D1 13 At that time, the current sub-block S 13 The preceding sub-block includes the first sub-block S in data block D1. 11 and the second sub-block S in data block D1 12 ; The second sub-block S in the current sub-block includes data block D1 12 and the second sub-block S in data block D2 22 At that time, the current sub-block S 12 The preceding sub-block includes the first sub-block S in data block D1. 11 Current sub-block S 22 The preceding sub-block includes the first sub-block S in data block D2. 21 ;The third sub-block S in the current sub-block includes data block D1 13 and the third sub-block S in data block D2 23 At that time, the current sub-block S 13 The preceding sub-block includes the first sub-block S in data block D1. 11 and the second sub-block S in data block D1 12 Current sub-block S 23 The preceding sub-block includes the first sub-block S in data block D2. 21 and the second sub-block S in data block D2 22 .

[0051] Incremental updates are performed on multiple current processing results of the original data based on the current sub-blocks included in at least one data block. This can be achieved by performing operations (such as arithmetic or logical operations) on the current processing result with the current sub-blocks included in at least one data block, or the processing results of the current sub-blocks included in at least one data block, to obtain the incrementally updated current processing result. Alternatively, the current sub-blocks included in at least one data block, or the processing results of the current sub-blocks included in at least one data block, can be appended to the current processing result. For example, if the current processing result includes M1 bits and the current sub-blocks included in at least one data block include M2 ​​bits, appending the current sub-blocks included in at least one data block to the current processing result yields M1+M2 bits of data, which is the incrementally updated current processing result. Examples of these two incremental update methods—operational processing and data appending—are provided below.

[0052] In an embodiment employing computational processing for incremental updates, the current sub-block may include a single sub-block within a single data block. The aforementioned multiple data processing operations may include a first data processing operation, which is used to process the sub-blocks within the single data block. Assume the current sub-block consists of multiple data blocks {D1, D2, ..., D...}. k If the j-th sub-block of the i-th data block is used, then a first data processing operation can be performed on the j-th sub-block of the i-th data block and the first current processing result after the (j-1)-th incremental update of the i-th data block stored in the internal storage unit of the processor 100 to obtain the first current processing result after the j-th incremental update of the i-th data block, and then the first current processing result after the j-th incremental update of the i-th data block is stored in the internal storage unit of the processor 100. Here, j is a positive integer greater than or equal to 2, and i is a positive integer. For example, when j=2, the current sub-block is the 2nd sub-block of the i-th data block, and the first data processing operation can be performed on the 2nd sub-block of the i-th data block and the first current processing result after the 1st incremental update of the i-th data block to obtain the first current processing result after the 2nd incremental update of the i-th data block. Similarly, when j=3, the current sub-block is the 3rd sub-block in the i-th data block. First data processing can be performed on the 3rd data block in the i-th data block and the first current processing result after the 2nd incremental update of the i-th data block to obtain the first current processing result after the 3rd incremental update of the i-th data block. And so on. Specifically, when j=1, the current sub-block is the 1st sub-block in the i-th data block. First data processing can be performed on the 1st sub-block in the i-th data block and a null value or a specified initial value (e.g., all 0s or all 1s) to obtain the first current processing result after the 1st incremental update of the i-th data block.

[0053] This embodiment of the disclosure uses sub-blocks as the granularity. Each time data processing is performed, a single sub-block from a single data block is read from the external storage unit of the processor 100 as the current sub-block. Multiple current processing results of the original data are then incrementally updated based on the current sub-block until the complete processing result of the data block containing the current sub-block is obtained. This process only requires one write operation from the external storage unit of the processor 100 to the internal storage unit of the processor 100 for each sub-block within a single data block to obtain the processing results corresponding to multiple data processing operations, reducing redundant data reads and writes.

[0054] Furthermore, if the j-th sub-block is the last sub-block in the i-th data block, after obtaining the first current processing result after the j-th incremental update of the i-th data block, the first current processing result after the j-th incremental update of the i-th data block can be output. Simultaneously, it can also be obtained from multiple data blocks {D1, D2, ..., D...} k In the next data block (i.e., the (i+1)th data block) of the i-th data block, the current sub-block is determined, and the first current processing result of the (i+1)th data block is updated. For details on the specific update method, please refer to the incremental update method for the first current processing result of the i-th data block in the aforementioned embodiment; it will not be repeated here.

[0055] In a specific application scenario, the first data processing operation is to obtain a data fingerprint (Finger Print, FP). The first current processing result after the j-th incremental update of the i-th data block is the data fingerprint after the j-th incremental update of the i-th data block. The data fingerprint acquisition operation can be implemented through XOR processing or other processing methods. The specific process of updating the first current processing result of the i-th data block is as follows: First, obtain the first sub-block of the i-th data block, and XOR the first sub-block of the i-th data block with a set of null values ​​to obtain the data fingerprint after the first incremental update of the i-th data block. Then, obtain the second sub-block of the i-th data block, and XOR the second sub-block of the i-th data block with the data fingerprint after the first incremental update of the i-th data block, thereby updating the data fingerprint after the second incremental update of the i-th data block. Then, the third sub-block in the i-th data block is obtained, and an XOR operation is performed between the third sub-block and the data fingerprint of the i-th data block after the second incremental update, thus obtaining the data fingerprint of the i-th data block after the third incremental update. This process is repeated. By performing the data fingerprint acquisition operation, the security and integrity of the data can be improved, reducing the risk of the original data being maliciously tampered with or forged.

[0056] In another specific application scenario, the first data processing operation is Cyclic Redundancy Check (CRC). CRC uses a hash function to generate a short, fixed-length checksum for the original data, primarily used to detect or verify errors that may occur after data transmission or storage. It is understood that this is merely an illustrative example and not intended to limit this disclosure. In other examples, the first data processing operation can also be any other operation capable of generating checksum data. The first current processing result after the j-th incremental update of the i-th data block is the checksum data after the j-th incremental update of the i-th data block. The specific process for updating the first current processing result of the i-th data block is as follows: First, obtain the first sub-block in the i-th data block, perform CRC on the first sub-block in the i-th data block, and obtain the checksum data after the first incremental update of the i-th data block. Then, the second sub-block of the i-th data block is obtained, and an XOR operation is performed between the second sub-block of the i-th data block and the checksum data after the first incremental update of the i-th data block, thereby updating the checksum data after the second incremental update of the i-th data block. Next, the third sub-block of the i-th data block is obtained, and an XOR operation is performed between the third sub-block of the i-th data block and the checksum data after the second incremental update of the i-th data block, thereby obtaining the checksum data after the third incremental update of the i-th data block. This process continues. By performing cyclic redundancy check on the data, it is possible to detect whether errors or corruption have occurred during data transmission or storage, thereby ensuring data integrity.

[0057] The above embodiments illustrate the case where, in an incremental update using an arithmetic processing method, the current sub-block comprises a single sub-block within a single data block. In other embodiments of incremental updates using an arithmetic processing method, the current sub-block may also comprise single sub-blocks comprised of multiple data blocks. For example, at least one data block may include multiple data blocks {D1, D2, ..., D...} k For each of the following steps, first determine the first sub-block in data block D1 as the current sub-block, process the data in the first sub-block of data block D1 to obtain the current processing result after the first incremental update; then determine the first sub-block in data block D2 as the current sub-block, process the data in the first sub-block of data block D2 and the current processing result after the first incremental update to obtain the current processing result after the second incremental update; then determine the first sub-block in data block D3 as the current sub-block, process the data in the first sub-block of data block D3 and the current processing result after the second incremental update to obtain the current processing result after the third incremental update. And so on.

[0058] In an embodiment employing an append-only approach for incremental updates, the current sub-block may include a single sub-block from at least two data blocks. The aforementioned multiple data processing operations may include a second data processing operation for processing the sub-blocks from the at least two data blocks. Assume that the at least two data blocks include multiple data blocks {D1, D2, ..., D...}. k The data block {D1, D2, ..., D} in} v}(v≤k, v is a positive integer greater than 1), the current sub-block can include data blocks {D1, D2, ..., D}. v The j-th sub-blocks included in the data block {D1, D2, ..., D}. v The second data processing operation is performed on the j-th sub-blocks included in each of the data blocks to obtain the data block {D1, D2, ..., D}. v The j-th second current processing result of} will be the data block {D1, D2, ..., D}. v The j-th second current processing result is appended to the data block {D1, D2, ..., D} stored in the internal storage unit of processor 100. v From the first j-1 second current processing results, the data block {D1, D2, ..., D} is obtained. v The first j second current processing results of the data block {D1, D2, ..., D} are then processed. v The first j second current processing results are stored in the internal storage unit of processor 100. Here, j is a positive integer greater than or equal to 2, and i is a positive integer.

[0059] For example, when j=2, the current sub-block includes data blocks {D1, D2, ..., D...} v The second sub-block of each, i.e., sub-block {S} 12 S 22 S v2}, can be applied to sub-blocks {S 12 S 22 S v2 The second data processing operation is performed to obtain the data block {D1, D2, ..., D}. v The second current processing result of} will be the data block {D1, D2, ..., D}. v The second current processing result of} is appended to the data block {D1, D2, ..., D}. v In the first and second current processing results of}, the data block {D1, D2, ..., D} is obtained. v The first two second current processing results of}. Similarly, when j=3, the current sub-block includes data blocks {D1, D2, ..., D}. v The third sub-block of each, i.e., sub-block {S} 13 S 23 Sv3}, can be applied to sub-blocks {S 13 S 23 S v3 The second data processing operation is performed to obtain the data block {D1, D2, ..., D}. v The third and second current processing result of} will be the data block {D1, D2, ..., D}. v The third result of the second current processing is appended to the data block {D1, D2, ..., D}. v From the first two second current processing results, the data block {D1, D2, ..., D} is obtained. v The first three are the second current processing results.

[0060] Specifically, when j=1, the current sub-block includes data blocks {D1, D2, ..., D...} v The first sub-block of each, i.e., sub-block {S} 11 S 21 S v1}, can be applied to sub-blocks {S 11 S 21 S v1 The second data processing operation is performed to obtain the data block {D1, D2, ..., D}. v The first and second current processing result of}.

[0061] This embodiment of the disclosure operates at the sub-block level. During each data processing iteration, at least one sub-block from each of multiple data blocks is read from the external storage unit of the processor 100 as the current sub-block. Based on the current sub-block, multiple current processing results of the original data are incrementally updated until the complete processing results of each current sub-block's data block are obtained. This process only requires one write operation from the external storage unit of the processor 100 to the internal storage unit of the processor 100 for each sub-block in the multiple data blocks to obtain the processing results corresponding to each of the multiple data processing operations, reducing redundant data reads and writes.

[0062] Furthermore, if the data block {D1, D2, ..., D...} v The j-th sub-blocks included in each block are data blocks {D1, D2, ..., D}. v The last sub-block included in each of the data blocks {D1, D2, ..., D} is obtained after acquiring the data block {D1, D2, ..., D}. v After the j-th second current processing result, the data block {D1, D2, ..., D} can be output. v The j-th second current processing result of} can also obtain multiple data blocks {D1, D2, ..., D}. k The next batch of data blocks in}, i.e., data block {D v+1 D v+2 D2v}, and for data block {D v+1 D v+2 D 2v The second current processing result of each sub-block in the data block {D1, D2, ..., D} is appended. For details on the appending method, please refer to the previous embodiment for the data block {D1, D2, ..., D}. k The appending method for the second current processing result of each sub-block in} will not be elaborated here.

[0063] In a specific application scenario, the second data processing operation is an encoding operation. The second current processing result of the aforementioned at least two data blocks is an encoded block obtained by encoding the current sub-blocks included in each of the aforementioned two data blocks. In some embodiments, the aforementioned encoding operation is erasure coding (EC) encoding. Erasure coding is a fault-tolerant encoding technique. Its basic principle is to divide the stored original data into k data blocks, and generate r EC encoded blocks from multiple data blocks through a certain encoding method. The original data can be reconstructed from any k parts of the k+r data. In this way, even if some data is lost, the system can still recover the original data. It is understood that this is only an illustrative example and is not intended to limit this disclosure. In other examples, other encoding operations that can achieve fault-tolerant encoding techniques or encoding operations that can achieve other purposes can also be used. By performing encoding, lost or damaged data can be recovered, thereby improving the integrity and reliability of the data.

[0064] The above embodiments illustrate the case where, in an incremental update using a data appending method, the current sub-block comprises individual sub-blocks that are each contained within a multiple data block. In other embodiments of incremental updates using a data appending method, the current sub-block may also comprise a single sub-block within a single data block. For example, the first sub-block in data block D1 can be determined as the current sub-block, and data processing can be performed on the first sub-block in data block D1 to obtain the current processing result of the first sub-block in data block D1; then, the second sub-block in data block D1 can be determined as the current sub-block, and data processing can be performed on the second sub-block in data block D1 to obtain the current processing result of the second sub-block in data block D1, which is then appended to the current processing result of the first sub-block in data block D1. This process continues until the current processing result of data block D1 is obtained.

[0065] In the above embodiments, the size of each sub-block can be determined based on the size of the processor 100's internal storage space. For example, if the processor 100's internal storage space is a register, assuming the size of a single register is B bytes, then the size of each sub-block can be less than or equal to B bytes. In some examples, individual sub-blocks can be further divided to obtain smaller-granularity data processing units, and at least one updated-granularity data processing unit from at least one sub-block is processed each time.

[0066] In step S18, after obtaining multiple processing results of the original data, the multiple processing results of the original data can be written from the internal storage unit of the processor 100 to the external storage unit of the processor 100.

[0067] When the original data comprises multiple data blocks, and each data block comprises multiple sub-blocks, it can be determined whether the current sub-block included in at least one data block is the last sub-block included in each of the aforementioned at least one data block. For example, when at least one data block is a single data block (assumed to be data block {D1, D2, ..., D...}),... k The i-th data block D in} i When ), it can be determined whether the current sub-block is data block D. i The last sub-block S iq For example, at least one data block includes at least two data blocks (assuming it includes data blocks {D1, D2, ..., D...}). v When}), it can be determined whether the current sub-block is a data block {D1, D2, ..., D}. v The last sub-block included in each of the sub-blocks, i.e., sub-block {S} 1q S 2q S vq}

[0068] If the determination result is negative, the next sub-block of the current sub-block included in at least one data block can be updated to the current sub-block included in the at least one data block, and the process returns to step S12 to continue processing the next sub-block included in the at least one data block. If the determination result is positive, it means that all sub-blocks in the data block containing the current sub-block have been processed. Therefore, the multiple processing results after incremental updates of the original data can be written from the internal storage unit of the processor 100 to the external storage unit of the processor 100.

[0069] In the above process, only when the current sub-block is the last sub-block in the corresponding data block is it necessary to write the multiple processing results of the incrementally updated original data from the internal storage unit of the processor 100 to the external storage unit of the processor 100. In other cases, it is not necessary to write the multiple processing results of the incrementally updated original data from the internal storage unit of the processor 100 to the external storage unit of the processor 100. Therefore, the operation of writing data from the internal storage unit of the processor 100 to the external storage unit of the processor 100 is effectively reduced, and the resource consumption generated by data reading and writing is reduced.

[0070] Any processing result after incremental update written to the external storage unit of processor 100 can be either the final result of the corresponding data processing operation or an intermediate result of the data processing operation. If the processing result after incremental update written to the external storage unit of processor 100 is an intermediate result of the corresponding data processing operation, the intermediate result can be written to the internal storage unit of processor 100 for further processing to obtain the final result of the corresponding data processing operation. In other words, the raw data obtained in step S12 can be data that has not been processed by any one of the multiple data processing operations, or it can be an intermediate result obtained after being processed by at least one of the multiple data processing operations.

[0071] In some embodiments, multiple data blocks {D1, D2, ..., D...} can be used. k The data is divided into at least two groups, each group including at least one data block. The number of data blocks in different groups can be the same or different. Multiple data blocks in step S12 can be data blocks from the same group. For the target data processing operation in multiple data processing operations, after processing each data block in the current group according to step S14 to complete the target data processing operation, an intermediate processing result corresponding to the target processing operation for the current group can be obtained and written to the external storage unit of the processor 100 for storage. Then, step S14 is executed on the data blocks in the next group to obtain the intermediate processing result corresponding to the target processing operation for the next group, and the intermediate processing result corresponding to the target processing operation for the next group is written to the external storage unit of the processor 100 for storage. After writing the intermediate processing results corresponding to the target processing operation for at least one group to the external storage unit of the processor 100, the intermediate processing results corresponding to the target processing operation for at least one group can be further written to the internal storage unit of the processor 100 for further processing. The data processing operation used for further processing can be the same as or different from the target processing operation.

[0072] For example, when multiple data processing operations include data encoding, data encoding can be performed on one group of data blocks first to obtain the encoded data result of that group of data blocks, which is then written to the external storage space of the processor 100. Then, data encoding is performed on the next group of data blocks to obtain the encoded data result of the next group of data blocks, which is also written to the external storage space of the processor 100. The encoded results of the two groups of data blocks are then read from the external storage space of the processor 100 and encoded again to obtain a secondary encoding result.

[0073] For example, in a data processing operation that includes data encoding and checksum generation, the data blocks of one group can be encoded and checksum generated first to obtain the data encoding result and checksum of that group, which are then written to the external storage space of the processor 100. Then, the next group of data blocks is encoded and checksum generated to obtain the encoding result and checksum of the next group, which are also written to the external storage space of the processor 100. The encoding results of the two groups of data blocks are then read from the external storage space of the processor 100, and the data encoding and checksum generation operations are performed again to obtain the secondary encoding result and checksum of the secondary encoding result.

[0074] When the internal storage capacity of the processor 100 is small, by grouping multiple data blocks, only the current sub-block of the data block in the group needs to be written to the internal storage capacity of the processor 100 each time, without having to write all the data blocks included in the original data to the internal storage capacity of the processor 100. This reduces the capacity requirement of the internal storage capacity of the processor 100, enabling the data processing method of this embodiment to be implemented even when the internal storage capacity of the processor 100 is small, thereby improving the applicability of the data processing method of this embodiment.

[0075] The following is combined with Figure 4 The following is an example illustrating the specific methods of performing multiple data processing operations in this embodiment. In this embodiment, the external storage space of the processor 100 is memory, and the internal storage space of the processor 100 is registers. The multiple data processing operations include erasure coding, cyclic redundancy check, and data fingerprint generation. In some embodiments, the process of performing erasure coding is as follows:

[0076] Split the original data into k data blocks {D1, D2, ..., Dk}. k}, multiplied by the corresponding r*k dimension encoding matrix, yields r EC encoded block data blocks {P1, ..., P}. r}, as shown below:

[0077]

[0078] Cyclic redundancy check can be performed on each data block {D1, D2, ..., D...} k} and each EC coded block {P1, ..., P r The check data are calculated according to the Cyclic Redundancy Check (CRC) algorithm, and denoted as {CRC(D1), CRC(D2), ..., CRC(D...}. k ), CRC(P1), ..., CRC(P r )}.

[0079] The operation of generating data fingerprints can be performed on each data block {D1, D2, ..., D...} k} and each EC coded block {P1, ..., P r Generate a data fingerprint, denoted as {FP(D1), FP(D2), ..., FP(D...}. k ), FP(P1), ..., FP(P r The data fingerprint calculation process includes, but is not limited to: splitting the same data block (or EC-coded block) into q sub-blocks, each of equal size, and performing an XOR operation on each sub-block within the same data block (or EC-coded block) to obtain the data fingerprint of that data block (or EC-coded block). The obtained data fingerprints of each data block and each EC-coded block also satisfy the above matrix relationship:

[0080]

[0081] Example 1

[0082] The data processing flow is as follows:

[0083] (1) Divide the original data into k data blocks of equal size, each data block being M bytes in size.

[0084] (2) Divide each data block into M / B sub-blocks of size B bytes. The value of B can be set according to the size of the register (e.g., 128 bits / 256 bits / 512 bits).

[0085] (3) Traverse all M / B sub-blocks, assuming that the j-th sub-block is currently being processed.

[0086] (4) Traverse the i-th data block among the k data blocks, assuming that the i-th data block is currently being processed.

[0087] (5) Read the j-th sub-block from the i-th data block and store it in register R. Sij The register storing erasure coding results, cyclic redundancy check data, and data fingerprints is initially set to 0.

[0088] (6) Take the j-th column of the encoding matrix and compare it with register R. Sij Multiplying the values ​​in the matrix yields r values, which are g. 1,j *S ij g 2,j *S ij , ..., g r,j *S ij Each value is then XORed with the corresponding register storing the EC encoded block, i.e., XOR = gz,j *S ij The incremental result of the j-th sub-block of each EC coding block is obtained and appended to the register storing the erasure coding results. (When the j-th sub-block of k data blocks {S 1j S 2j S kj After traversing all data blocks, the j-th sub-block of each EC-coded block is obtained. After traversing all q sub-blocks within each of the k data blocks, the EC-coded blocks are obtained.

[0089] (7) Using the algorithm corresponding to cyclic redundancy check, register R Sij The values ​​in the register are processed, and the result is further processed along with the value in the register storing the cyclic redundancy check (CRUD) data to obtain the incremental check data for the i-th data block. This incremental check data is then updated in the register storing the CRUD check data. (After all M / B sub-blocks in the i-th data block have been traversed, the register storing the CRUD check data contains the check data for the i-th data block.)

[0090] (8) According to the algorithm for generating data fingerprints, process register R Sij The values ​​in the register are processed, and the processing result is further processed together with the value in the register storing the data fingerprint to obtain the incremental data fingerprint of the i-th data block, which is then updated in the register storing the data fingerprint. (After all M / B sub-blocks in the i-th data block have been traversed, the register storing the data fingerprint contains the data fingerprint of the i-th data block.)

[0091] (9) Return to step (4) until all q sub-blocks included in each of the k data blocks have been traversed.

[0092] (10) After step (9) is completed, the j-th sub-block of each EC-encoded block is stored. Next, following the incremental calculation method in steps (7) and (8), the incremental check data and incremental data fingerprint of each EC-encoded block are calculated and stored in the register. Then, the memory stored in the EC-encoded block is released, and the occupied register is handed over to the next sub-block for processing. At this time, the j-th sub-block of each EC-encoded block is stored. After all M / B sub-blocks have been traversed, the check data and data fingerprint corresponding to the EC-encoded block are stored.

[0093] (11) Return to step (3) until all M / B sub-blocks have been processed.

[0094] (12) After step (9) is completed, the check data and data fingerprint of each data block / EC code block are stored in the register. At this time, the check data and data fingerprint of each data block / EC code block are the size of the register, which can be further calculated and processed to output the corresponding value according to the required check data and data fingerprint size.

[0095] This embodiment performs iterative operations on each data block at a granularity of B bytes (i.e., a sub-block). Each operation reads memory once, stores the sub-block in a register, and then performs operations between registers through instructions to complete three data processing operations: erasure coding, cyclic redundancy check, and data fingerprint generation, thereby reducing memory bandwidth consumption.

[0096] Example 2

[0097] In the first embodiment described above, the intermediate calculation results of the three data processing operations—erasure coding, cyclic redundancy check, and data fingerprint generation—occupy a large number of registers. However, the number of registers in the processor 100 is limited, therefore, it may not be possible to store all the data in the registers at once. To address this scenario, this disclosure adopts the following data processing method:

[0098] (1) Divide the k data blocks into m groups. The number of data blocks in each group is determined according to the number of registers. It is necessary to ensure that incremental data can be stored in registers when performing three data processing operations on each group of data blocks: erasure coding, cyclic redundancy check, and data fingerprint generation.

[0099] (2) Iterate through m sets of data.

[0100] (3) When processing the uth group of data according to the steps in Example 1:

[0101] Step (5) is changed to "Read the j-th sub-block from the i-th data block and store it in register R". SijIn the middle. Read the j-th sub-block from each EC-coded block into the register. The register storing the erasure coding result, cyclic redundancy check data, and data fingerprint is initially set to 0.

[0102] In step (6), after the j-th sub-block in all data blocks has been traversed, the incremental result of the j-th sub-block in the EC-coded block calculated from the data blocks in the first u groups is stored from the register into memory. When all m groups have been processed, the j-th sub-block in the EC-coded block is stored.

[0103] In steps (7) and (8), if the sub-block to which the cyclic redundancy check and data fingerprint generation are targeted is a sub-block in the EC-coded block, then the operation of calculating the check data and data fingerprint is only performed when processing the m-th group; it is not performed when processing the first m-1 groups.

[0104] In step (12), after obtaining the verification data and data fingerprint of the data block in the u-th group, the verification data and data fingerprint of the data block in the u-th group can be written into memory, the register is released, and then the data block of the next group is processed.

[0105] It is understood that although the above embodiments describe multiple data processing operations that simultaneously include erasure coding, cyclic redundancy check, and data fingerprint generation, in practical applications, multiple data processing operations may also include only any two of these operations, or include any one of the above operations with other types of operations, and all of these can achieve the technical effects of reducing repeated data reading and writing, reducing memory bandwidth consumption, and reducing processor resource consumption.

[0106] The solution of this disclosure implements multiple data processing operations through a single memory read operation, and can perform data migration and calculation entirely between registers, achieving extremely low memory bandwidth consumption, reducing instruction overhead in the data processing process, significantly reducing data processing time, and improving end-to-end performance.

[0107] The data processing scheme of this disclosure can be applied to the Elastic Block Store (EBS) application scenario. In this application scenario, the processor can be a cloud processor, the external storage unit of the processor can be a cloud disk provided by the EBS service, and the internal storage unit of the processor can be a register.

[0108] like Figure 5 As shown in the embodiments of this disclosure, a data processing method is also provided, applied to a cloud server. The cloud server includes a cloud disk and a cloud processor, and the registers of the cloud disk and the cloud processor are connected via an I / O stack. The method includes:

[0109] Step S22: Retrieve raw data from the cloud drive;

[0110] Step S24: Write the raw data to the register using the I / O stack;

[0111] Step S26: Perform multiple data processing operations on the original data stored in the register to obtain multiple processing results of the original data;

[0112] Step S28: Write the multiple processing results of the original data from the register to the cloud disk through the IO stack.

[0113] For detailed implementation of the embodiments disclosed herein, please refer to the foregoing embodiments, which will not be repeated here.

[0114] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0115] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0116] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0117] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computing device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0118] The systems, devices, modules, or units described in the above embodiments can be implemented by computer devices or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0119] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0120] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. A data processing method, the method comprising: Retrieve raw data from the processor's external storage unit; The raw data is written into the processor's internal storage unit; Multiple data processing operations are performed on the original data stored in the internal storage unit to obtain multiple processing results of the original data; Multiple processing results of the original data are written from the internal storage unit to the external storage unit.

2. The method according to claim 1, wherein the original data comprises multiple data blocks, and each of the multiple data blocks comprises multiple sub-blocks; The step of writing the raw data into the processor's internal storage unit includes: Write the current sub-blocks included in at least one of the plurality of data blocks into the processor's internal storage unit; The process involves performing multiple data processing operations on the original data stored in the internal storage unit to obtain multiple processing results for the original data, including: Based on the current sub-blocks included in each of the at least one data block, the multiple current processing results of the original data stored in the internal storage unit are incrementally updated; The multiple current processing results correspond to multiple data processing operations, and any one current processing result is obtained by performing the corresponding data processing operation on the preceding sub-block of the current sub-block included in the at least one data block.

3. The method according to claim 2, further comprising: If the current sub-block included in each of the at least one data block is not the last sub-block included in each of the at least one data block, the next sub-block included in the current sub-block of each of the at least one data block is updated to the current sub-block included in each of the at least one data block, and the process returns to the step of writing the current sub-block included in at least one of the plurality of data blocks into the processor's internal memory unit.

4. The method according to claim 2, wherein writing the multiple processing results of the original data from the internal storage unit to the external storage unit comprises: If the current sub-block included in each of the at least one data block is the last sub-block included in each of the at least one data block, then the multiple current processing results after incremental update of the original data are written from the internal storage unit to the external storage unit.

5. The method according to claim 2, wherein the at least one data block comprises a single data block, and the current sub-block comprises a single sub-block within the single data block; the plurality of data processing operations include a first data processing operation, the first data processing operation being used to process the sub-block within the single data block; The incremental update of multiple current processing results of the original data stored in the internal storage unit based on the current sub-blocks included in the at least one data block includes: The j-th sub-block of the i-th data block is taken as the current sub-block. The first data processing operation is performed on the j-th sub-block of the i-th data block and the first current processing result after the (j-1)-th incremental update of the i-th data block stored in the internal storage unit to obtain the first current processing result after the j-th incremental update of the i-th data block. The first current processing result after the j-th incremental update of the i-th data block is then stored in the internal storage unit. The first current processing result of the first incremental update of the i-th data block stored in the internal storage unit is obtained by performing the first data processing operation on the first sub-block of the i-th data block and a null value or a specified initial value, where i and j are both positive integers and j is greater than 1.

6. The method according to claim 5, wherein the first data processing operation is an operation of obtaining a data fingerprint, and the first current processing result after the j-th incremental update of the i-th data block is the data fingerprint after the j-th incremental update of the i-th data block.

7. The method according to claim 5, wherein the first data processing operation is cyclic redundancy check, and the first current processing result after the j-th incremental update of the i-th data block is the check data after the j-th incremental update of the i-th data block.

8. The method according to claim 2, wherein the at least one data block comprises at least two data blocks, and the current sub-block comprises individual sub-blocks respectively included in the plurality of data blocks; the plurality of data processing operations include a second data processing operation, the second data processing operation being used to process the sub-blocks in the at least two data blocks; The incremental update of multiple current processing results of the original data stored in the internal storage unit based on the current sub-blocks included in the at least one data block includes: Take the j-th sub-block included in each of the at least two data blocks as the current sub-block, and perform the second data processing operation on the j-th sub-block included in each of the at least two data blocks to obtain the j-th second current processing result of the at least two data blocks; The j-th second current processing result of the at least two data blocks is appended to the first j-1 second current processing results of the at least two data blocks stored in the internal storage unit to obtain the first j second current processing results of the at least two data blocks, and the first j second current processing results of the at least two data blocks are stored in the internal storage unit. The first second current processing result of the at least two data blocks stored in the internal storage unit is obtained by performing the second data processing operation on the first sub-blocks included in each of the at least two data blocks.

9. The method according to claim 8, wherein the second data processing operation is an encoding operation, and the second current processing result of the at least two data blocks is an encoded block obtained by encoding the current sub-blocks included in the two data blocks respectively.

10. The method of claim 9, wherein the plurality of data blocks are divided into at least two groups, and the at least two data blocks are data blocks in the same group; the method further comprises: If the j-th sub-block included in each of the at least two data blocks is the last sub-block of the at least two data blocks of the current group in the at least two groups, update the next group of the current group in the at least two groups to the current group, and return to the step of obtaining the original data from the processor's external storage unit.

11. The method according to any one of claims 1-10, wherein the original data is data that has not been processed by any one of the plurality of data processing operations; or The original data is the current processing result obtained by processing at least one of the multiple data processing operations.

12. A data processing method applied to a cloud server, the cloud server including a cloud disk and a cloud processor, wherein the registers of the cloud disk and the cloud processor are connected via an I / O stack; the method includes: Obtain the raw data from the cloud drive; The raw data is written to the register using the I / O stack; Multiple data processing operations are performed on the original data stored in the register to obtain multiple processing results of the original data; The I / O stack is used to write multiple processing results of the original data from the register to the cloud disk.

13. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method of any one of claims 1 to 12.

14. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method of any one of claims 1 to 12.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

Images