Data processing method, device and equipment

Abstract

The invention discloses a data processing method, device and equipment, and relates to the technical field of computers, which comprises the following steps of: simultaneously carrying out erasure code coding calculation and cyclic redundancy check calculation on a data block in a continuous time period when original data is loaded to a processor cache, so that multiple calculation operations on the same data are completed in a cache period, and the data processing efficiency is improved. The time for waiting for data loading of the processor and the memory bandwidth occupation are reduced, the cache utilization rate is improved, the total delay of data processing is reduced, and the input / output path is shortened. Therefore, the technical problems of repeated data loading and frequent swap-in and swap-out of the cache caused by multiple serial calculations are solved, and the technical effect of improving the data processing efficiency and the throughput capacity is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, and device. Background Technology

[0002] Distributed storage systems have become the core architecture of modern data storage. In these systems, data is distributed across multiple nodes, which improves scalability and fault tolerance, but also presents challenges in reliability and storage costs. Data may be lost or corrupted during transmission or storage due to hardware failures or network fluctuations, severely impacting integrity. While traditional multi-replica strategies can improve reliability, they require significant storage resources and are costly, making them unsuitable for large-scale applications. To address this, distributed storage systems introduce erasure coding (EC) computation. By adding verification information to the original data, data fault tolerance and recovery are achieved, while significantly reducing storage overhead and improving resource utilization.

[0003] Cyclic Redundancy Check (CRC), as an efficient data error detection technology, generates checksums during data transmission and storage, and compares them during retrieval to determine whether data errors have occurred, ensuring data integrity. However, in all-flash environments, with the performance improvements of Solid State Drives (SSDs), the widespread adoption of high-capacity QLC (Quad-Level Cell) SSDs, and the expansion of cluster scale, massive data processing places higher demands on processor computing power and cache resources. Insufficient processor computing power and cache strain can lead to decreased processing speed and increased system response latency.

[0004] More seriously, EC and CRC calculations typically require processing the same data multiple times. During the write path, the system first performs EC encoding calculation, then performs CRC calculation for multiple data loads before transmission or storage. This repeated loading and calculation not only increases processing latency and lengthens the I / O path, but also consumes significant storage device bandwidth. Frequent cache swapping reduces the hit rate, causing the processor to repeatedly wait for data loading, creating a vicious cycle that increases the computational burden, reduces the efficiency of storage devices and cache utilization, and limits the system's throughput in high-performance, large-scale scenarios. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and device to at least solve the technical problems of repeated data loading and frequent cache swapping caused by multiple serial calculations in related technologies, thereby achieving the technical effect of improving data processing efficiency and throughput.

[0006] This application provides a data processing method applied to a processor. The data processing method includes: acquiring raw data to be processed; dividing the raw data into multiple raw data blocks; loading the raw data blocks into the processor's cache; performing erasure coding calculation on the raw data blocks during the time period during which the raw data blocks are loaded into the cache to obtain a first check data block; and performing cyclic redundancy check calculation on the raw data blocks to obtain a second check data; and outputting the first check data block and the second check data.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above data processing methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0010] This application achieves this by simultaneously performing erasure coding and cyclic redundancy check calculations on the data block during the period when the original data is loaded into the processor cache. This allows multiple calculations on the same data to be completed during the caching period, reducing the processor's waiting time for data loading and memory bandwidth usage, improving cache utilization, reducing the overall latency of data processing, and shortening the input / output path. Therefore, this application solves the technical problems of repetitive data loading and frequent cache swapping caused by multiple serial calculations, achieving the technical effect of improving data processing efficiency and throughput. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a data processing method in related technologies;

[0013] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0014] Figure 3A flowchart of data prefetching and loading is provided for embodiments of this application;

[0015] Figure 4 A flowchart of a data fusion calculation provided in an embodiment of this application;

[0016] Figure 5 A flowchart illustrating another data fusion calculation provided in this application embodiment;

[0017] Figure 6 This is a schematic diagram illustrating different verification strategies provided in the embodiments of this application;

[0018] Figure 7 A flowchart illustrating the verification process of a receiver for different verification strategies, provided as an embodiment of this application;

[0019] Figure 8 A flowchart illustrating the data loading and verification process at a receiving end, as provided in this application embodiment;

[0020] Figure 9 This is a flowchart illustrating the loading and verification of data from a storage device, as provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1In related technologies, the first loading and calculation (for network transmission verification) involves: firstly, loading the original data from the storage medium (such as memory) into the processor's computing unit (such as cache); performing a Cyclic Redundancy Check (CRC) calculation on the loaded original data; and storing the calculated CRC check value (transmission CRC) to be sent along with the data for verifying the accuracy of network transmission. The second loading and calculation (for data fault tolerance) involves loading the same original data from the storage medium into the processor again; performing erasure coding (EC) encoding calculation on the reloaded original data to generate a check block; and storing the calculated EC check block for data recovery in case of loss. The third loading and calculation (for persistent storage verification) involves loading the same original data from the storage medium into the processor a third time; performing a Cyclic Redundancy Check (CRC) calculation on the reloaded original data (this time typically to generate check values ​​at different granularities, such as disk sector size); and writing the calculated CRC check value (persistent CRC) along with the data into a persistent storage device (such as an SSD) for verifying data integrity during subsequent data retrieval.

[0025] The problem is that the same original data was loaded and calculated three times independently. The multiple data loading consumed a lot of memory bandwidth and processor time. Each calculation required reloading the data, resulting in a low cache hit rate, serious waste of cache resources, a lengthened I / O path, and an increase in system response time.

[0026] To address the aforementioned technical problems, this application provides a data processing method. First, the specific application environment architecture or hardware architecture upon which the execution of this data processing method depends is described below. This data processing method is primarily applied to a processor used for sending or receiving data in a distributed storage system. This processor typically includes multi-level caches and a bus interface connected to high-performance memory and storage devices. Under this architecture, raw data blocks can be loaded and accessed at high speed in the cache, supporting parallel or serial erasure coding and cyclic redundancy check (CRC) calculations during the time the data block is loaded into the cache. This fully utilizes cache bandwidth and processor computing power, reduces access to low-speed storage devices, and improves data processing efficiency. Furthermore, this hardware architecture may also include a non-volatile storage device connected to the processor's bus interface for storing generated erasure code verification data (first verification data block) and CRC verification data (second verification data). By optimizing the data transfer path between the cache and storage devices, the performance overhead caused by redundant data loading can be reduced. Simultaneously, it adapts to all-flash environments and large-capacity QLC SSD clusters, achieving high-performance, large-scale distributed storage data reliability assurance.

[0027] like Figure 2The embodiments of this application provide a data processing method, and the method is described in detail below in conjunction with the execution flow of the data processing method.

[0028] This method is applied to a processor, and the data processing methods include:

[0029] S11: Obtain the raw data to be processed, divide the raw data into multiple raw data blocks, and load the raw data blocks into the processor's cache;

[0030] In this embodiment, the original data to be processed is first obtained, such as... Figure 3 The data is then divided according to a pre-defined K+M erasure coding scheme. Specifically, the original data is divided into K data blocks, and memory storage space is allocated for M first check data blocks. This partitioning method provides the basic units for subsequent erasure coding calculations during the data processing stage. Each data block is cached and calculated as an independent processing unit, thereby ensuring the processing independence between data blocks and the controllability of the calculation.

[0031] K data blocks are loaded sequentially into the processor's cache according to stripe unit granularity. This loading process leverages the high-speed access characteristics of multi-level caches, allowing the processor to directly access the original data blocks during their cached duration, reducing frequent accesses to slow storage devices. By maintaining the residency of data blocks in the cache, cache hit rate can be improved in subsequent computations, reducing latency and resource overhead caused by data movement.

[0032] Furthermore, this step is designed to ensure that memory allocation and computation operations are performed on the same NUMA node, meaning that the memory and processor core reside within the same NUMA domain. This arrangement reduces cross-node access latency and improves memory bandwidth utilization efficiency.

[0033] S12: During the time period when the original data block is loaded into the cache, the erasure coding calculation is performed on the original data block to obtain the first check data block, and the cyclic redundancy check calculation is performed on the original data block to obtain the second check data.

[0034] In this embodiment, erasure coding (EC) is first performed on the original data block loaded into the cache. Taking Reed-Solomon (4+2) as an example, the 1MB of original data is divided into 32KB stripe units, and EC is performed stripe by stripe to generate two 32KB parity blocks. During this calculation process, SIMD instructions are called to achieve high parallelism based on the processor instruction set support, thereby improving the execution efficiency of EC encoding calculation. The EC calculation result is directly written to the processor's registers or the stripe buffer in the processor's cache, without the need to write back to the storage device, reducing data transfer overhead.

[0035] In addition, a Cyclic Redundancy Check (CRC) calculation is performed on the original data blocks still residing in the cache. The CRC calculation reads the original data blocks directly from the cache, rather than reloading them from storage, to fully utilize the "hot" nature of data in the cache. Depending on the processor instruction set optimizations, a dedicated CRC instruction is invoked to output a second checksum, thereby achieving efficient data integrity detection.

[0036] Throughout the computation, the original data block remains in a hot state in the cache, and intermediate calculation results are only transferred between the cache and registers without being written back to memory. This design maximizes the use of the spatiotemporal locality of data, reduces memory access latency, and minimizes cache pollution and bandwidth consumption. After the EC calculation is completed, the original data is still directly accessible for CRC calculation, enabling continuous processing without repeated loading.

[0037] This cache-resident and serial / parallel computing approach enables EC encoding and CRC calculation to be completed within the same data block loading cycle, effectively improving data processing efficiency, shortening input / output paths, reducing processor load, fully leveraging the performance advantages of multi-level caching and SIMD parallel computing, while ensuring data reliability and integrity.

[0038] In one exemplary embodiment, during the duration of the original data block being loaded into the cache, erasure coding calculation is performed on the original data block to obtain a first check data block, and cyclic redundancy check calculation is performed on the original data block to obtain a second check data. This includes: during the duration of the original data block being loaded into the cache, erasure coding calculation is performed on the original data block in a serial calculation order to obtain a first check data block, and the first check data block is stored in the cache; after obtaining the first check data block, cyclic redundancy check calculation is performed on the original data block in the cache to obtain the second check data, and the second check data is stored in the cache.

[0039] In this embodiment, during erasure coding calculation, the processor divides the original data block in the cache into several stripe units and performs parallel or segmented calculations according to a preset erasure coding scheme to generate the first parity data block. This process utilizes the processor's registers and instruction set to complete the data encoding operation, and the calculation result is directly written to the cache without being written back through the storage device, thereby keeping the data in a high-speed access state during the calculation process.

[0040] After obtaining the first checksum data block, the processor continues to perform cyclic redundancy check (CRC) calculations on the original data blocks in the cache within the same cache cycle to generate the second checksum data. Since the original data blocks still reside in the cache, the processor can directly access the data for CRC calculations without reloading the data from memory. After the CRC calculation is complete, the resulting second checksum data is also stored in the cache. Once all original data blocks have undergone erasure coding and CRC calculations, the original data blocks, the first checksum data block, and the second checksum data are output to the connected storage device via the processor's bus interface, realizing the writing and storage of the checksum data. The entire calculation process is completed within the cache space, avoiding frequent data movement.

[0041] This embodiment completes two types of calculations within the cache residency period, enabling erasure coding and cyclic redundancy check (CRC) calculations to be executed sequentially within the same data loading cycle. Since the computation core directly processes the cached data, both the original data block and the generated check data block are temporarily retained in the cache, ensuring continuous data processing, simplified data paths, and a clear computational sequence. This design guarantees that after encoding and verification are completed, the data can be directly output to the storage device via the bus interface, achieving synchronous generation and output of original data and check data within the same processing flow.

[0042] In one exemplary embodiment, during the duration of the original data block being loaded into the cache, erasure coding calculation is performed on the original data block to obtain a first check data block, and cyclic redundancy check calculation is performed on the original data block to obtain a second check data. This includes: during the duration of the original data block being loaded into the cache, erasure coding calculation and cyclic redundancy check calculation are performed on the original data block simultaneously in a parallel computing manner to obtain the first check data block and the second check data, respectively; the first check data block is stored in the cache, and the second check data is stored in the cache.

[0043] like Figure 4 In this embodiment, after the original data block is loaded into the processor cache, the processor performs erasure coding and cyclic redundancy check (CRC) calculations simultaneously in parallel during the duration the data resides in the cache. Specifically, the processor divides the original data block into several independent data segments according to the instruction scheduling mechanism, with each segment corresponding to a different computation thread or execution unit. Some execution units are responsible for performing erasure coding on the original data block to generate a first check data block; other execution units simultaneously perform CRC calculations on the same original data block to generate a second check data block. Since the two calculations are performed in parallel within the same data loading cycle, the original data block does not need to be repeatedly loaded or copied and can be accessed by both computational logics simultaneously.

[0044] During erasure coding calculations, the processor performs group calculations on each data segment in the cache based on a preset coding matrix. The calculation results are written to registers or the cache area in real time to form the first check data block. Meanwhile, the cyclic redundancy check (CRC) calculation unit reads the contents of the same original data block from the cache within the same time period and generates the second check data through CRC calculation. Throughout the entire calculation process, data is transferred between the cache, registers, and execution units without any write-back to memory. By executing the two calculations in parallel within the same cache cycle, the processor can maintain the continuous existence of data in the high-speed access path, reducing the round-trip loading of data between memory and cache.

[0045] In this embodiment, the result of erasure coding calculation, i.e., the first check data block, is retained in the cache for subsequent data output or persistent storage; simultaneously, the second check data obtained from cyclic redundancy check (CRC) calculation is also stored in the cache. After all original data blocks have undergone erasure coding and CRC calculation, the original data blocks, the first check data block, and the second check data are directly output to the connected storage device through the processor's bus interface. This embodiment achieves parallel execution of EC and CRC calculations within the cache residency period, enabling continuous processing of original data, check data blocks, and output data in the same data path, ensuring the efficiency and consistency of the data processing flow.

[0046] S13: Output the first check data block and the second check data.

[0047] In this embodiment, the erasure coding result and the cyclic redundancy check result are quickly output to a designated storage medium or downstream processing unit after the calculation is completed. The processor keeps the cached data active during the output phase, ensuring that access to and output of the verification data are completed within the same data residency cycle. Through this processing logic, continuous operation from calculation to output can be achieved without repeatedly loading or writing back the original data, thereby ensuring the stability and consistency of the data processing flow.

[0048] like Figure 5The core idea of ​​this application is to completely solve the performance bottleneck and resource waste caused by multiple serial processing of the same data in traditional architectures through single data loading and policy-driven parallel computing. The specific process is as follows: First, the original data block corresponding to the original data is loaded into the cache all at once; then, according to a preset configurable policy, erasure coding (EC) calculation and cyclic redundancy check (CRC) calculation are performed synchronously or immediately; finally, the check block generated by the EC calculation is directly output, while the result of a single CRC calculation can, under the scheduling of the policy, simultaneously meet the dual requirements of network transmission verification and data persistence verification, achieving dual-purpose computation. This design, by maximizing cache reuse and eliminating redundant data handling and computation, achieves deep optimization of processor, cache, and memory bandwidth resources, significantly improving data processing efficiency.

[0049] In one exemplary embodiment, a cyclic redundancy check (CRC) calculation is performed on the original data block to obtain second check data, including:

[0050] The target verification strategy is determined based on the preset requirements; different verification strategies correspond to different data granularities.

[0051] Based on the target verification strategy, the original data block is subjected to cyclic redundancy check calculation according to the target verification granularity to obtain the second verification data;

[0052] The second verification data is associated with the corresponding erasure coding data block and output through the processor's bus interface, which is configured to connect to the storage device.

[0053] In this embodiment, the process of performing cyclic redundancy check (CRC) calculations on the original data block to obtain the second check data first requires determining the target check strategy. The determination of the target check strategy is automatically completed by the system based on current preset requirements, such as selecting different data granularities according to the data processing context, transmission path, or storage method. Different check strategies correspond to different check granularities, such as global check based on data fragments or local check based on sectors. In this embodiment, after loading the original data block, the processor reads the strategy parameters through internal control logic and selects the target check strategy matching the current task, thereby clarifying the granularity and calculation range required for subsequent CRC calculations.

[0054] After determining the target verification strategy, the processor performs cyclic redundancy check (CRC) calculations on the original data block according to the strategy. This calculation process is based on a preset CRC polynomial implementation; the processor calls the CRC instruction from its internal instruction set to calculate the CRC value byte-by-byte on the data loaded in the cache. In this embodiment, the CRC calculation does not rely on externally stored intermediate data but is performed directly on the cached data, and the calculation result is stored in a register or cache. In this way, the CRC calculation can be completed within the duration of the data residing in the cache, reducing round-trip operations between memory and cache. The calculated second checksum is marked as associated with the corresponding original data block so that a one-to-one match can be performed in the subsequent output stage.

[0055] After the second verification data is generated, the processor outputs it via the internal bus interface. In this embodiment, the bus interface is configured to connect to a storage device, and the second verification data is transmitted to a designated storage area via the bus for subsequent data integrity or redundancy verification. During the output phase, the processor uses its internal transmission scheduling unit to ensure that each second verification data item maintains a correspondence with the corresponding erasure coding calculation data block, thus enabling correct association during subsequent verification or recovery processes. This embodiment completes the data association and output during the verification calculation phase, allowing the correspondence between cyclic redundancy check data and erasure coding data to be naturally formed within the processing flow, ensuring data processing consistency and traceability.

[0056] like Figure 6 In one exemplary embodiment, the verification strategy includes at least a first verification strategy and a second verification strategy.

[0057] The first verification strategy is configured to generate a global cyclic redundancy check code and use it as the second verification data, with the data fragments used for erasure coding calculation as the granularity.

[0058] The second verification strategy is configured to calculate an independent cyclic redundancy check (CRC) code corresponding to each sector of data, using the sector size of the storage device connected to the processor's bus interface as the granularity, and to use all independent CRC codes as the second verification data.

[0059] In this embodiment, the verification strategy includes at least a first verification strategy and a second verification strategy. The main difference between the two strategies lies in the data granularity and application scenario of the cyclic redundancy check (CRC) calculation. The first verification strategy is configured to generate a global CRC code at the granularity of the data fragments used for erasure coding calculation, and output this check code as the second verification data. In this embodiment, when performing erasure coding calculation, the original data is usually divided into several fragments. For example, in the (4+2) EC scheme, 1MB of original data is divided into 4 data fragments, each fragment being 256KB in size. According to the first verification strategy, the processor calculates a CRC value for each data fragment in 256KB units. This CRC value is used to describe the integrity status of the entire data fragment, thereby enabling overall data verification during network transmission or distributed writing.

[0060] When executing the first verification strategy, after loading the original data block, the processor reads the data piece by piece from the cache and performs cyclic redundancy check (CRC) calculations through internal instructions. Each CRC calculation is bounded by a fixed piece length, and the calculation result corresponds to the checksum of a complete piece. In this embodiment, after completing the CRC calculation for a data piece, the processor stores the calculation result in the cache area corresponding to that piece for subsequent transmission encapsulation. For persistent write requirements in some system architectures, after completing the CRC calculation at the global piece granularity, this embodiment can also repeat a round of CRC calculation at another granularity (e.g., 4KB) according to a preset configuration, so as to reserve checksums for subsequent storage stages and realize the verification of the same data at different levels.

[0061] The second verification strategy is configured to calculate an independent cyclic redundancy check (CRC) code for each sector of data, using the sector size of the storage device connected to the processor as the granularity, and output all independent CRC codes as the second verification data. In this embodiment, the processor performs CRC calculations separately for each sector of data according to the sector size agreed upon with the SSD device interface (e.g., 4KB). After generating an independent check code for each sector, the processor stores the check code along with the logical address of the sector data. This strategy allows the verification results to be used directly during SSD write or read operations, enabling rapid verification of data correctness on the storage device. When performing this type of CRC calculation, the processor still completes data loading and computation in the cache, and the results are stored in the structure or pointer location corresponding to the EC data block.

[0062] After the CRC calculation is completed, the result is organized and output according to its corresponding verification strategy. In this embodiment, independent result storage areas are allocated for different verification strategies. For the global CRC value generated by the first verification strategy, the result is stored in the index information of the corresponding EC data fragment; for multiple sector-level CRC values ​​generated by the second verification strategy, the processor writes the result to the memory address corresponding to each sector. This address is usually adjacent to the data block so that it can be directly accessed during data transmission or persistent writing. By establishing a clear association between each CRC result and its corresponding data block in the output stage, this embodiment can quickly retrieve matching CRC values ​​in subsequent data transmission or verification stages, thereby maintaining the continuity and consistency of the data verification process.

[0063] like Figure 7 In one exemplary embodiment, it further includes:

[0064] When receiving a set of data blocks and second check data from the sender, the identifier of the target check strategy corresponding to the second check data is obtained; the set of data blocks includes the original data blocks and the first check data blocks generated by erasure coding.

[0065] Based on the identifier of the target verification strategy, determine the target verification method for performing cyclic redundancy check on the data block set;

[0066] The correctness of the received data block set is verified according to the target verification method.

[0067] In this embodiment, after receiving the data block set and second check data from the sender, the processor at the receiving end parses the target check strategy identifier corresponding to the second check data from the received data packet. This check strategy identifier is written by the sender when generating the second check data and contains granularity information and calculation method for cyclic redundancy check (CRC) calculation. By reading this identifier, the receiving end can determine which check strategy the currently received data block set was generated based on, such as a global check strategy with erasure coding data fragmentation as the granularity, or a local check strategy with storage device sectors as the granularity. In this embodiment, after parsing the identifier, the processor at the receiving end temporarily stores it in the control register, providing a basis for subsequently determining the target check method.

[0068] Once the receiving end determines the target verification strategy, it determines the target verification method for cyclic redundancy check (CRC) based on the identifier. In this embodiment, if the target verification strategy identifier corresponds to the first verification strategy, the processor will perform overall CRC verification at the granularity of erasure coding fragments to verify the integrity of each fragment of data. If the identifier corresponds to the second verification strategy, it will perform CRC calculation independently on each small data block at the sector granularity to compare the correctness of each sector of data. During execution, the processor will configure the segment length of the CRC calculation unit according to the granularity parameter carried in the identifier, and sequentially read the original data blocks and the first verification data block in the data block set to complete the segment-by-segment comparison.

[0069] After determining the target verification method, the receiving end performs correctness verification on the received data block set according to the corresponding calculation strategy. In this embodiment, the processor calls the CRC verification module to compare the CRC result recalculated by the receiving end with the received second verification data. When the two match, it indicates that no errors occurred during data transmission or storage; when the results do not match, the system marks the corresponding data block as abnormal data and triggers the subsequent erasure coding decoding and recovery process as needed. In this way, this embodiment can achieve flexible correctness verification under different verification strategies, ensuring reliable reception of the data block set and correct execution of subsequent processing.

[0070] like Figure 7 and Figure 8 In one exemplary embodiment, the correctness of the received data block set is verified according to the target verification method, including:

[0071] When the identifier of the target verification strategy corresponds to the first verification strategy, the first global cyclic redundancy check (GC) code is calculated on the entire set of data blocks that have been erasure-coded, with the size of the data fragment used for erasure coding calculation as the granularity. The calculated first GC code is compared with the received second verification data. If the comparison is consistent, it is determined that the data block set is correct during transmission.

[0072] When the identifier of the target verification strategy corresponds to the second verification strategy, the independent cyclic redundancy check (CRC) code is calculated for each sector of the received erasure-coded data block set, using the sector size of the storage device connected to the processor's bus interface as the granularity. Each calculated CRC code is then compared with the corresponding part of the received second verification data. If all comparisons are consistent, the data block set is determined to be correct during transmission, and the received second verification data is directly associated with the corresponding data block for use when storing it in the storage device.

[0073] In this embodiment, when verifying the correctness of the received data block set according to the target verification method, if the identifier of the target verification strategy corresponds to the first verification strategy, the receiving end performs global cyclic redundancy check (CRC) calculation on the entire erasure-coded data block set with the data fragment size used for erasure coding calculation as the granularity. This process involves the receiving end's processor calling the CRC module to perform polynomial division on the contents of each data fragment sequentially, generating a first CRC code. Subsequently, this first CRC code is compared with the second verification data received along with the data block set. If the comparison results match, it indicates that the data block set has remained correct during network transmission, without bit errors or data corruption; if the comparison results do not match, an abnormal state is recorded, and the erasure coding decoding and recovery process is triggered to ensure the integrity and recoverability of the data.

[0074] When the target verification strategy identifier corresponds to the second verification strategy, the receiving end performs independent cyclic redundancy check (CRC) calculations on each sector of the received erasure-coded data block set, using the sector size of the storage device connected to the processor as the granularity. During this process, the processor sequentially reads the data content of each sector, generates a corresponding CRC code for each sector, and compares the calculation result with the corresponding data segment in the received second verification data. This sector-by-sector verification method can precisely locate specific erroneous sectors, enabling the system to recover only the erroneous portion when data anomalies occur, without needing to re-decode the entire data block.

[0075] After completing all sector-level comparisons, when all calculated independent cyclic redundancy check (CRC) codes match the corresponding portions of the received second check data, the receiving end determines that the data block set has remained correct during transmission. At this point, an association is established between the received second check data and the corresponding erasure-coded data block, and their corresponding logical addresses are recorded in the internal storage mapping table. This allows for direct retrieval of the check data during subsequent persistent storage on the storage device, achieving seamless integration with the persistent storage process.

[0076] In this embodiment, the entire correctness verification process is completed locally at the receiving end without reloading or converting the data structure, ensuring the consistency and reliability of the data during transmission and disk storage.

[0077] like Figure 9 In one exemplary embodiment, it further includes:

[0078] Load the target data block and the cyclic redundancy check code associated with the target data block from the storage device connected to the processor's bus interface;

[0079] Perform cyclic redundancy check on the loaded target data block;

[0080] When the cyclic redundancy check passes and the identifier of the check strategy corresponds to the first check strategy, the global cyclic redundancy check code is calculated for the target data block with the data fragment size used for erasure coding calculation as the granularity, and the calculated global cyclic redundancy check code is used for subsequent network transmission verification.

[0081] When the cyclic redundancy check passes and the check strategy identifier corresponds to the second check strategy, the cyclic redundancy check code loaded from the storage device will be directly used for subsequent network transmission check.

[0082] In this embodiment, by loading the target data block stored in the storage device and its corresponding cyclic redundancy check (CRC) code, seamless data verification between persistent storage and network transmission is achieved. Specifically, CRC is first performed on the target data loaded from the storage device to verify the integrity and correctness of the current data during the storage phase, thereby preventing error propagation caused by storage medium errors or read anomalies. When the CRC result indicates that the data is correct, different subsequent processing paths are selected based on the verification strategy adopted.

[0083] When the verification strategy corresponds to the first verification strategy, the global cyclic redundancy check (CRC) code is recalculated on the target data, using the size of the erasure coding-encoded data fragments as the verification granularity. This global check code reflects the overall consistency of the data at the logical level and can be used as the basis for integrity verification during subsequent network transmission to detect data corruption or packet loss across nodes and links. When the verification strategy corresponds to the second verification strategy, there is no need for recalculation; instead, the CRC code loaded from the storage device is directly reused as the verification data during the transmission stage. This design reduces the processing overhead caused by repetitive calculations, achieves resource optimization and verification efficiency balance under different verification strategies, and ensures the correctness and consistency of data at each stage from storage to transmission.

[0084] Furthermore, this embodiment achieves a dynamic balance between data verification overhead and real-time performance in different application scenarios by distinguishing between a first verification strategy and a second verification strategy. Specifically, the first verification strategy is suitable for scenarios with high integrity requirements and long data transmission links, such as cross-node synchronization or remote backup. In this case, by recalculating the global cyclic redundancy check (CRC) code, the overall consistency and end-to-end reliability of the currently transmitted data can be ensured. The second verification strategy is more suitable for high-speed transmission scenarios between storage devices or within the same node. The system directly reuses the CRC generated in the storage stage without repeated calculation, thereby significantly reducing processor computational load and system latency, and improving data forwarding and reuse efficiency.

[0085] Furthermore, this embodiment can incorporate a metadata management mechanism into the verification process design. By synchronously loading the associated policy identifier and checksum mapping information during the data block loading stage, the system can automatically identify the verification policy corresponding to the target data block, completing the automatic branching judgment of the verification logic without manual intervention. For example, in one specific implementation, a corresponding metadata entry is established for each data block in the storage device. The metadata entry records the policy identifier, cyclic redundancy check (CRC) code location index, and their mapping relationship with the data block. When the processor loads the target data block from the storage device, it simultaneously reads the corresponding metadata entry and automatically determines the verification path to be used based on the policy identifier stored in the entry. For example, when the metadata entry is identified as the first verification policy, the processor automatically calls the global CRC calculation module; when it is identified as the second verification policy, it directly loads and references the CRC code at the corresponding position for rapid comparison. This process is implemented at the underlying level through the parallel execution of metadata indexing and data block loading operations, enabling policy determination and verification logic to be completed automatically during the data reading stage without additional control instructions or manual configuration. This not only improves the system's intelligence and scalability, but also allows different types of data blocks to flexibly switch strategies under a unified verification framework, thereby achieving adaptive optimization between data reliability and transmission performance.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0087] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data processing method embodiments.

[0088] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when it is run.

[0089] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0090] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0091] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0092] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0093] The data processing method, apparatus, and device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized in that, Applied to a processor, the data processing method includes: Obtain the raw data to be processed, divide the raw data into multiple raw data blocks, and load the raw data blocks into the processor's cache; During the duration of the original data block being loaded into the cache, erasure coding calculation is performed on the original data block to obtain a first check data block, and cyclic redundancy check calculation is performed on the original data block to obtain a second check data. Output the first verification data block and the second verification data.

2. The data processing method according to claim 1, characterized in that, Cyclic redundancy check (CRC) calculation is performed on the original data block to obtain second check data, including: The target verification strategy is determined based on the preset requirements; different verification strategies correspond to different data granularities. According to the target verification strategy, the original data block is subjected to cyclic redundancy check calculation at the target data granularity to obtain the second verification data; The second verification data is associated with the corresponding erasure coding data block and output through the processor's bus interface, which is configured to connect to a storage device.

3. The data processing method according to claim 2, characterized in that, The verification strategy includes at least a first verification strategy and a second verification strategy; The first verification strategy is configured to generate a global cyclic redundancy check code and use it as the second verification data, with the data fragments used for erasure coding calculation as the granularity. The second verification strategy is configured to calculate an independent cyclic redundancy check code corresponding to each sector of data, using the sector size of the storage device connected to the processor's bus interface as the granularity, and to use all independent cyclic redundancy check codes as the second verification data.

4. The data processing method according to claim 3, characterized in that, Also includes: When receiving a set of data blocks and second verification data from the sending end, the identifier of the target verification strategy corresponding to the second verification data is obtained; the set of data blocks includes the original data blocks and the first verification data block generated by erasure coding; Based on the identifier of the target verification strategy, determine the target verification method for performing cyclic redundancy check on the data block set; The correctness of the received data block set is verified according to the target verification method.

5. The data processing method according to claim 4, characterized in that, According to the target verification method, the correctness verification of the received data block set is performed, including: When the identifier of the target verification strategy corresponds to the first verification strategy, the first global cyclic redundancy check (GC) code is calculated on the entire set of data blocks that have been erasure-coded, with the data fragment size used for erasure coding calculation as the granularity; the calculated first GC code is compared with the received second verification data; if the comparison is consistent, it is determined that the set of data blocks is correct during transmission. When the identifier of the target verification strategy corresponds to the second verification strategy, using the sector size of the storage device connected to the processor as the granularity, for each sector of the received erasure-coded data block set, the corresponding independent cyclic redundancy check (CRC) code is calculated; each calculated CRC code is compared with the corresponding part of the received second verification data; if all comparisons are consistent, it is determined that the data block set is correct during transmission, and the received second verification data is directly associated with the corresponding data block for use when storing it in the storage device.

6. The data processing method according to claim 3, characterized in that, Also includes: Load the target data block and the cyclic redundancy check code associated with the target data block from the storage device connected to the bus interface of the processor; Perform cyclic redundancy check on the loaded target data block; When the cyclic redundancy check passes and the identifier of the check strategy corresponds to the first check strategy, the global cyclic redundancy check code is calculated for the target data block with the data fragment size used for erasure coding calculation as the granularity, and the calculated global cyclic redundancy check code is used for subsequent network transmission check. When the cyclic redundancy check passes and the identifier of the check strategy corresponds to the second check strategy, the cyclic redundancy check code loaded from the storage device is directly used for subsequent network transmission check.

7. The data processing method according to any one of claims 1-6, characterized in that, During the duration during which the original data block is loaded into the cache, erasure coding calculation is performed on the original data block to obtain a first check data block, and cyclic redundancy check calculation is performed on the original data block to obtain a second check data block, including: During the time period during which the original data block is loaded into the cache, the original data block is first subjected to erasure coding calculation in the order of serial calculation to obtain the first check data block, and the first check data block is stored in the cache; After obtaining the first verification data block, a cyclic redundancy check is performed on the original data block in the cache to obtain the second verification data, and the second verification data is stored in the cache.

8. The data processing method according to any one of claims 1-6, characterized in that, During the duration during which the original data block is loaded into the cache, erasure coding calculation is performed on the original data block to obtain a first check data block, and cyclic redundancy check calculation is performed on the original data block to obtain a second check data block, including: During the period when the original data block is loaded into the cache, erasure coding and cyclic redundancy check calculations are performed on the original data block simultaneously in a parallel computing manner to obtain the first check data block and the second check data, respectively. The first verification data block is stored in the cache, and the second verification data is stored in the cache.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 8.

Images