Data processing method and apparatus

Abstract

A data processing method, comprising: after an intermediate device receives the first batch of data packets of a service, the intermediate device calculating, on the basis of the first batch of data packets, an intermediate parameter related to the service, caching the intermediate parameter, and sending the first batch of data packets to a next-level device; after the intermediate device receives each subsequent batch of data packets of the service, the intermediate device updating, on the basis of the batch of data packets, the cached intermediate parameter, and sending the batch of data to the next-level device; and after the intermediate device receives the last batch of data packets of the service and completes related calculation on the basis of the last batch of data packets, the intermediate device obtaining a result parameter of the service, sending the result parameter to the next-level device, and sending the last batch of data to the next-level device. In the method, a related service can be completed while bandwidth overheads and memory resource overheads are reduced.

Description

A data processing method and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410565549.9, filed on May 8, 2024, entitled “A Data Processing Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology

[0003] In various storage scenarios, business processing is required on the raw data to be stored. For example, in storage scenarios requiring data redundancy protection, erasure coding (EC) is performed on the raw data to obtain a checksum block. Then, the raw data and the checksum block are persistently stored. The checksum block is used to recover the original data in case of loss.

[0004] Due to limitations in packet payload size, the data source needs to send raw data to the storage device in multiple packets. Each time the storage device receives a packet of raw data, it temporarily stores the data in memory, also known as main memory. When the storage device has received all the packets of the raw data, the EC (Extended Error Correction) engine reads the complete raw data from memory to calculate erasure codes. This process involves storing data in and retrieving data from memory, which increases bandwidth overhead. Furthermore, all raw data must be temporarily stored in memory before EC calculation, further increasing memory resource overhead. Summary of the Invention

[0005] This application provides a data processing method and apparatus that can complete related business while reducing bandwidth and memory resource consumption.

[0006] Firstly, a data processing method is provided. This method can be used to complete data storage-related tasks, such as erasure coding calculation, fingerprint information calculation, XOR calculation, accumulation calculation, cyclic redundancy check, data compression, and data decompression, among others. Specifically, the data to be stored, or the data required by the business, can be carried in multiple data packets. An intermediate device for executing the business can receive one or more of these data packets in batches from its upstream device. Whenever the intermediate device receives a batch of data packets, it calculates or updates the intermediate parameters of the business based on that batch of data packets and sends the batch of data packets to the next-level device. Specifically, when the intermediate device receives the first batch of data packets, it calculates the intermediate parameters related to the business based on the first batch of data packets and caches these intermediate parameters (e.g., records the intermediate parameters in the intermediate device's memory or data exchange buffer). When the intermediate device receives each subsequent batch of data packets, it updates the cached intermediate parameters based on that batch of data packets. The intermediate device obtains the result parameters of the service when it receives the last batch of data packets from the multiple data packets and completes the relevant calculations based on the last batch of data packets. It then sends the result parameters to the next-level device, as well as the last batch of data packets.

[0007] The result parameters of a business process are the data obtained when the business is completed; these can be referred to as the processing result. Intermediate parameters, on the other hand, refer to intermediate or temporary data obtained during the business process.

[0008] A batch of data packets can contain one or more packets. When a batch of data packets includes multiple data packets, these multiple data packets can be received simultaneously by intermediate devices or received at different times.

[0009] In short, in this method, each time an intermediate device receives a batch of data packets, it performs business processing on that batch to obtain intermediate parameters. Then, it can send this batch of data packets (or the valid data carried within them) to the next-level device. The intermediate parameters are used in conjunction with subsequently arriving data packets to obtain the business result parameters. Thus, after obtaining the intermediate parameters, the intermediate device can discard the received data packets, eliminating the need to temporarily store all data packets in memory, thereby saving memory resources and bandwidth.

[0010] In addition, each time an intermediate device receives a batch of data packets, it can send the batch of data packets to the next level device for storage or forwarding, thereby improving data transmission and storage efficiency.

[0011] In one possible implementation, intermediate parameters do not need to be sent to the next-level device. That is, the method also includes: the intermediate device does not send intermediate parameters to the next-level device. Intermediate parameters are temporary data that exists before the service is completed. These intermediate parameters are only cached on the intermediate device and do not need to be sent to the next-level device, thus saving communication resources while ensuring the service can continue.

[0012] In one possible implementation, the service data packets carry a portion of data to be stored. The intermediate device performs erasure coding on the received data packets to obtain checksum data, which serves as the intermediate parameter. In other words, the intermediate parameter is the checksum data calculated using EC on the service data packets received by the intermediate device. For example, the intermediate parameter includes m first checksum data blocks, where m is an integer greater than or equal to 1. Each time the intermediate device receives a batch of data packets, it updates the cached intermediate parameter by performing erasure coding on the service data carried in that batch of data packets to obtain m second checksum data blocks. The m first checksum data blocks and m second checksum data blocks correspond one-to-one. The corresponding checksum data blocks from the m first checksum data blocks and m second checksum data blocks are added together to obtain the updated intermediate parameter. Thus, through continuous iterative updates of the intermediate parameter, when the last batch of service data packets is received, the checksum data blocks of the data to be stored can be obtained, thereby achieving redundant protection of the data to be stored.

[0013] In one possible implementation, updating the cached intermediate parameters specifically involves calculating new intermediate parameters based on the currently received batch of data packets and the cached intermediate parameters. The intermediate device can first calculate the initial parameters from the currently received data packets. Then, it calculates the initial parameters and the cached intermediate parameters to obtain the new intermediate parameters. For example, in erasure coding calculation, the intermediate device calculates m checksum blocks from the currently received data packets. Then, the intermediate device adds these m checksum blocks to the cached m checksum blocks to obtain new m checksum blocks. Alternatively, the intermediate device can directly calculate the new intermediate parameters from the cached intermediate parameters and the currently received data packets. For example, in cumulative calculation, the cached intermediate parameters can be added to the data carried in the currently received data packets to obtain the new intermediate parameters.

[0014] In one possible implementation, the next-level device is a storage device with persistent storage capabilities. This storage device can receive and persistently store service data packets sent by intermediate devices; specifically, it persistently stores the service data carried in the data packets. The storage device can also receive and persistently store the service result parameters sent by intermediate devices. This allows for the associated storage of service data and its result parameters, facilitating subsequent use of the result parameters to recover or identify service data.

[0015] In one possible implementation, the upstream device, intermediate device, and downstream device are different computing devices that can operate independently but are connected via a network. The intermediate device could be a switch; alternatively, the upstream and downstream devices could be different computing devices, with the intermediate device being the network interface card (NIC) of the upstream device; or the upstream and downstream devices could be different computing devices, with the intermediate device being the NIC of the downstream device; or the upstream, intermediate, and downstream devices could be different physical components within the same computing device, for example, the upstream device being the NIC, the intermediate device being the processor, and the downstream device being the hard drive.

[0016] In other words, this method can be applied to various types of computing or storage systems, and has high versatility.

[0017] Secondly, a data processing apparatus is provided, configured in an intermediate device, comprising: a receiving unit for receiving the first batch of data packets of a service from a higher-level device; a calculation unit for calculating intermediate parameters related to the service based on the first batch of data packets, caching the intermediate parameters, and sending the first batch of data packets to a lower-level device; a first updating unit for receiving subsequent batches of data packets of the service from the higher-level device, wherein the intermediate device updates the cached intermediate parameters and sends this batch of data packets to the lower-level device each time it receives a batch of data packets; and a second updating unit for receiving the last batch of data packets of the service from the higher-level device, updating the cached intermediate parameters to the result parameters of the service based on the last batch of data packets, and sending the last batch of data packets and the result parameters to the lower-level device.

[0018] In one possible implementation, the apparatus further includes a transmitting unit for not transmitting intermediate parameters to the next-level device.

[0019] In one possible implementation, the intermediate parameter is the verification data calculated by erasure coding (EC) on the data packets of services received by the intermediate device.

[0020] In one possible implementation, the first update unit is used to: calculate the new intermediate parameters based on the currently received batch of data packets and the cached intermediate parameters.

[0021] In one possible implementation, the next-level device is a storage device, which is used to: receive data packets of services sent by intermediate devices and persistently store the data packets of services; and receive result parameters of services sent by intermediate devices and persistently store the result parameters of services.

[0022] In one possible implementation, the upstream device and the downstream device are different computing devices, and the intermediate device is a switch; or, the upstream device and the downstream device are different computing devices, and the intermediate device is the network interface card (NIC) of the upstream device; or, the upstream device and the downstream device are different computing devices, and the intermediate device is the NIC of the downstream device; or, the upstream device, the intermediate device, and the downstream device are different physical components within the same computing device.

[0023] Thirdly, a computing device is provided, comprising: a memory for storing an executable program; and a processor for executing the method provided in the first aspect by running the executable program.

[0024] Fourthly, a computer-readable storage medium is provided, comprising: computer program instructions, which, when executed by a computer device, cause the computer device to perform the method provided in the first aspect.

[0025] Fifthly, a computer program product containing instructions is provided, characterized in that, when the instructions are executed by a computer device, the computer device performs the method provided in the first aspect.

[0026] The beneficial effects of the second to fifth aspects can be referred to the introduction of the beneficial effects of the first aspect above, and will not be repeated here. Attached Figure Description

[0027] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0028] Figure 2 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0029] Figure 3 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0030] Figure 4 is a schematic diagram of a data packet provided in an embodiment of this application;

[0031] Figure 5 is a schematic diagram of an intermediate device provided in an embodiment of this application;

[0032] Figure 6 is a flowchart of a data processing method provided in an embodiment of this application;

[0033] Figure 7 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0034] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0035] The solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings. In the embodiments of this application, "multiple" or "several" refers to two or more. Terms such as "first," "second," etc., are merely used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0036] To facilitate understanding of the solutions provided in the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be introduced first.

[0037] Erasure coding (EC) is a data redundancy technique that encodes original data using an erasure coding algorithm to generate additional redundant data blocks. These redundant data blocks are stored along with the original data. Specifically, the original data can be divided into n original data blocks. An erasure coding algorithm is then used to calculate m redundant data blocks from these n original data blocks. These redundant data blocks, also known as parity blocks, are used to recover the original data in case of loss or corruption. Typically, different data blocks from the n original data blocks and m parity blocks are stored on different storage nodes or disks. When data blocks in m or fewer of these storage nodes or disks are lost, the remaining data blocks can be used to recover the lost data blocks. Common erasure coding algorithms include Galois algorithms and reed-solomon codes (RS codes).

[0038] Remote direct memory access (DMA) is a technique that bypasses the host operating system kernel to access data in its memory (such as RAM). Because it does not involve the operating system, DMA not only saves significant central processing unit (CPU) resources, but also increases system throughput and reduces network communication latency, making it particularly suitable for widespread use in large-scale parallel computer clusters.

[0039] Remote direct memory access (RDMA) is a technique that allows access to data in the memory of a remote host by bypassing the operating system kernel of the remote host.

[0040] Intermediate parameters of a business: also known as intermediate processing results or stage processing results of a business, refer to intermediate or temporary data generated during the business process.

[0041] Business result parameters: also known as business processing results or final results, refer to the result data obtained when the business is completed.

[0042] This application provides a data processing method. An intermediate device can receive service data packets from a higher-level device in batches. A service data packet refers to a data packet carrying service data. There are multiple service data packets, and each batch of service data packets received by the intermediate device may include one or more data packets. Specifically, when the intermediate device receives the first batch of service data packets, it calculates intermediate parameters related to the service based on these data packets and caches the intermediate parameters. Subsequently, whenever the intermediate device receives a service data packet, it updates the cached intermediate parameters based on the currently received data packet. When the currently received data packet is the last batch of service data packets, the intermediate device updates the cached intermediate parameters to the service result parameters based on the currently received data packet and sends the service result parameters to the next-level device. Furthermore, whenever the intermediate device receives a data packet, it sends the data packet to the next-level device instead of storing the data packet in memory. The service here includes a set of data that is sent from the higher-level device of the intermediate device and received by the intermediate device in multiple batches.

[0043] In this method, the intermediate device updates the service's intermediate parameters using cached intermediate parameters and the currently received data packets. When the currently received data packet is the last batch of data packets for the service, the result parameters for the service can be obtained. Thus, the service can be completed without reading all data packets from memory, saving bandwidth overhead. Furthermore, whenever the intermediate device receives a data packet for the service, it sends the packet to the next-level device instead of storing it in memory, thereby saving memory resources.

[0044] Next, the data processing method provided in the embodiments of this application will be illustrated with examples.

[0045] Figure 1 illustrates a system architecture that can be used to implement this method. The system architecture includes a higher-level device 100, an intermediate device 200, and a lower-level device 300. The higher-level device 100, intermediate device 200, and lower-level device 300 are distinguished according to the direction of data packet transmission, and do not necessarily imply a hierarchical or subordinate relationship between them.

[0046] In some embodiments, the upstream device 100 and the downstream device 300 are different computing devices. In one example, as shown in FIG2, the intermediate device 200 can be the network interface card (NIC) of the upstream device 100. In one example, as shown in FIG3, the intermediate device 200 is the NIC of the downstream device 300. In one example, the intermediate device 200 is a switch connecting the upstream device 100 and the downstream device 300.

[0047] In one example of this embodiment, the upstream device 100, also known as a client, can generate business data, encapsulate the business data into data packets, and send out the data packets. In one example, the upstream device 100 can be a terminal device such as a mobile phone, tablet computer, in-vehicle terminal, laptop computer, PDA, or computer. In another example, the upstream device 100 can be a data private client (DPC). In yet another example, the upstream device 100 can be a data processing unit (DPU). The upstream device 100 can carry business data into data packets and send out the business data by sending out the data packets.

[0048] In one example of this embodiment, the upstream device 100 can be a data forwarding device between the data source and the intermediate device 200, such as a base station (BS), access point (AP), or switch. The upstream device 100 can receive service data packets from its upstream device (e.g., the data source) and forward the received data packets to the intermediate device 200.

[0049] In one example of this embodiment, the next-level device 300 is a storage device with persistent storage capabilities, including one or more hard disks, such as solid-state disks (SSDs). The next-level device 300 can store the data in the received data packets into the hard disk to achieve persistent data storage.

[0050] In one example of this embodiment, the next-level device 300 can be a data forwarding device between the intermediate device 200 and the storage device, such as a switch, base station, or access point. The next-level device 300 can send the received data packets to the storage device to persistently store the data in the data packets in the storage device.

[0051] In some embodiments, the upstream device 100, the intermediate device 200, and the downstream device 300 are different physical components within the same computing device. For example, the upstream device 100 is the network interface card (NIC) of the computing device, the intermediate device 200 is the CPU of the computing device, and the downstream device 300 is the hard disk of the computing device.

[0052] Intermediate device 200 can receive multiple data packets from upstream device 100 in batches, and these multiple data packets belong to the same service. Specifically, the total amount of data required by the service may exceed the maximum transmission unit (MTU) of the data packets. Therefore, it is necessary to split all the data required by the service into multiple data packets, and carry different data within these multiple data packets into different data packets. In one example, the data packets could be RDMA (RDMA over converged Ethernet, RoCE) protocol data packets. In another example, the data packets could be network attached storage (NAS) protocol data packets.

[0053] Figure 4 illustrates a data packet. The payload of this data packet is the data carried by the packet. The packet header carries control information for the packet, such as the packet type, source address, destination address, and protocol version. In some embodiments, a proprietary field is added to the packet header. This proprietary field, also called a proprietary header, is used to carry service control information.

[0054] In one example, business control information includes computation block identifiers, indexes, computation types, and window properties. A computation block refers to all the data required by the business. The computation block identifier of a data packet indicates the computation block to which the data carried by that data packet belongs. The index of a data packet is its sequence number. Multiple data packets are sorted according to the order of the data carried by each packet within the total data required by the business, forming a queue. The sequence number of the data packet indicates its position in the queue. The computation type, also known as the business type, refers to the specific business to be performed. Business types can include erasure coding calculation, fingerprint information calculation, XOR calculation, accumulation calculation, cyclic redundancy check, data compression, or data decompression, etc. Window properties include window size, etc. The window size refers to the granularity of the calculation or the granularity of the business processing. The window size of a data packet can specifically be the amount of business data carried by the data packet, instructing the intermediate device 200 to perform calculations at the data packet level.

[0055] The intermediate device 200 is used to execute services, that is, to obtain or update intermediate parameters or result parameters of the services based on the data packets received.

[0056] In some embodiments, as shown in FIG5, the intermediate device 200 includes a data packet parsing module 210, a data processing module 220, a caching module 230, and an output module 240. The data packet parsing module 210 parses the data packets received by the intermediate device 200 to obtain the data within the data packets. The data processing module 220 obtains or updates intermediate or result parameters of the service based on the data in the data packets. The caching module 230 caches the intermediate parameters of the service. When the result parameters of the service are obtained, the output module 240 outputs the result parameters of the service. The output module 240 can send the result parameters of the service to the next-level device 300 for persistent storage in the next-level device 300. Additionally, whenever the intermediate device 200 receives a data packet, the output module 240 sends the data packet to the next-level device 300 for persistent storage in the next-level device 300. Specifically, the output module 240 sends the service data in the data packet to the next-level device 300 for persistent storage in the next-level device 300.

[0057] The foregoing example illustrates a system architecture provided by an embodiment of this application. Next, with reference to the accompanying drawings, the flow of the data processing method provided by this application embodiment will be illustrated using an implementation within this system architecture as an example.

[0058] As shown in Figure 6, in step 601, the intermediate device 300 receives the first batch of service data packets from the upstream device 100. The first batch of data packets is a subset of multiple service data packets, specifically the data packet that arrives at the intermediate device 200 first among these multiple data packets. The first batch of data packets can be a single data packet or several data packets.

[0059] Next, in step 602, the intermediate device 200 calculates and caches intermediate parameters related to the service based on the first batch of data packets. These intermediate parameters, also called intermediate results or service state, refer to the results of calculations performed by the intermediate device 200 on the received service data packets. Specifically, the intermediate device 200 performs calculations on the service data carried in the data packets to obtain the service-related intermediate parameters.

[0060] In some embodiments, the service can be calculated using erasure coding. In this case, the intermediate parameter is the verification data obtained by the intermediate device 200 through erasure coding calculation on the received service data packets. In other words, the intermediate parameter is calculated by the intermediate device 200 according to the erasure coding calculation logic on the received service data packets. The verification data is also called a verification data block.

[0061] In some embodiments, the service may be fingerprint information calculation. That is, the service is to calculate fingerprint information from the data in the multiple data packets. In one example of this embodiment, the fingerprint information may specifically be a fingerprint image (FPI).

[0062] In some embodiments, the service may be an exclusive OR (XOR) calculation. That is, the service performs an XOR calculation on the data in the multiple data packets.

[0063] In some embodiments, the service may be an additive calculation. That is, the service involves adding up the data in multiple data packets. In some embodiments, the service is a cyclic redundancy check (CRC). That is, the service performs a CRC check on the data in multiple data packets.

[0064] In some embodiments, the service can be data compression. That is, the service is to compress data in multiple data packets.

[0065] In some embodiments, the service can be data decompression. That is, the service decompresses data from multiple data packets.

[0066] In other embodiments, the service can also be other types of services. These will not be listed here.

[0067] In some embodiments, in step 602, the intermediate device 200 calculates the first batch of data packets based on the service processing logic to obtain service-related intermediate parameters. Taking erasure coding calculation as an example, the service processing logic may include: dividing the original data into n original data blocks, and using the erasure coding algorithm to calculate the n original data blocks to obtain m check data blocks, which are the service-related intermediate parameters. n and m are both integers greater than or equal to 1.

[0068] In one example, the erasure coding algorithm can be set to Solomon coding. Solomon coding calculates the erasure code using formula (1).

[0069] Where b represents the calculation parameters of the check data block, d represents the original data block, d1 represents the first check data block among n original data blocks, and d n c represents the nth check data block out of n original data blocks, c1 represents the first check data block out of m check data blocks, and c represents the first check data block out of m original data blocks. m This represents the m-th check data block out of m check data blocks.

[0070] The data carried by the first batch of data packets can be called data D. 1 Data D can be used. 1 Divide into n original data blocks d 1 It can be based on n original data blocks d 1 m verification data blocks c are obtained. 1 Referring to formula (1), we obtain the formula for using n original data blocks d. 1 m verification data blocks c are obtained. 1 Formula (2). In step 602, calculations can be performed according to formula (2). When using formula (2) for calculation, the original data block d can be padded with zeros as needed. 1 .

[0071] in, Represents n original data blocks d 1 The first check data block in the data. Represents n original data blocks d 1 The nth check data block in c represents m check data blocks 1 The first check data block in the data. c represents m check data blocks 1 The m-th check data block in the data.

[0072] m verification data blocks c are obtained through formula (2). 1 These constitute the intermediate parameters.

[0073] The above example, using erasure coding calculation, illustrates the calculation process of intermediate parameters. Similarly, for other types of services, intermediate parameters can be calculated in step 602. Taking the cumulative calculation service as another example, when the first batch of data packets contains only one data packet, the data in that data packet is used as the intermediate parameter of the service. If the first batch of data packets includes several data packets, the data in each of those several data packets is multiplied together to obtain the intermediate parameter of the service.

[0074] The intermediate parameters related to the service can be obtained through the above method. The intermediate device 200 can cache these intermediate parameters, for example, by storing them in the data exchange buffer (cache) of the processor of the intermediate device 200, or in the memory of the intermediate device 200. In some embodiments, the intermediate device 200 does not send the intermediate parameters of the service to the next-level device 300 to save communication resources.

[0075] Additionally, in step 603, the intermediate device 200 sends the first batch of data packets to the next-level device 300. That is, whenever the intermediate device 200 receives a service data packet, it immediately sends the data packet to the next-level device 200, without storing the data packet in memory. Steps 603 and 602 can be executed simultaneously, or step 603 can be executed after step 602, or step 603 can be executed before step 602.

[0076] Then, in step 604, the intermediate device 200 can receive subsequent batches of data packets for the service from the upper-level device 100. Each batch of data packets includes one or more data packets. Upon receiving a batch of data packets, in step 605, the intermediate device 200 updates the cached intermediate parameters based on the currently received batch of data packets. Specifically, it calculates new intermediate parameters based on the currently received batch of data packets and the cached intermediate parameters, and caches the new intermediate parameters.

[0077] Taking erasure coding calculation as an example, the business processing logic may include: dividing the original data into n original data blocks; using the erasure coding algorithm, calculating on the n original data blocks to obtain m check data blocks; adding the check data blocks from these m check data blocks to the corresponding check data blocks from the previously obtained m check data blocks (i.e., cached intermediate parameters) to obtain m new check data blocks, which are the updated intermediate parameters. n and m are both integers greater than or equal to 1.

[0078] For example, the data carried in the currently received batch of data packets can be called data D. 2 Data D can be used. 2 Divide into n original data blocks d 2 It can be based on n original data blocks d 2 m verification data blocks c are obtained. 2 Referring to formula (1), we obtain the formula for using n original data blocks d. 2 m verification data blocks c are obtained. 2 Formula (3). In step 605, calculations can be performed according to formula (3). When using formula (3) for calculation, the original data block d can be padded with zeros as needed. 2 .

[0079] in, Represents n original data blocks d 2 The first check data block in the data. Represents n original data blocks d 2 The nth check data block in c represents m check data blocks 2 The first check data block in the data. c represents m check data blocks 2 The m-th check data block in the data.

[0080] m verification data blocks c are obtained through formula (3). 2 .

[0081] The intermediate parameter for caching can be set to m checksum blocks c. x m verification data blocks c 2 and m check data blocks c x It is a one-to-one correspondence. Specifically, there are m verification data blocks c. x Includes verification data blocks …, Verification data block Among them, the verification data block and m check data blocks c x In Corresponding to, ..., verification data blocks and m check data blocks c 2 In Corresponding. That is, m verification data blocks c x and m check data blocks c 2 Check data blocks with the same subscript are corresponding. In step 605, the m check data blocks c x and m check data blocks c 2 The corresponding check data blocks are added together to obtain m check data blocks c. x+1 Among them, there are m verification data blocks c x+1 This forms the updated intermediate parameters.

[0082] Additionally, whenever intermediate device 200 receives a data packet from a subsequent batch of services, it sends the data packet to the next-level device 200 via step 606, without storing the data packet in memory. Steps 606 and 605 can be executed simultaneously, or step 606 can be executed after step 605, or step 606 can be executed before step 605.

[0083] Iteratively execute steps 604-606, continuously updating the cached intermediate parameters of the business logic.

[0084] Referring again to Figure 6, in step 607, intermediate device 200 receives the last batch of data packets for the service from the previous level device 100. The last batch of data packets may include one or more data packets.

[0085] Intermediate device 200 can determine whether the currently received batch of data packets is the last batch. Specifically, if the currently received batch of data packets includes the last data packet, it can be confirmed that the currently received batch of data packets is the last batch. This last data packet refers to the last data packet received by intermediate device 200 among the aforementioned multiple data packets. In other words, among the multiple data packets of the aforementioned service, this last data packet is the last one received by intermediate device 200.

[0086] In some embodiments, the upstream device 100 may notify the intermediate device 200 of the number of data packets in the multiple data packets of the aforementioned service. For example, the multiple data packets may be K data packets, where K is an integer greater than 1. When a data packet is the Kth data packet received by the intermediate device 200, it can be confirmed that the data packet is the last data packet. Otherwise, it is confirmed that the data packet is not the last data packet.

[0087] When the intermediate device 200 receives the last batch of data packets, it can execute step 608 to update the cached intermediate parameters to the service result parameters based on the last batch of data packets. Specifically, the service result parameters can be calculated based on the last batch of data packets and the cached intermediate parameters according to the service processing logic.

[0088] In some embodiments, taking erasure coding computation as an example, the data carried by the last batch of data packets can be referred to as data D. 3 Data D can be used. 3 Divide into n original data blocks d 3 It can be based on n original data blocks d 3 m verification data blocks c are obtained. 3 Referring to formula (1), we obtain the formula for using n original data blocks d. 3 m verification data blocks c are obtained. 3 Formula (4) is used, and calculations are performed according to Formula (4).

[0089] in, Represents n original data blocks d 3 The first check data block in the data. Represents n original data blocks d 3 The nth check data block in c represents m check data blocks 3The first check data block in the data. c represents m check data blocks 3 The m-th check data block in the data.

[0090] In addition, when using formula (4) for calculation, the original data block d can be padded with zeros as needed. 3 .

[0091] The intermediate parameter for caching can be set to m checksum blocks c. k-1 Among them, there are m verification data blocks c k-1 and m check data blocks c 3 The corresponding check data blocks in the [database name]. Transfer m check data blocks c [to the data base name]. k-1 and m check data blocks c 3 The corresponding check data blocks are added together to obtain m check data blocks c. k Among them, there are m verification data blocks c k The result parameters that make up the business.

[0092] After obtaining the result parameters of the service, the intermediate device 200 can execute step 610 to send the result parameters of the service to the next-level device 300.

[0093] Additionally, when intermediate device 200 receives the last batch of data packets for a service, intermediate device 200 can send the last batch of data packets to the next-level device 200 through step 609, without storing the last batch of data packets in memory. Steps 609 and 608 can be executed simultaneously, or step 609 can be executed after step 608, or step 609 can be executed before step 608.

[0094] In some embodiments, the next-level device 300 is a storage device. This storage device can receive data packets of a service sent by the intermediate device 300 and persistently store the received data packets. This storage device can also receive result parameters of a service sent by the intermediate device 200 and persistently store the result parameters of the service.

[0095] In some embodiments, the next-level device 300 may be a data forwarding device between the intermediate device 200 and the storage device. This data forwarding device can receive data packets of services sent by the intermediate device 300 and forward the received data packets to the storage device. The data forwarding device can also receive result parameters of services sent by the intermediate device 200 and forward the received result parameters to the storage device.

[0096] In summary, the data processing method provided in this application's embodiments executes business logic and obtains business result parameters without needing to read the required data from memory, thereby saving bandwidth. Furthermore, this method eliminates the need to store data packets in memory, thus saving memory resources.

[0097] This application provides a data processing apparatus 700. The apparatus 700 is configured in an intermediate device, such as the intermediate device 700 described above. As shown in FIG7, the apparatus 700 includes:

[0098] The receiving unit 710 is used to receive the first batch of data packets of the service from the previous level device;

[0099] The calculation unit 720 is used to calculate intermediate parameters related to the service based on the first batch of data packets, cache the intermediate parameters, and send the first batch of data packets to the next level device.

[0100] The first update unit 730 is used to receive subsequent batches of data packets of the service from the upper-level device, wherein each time the intermediate device receives a batch of data packets, it updates the cached intermediate parameters and sends the batch of data packets to the next-level device.

[0101] The second update unit 740 is used to receive the last batch of data packets of the service from the previous level device, update the cached intermediate parameters to the result parameters of the service according to the last batch of data packets, and send the last batch of data packets and the result parameters to the next level device.

[0102] In some embodiments, the apparatus 700 further includes a transmitting unit 750, configured not to transmit the intermediate parameters to the next-level device.

[0103] In some embodiments, the intermediate parameter is verification data calculated by performing erasure coding (EC) on the data packets of the service received by the intermediate device.

[0104] In some embodiments, the first update unit 730 is used to: calculate the currently received batch of data packets and the cached intermediate parameters to obtain new intermediate parameters.

[0105] In some embodiments, the next-level device is a storage device, wherein the storage device is configured to: receive data packets of the service sent by the intermediate device and persistently store the data packets of the service; receive result parameters of the service sent by the intermediate device and persistently store the result parameters of the service.

[0106] In some embodiments, the upstream device and the downstream device are different computing devices, and the intermediate device is a switch; or, the upstream device and the downstream device are different computing devices, and the intermediate device is the network interface card (NIC) of the upstream device; or, the upstream device and the downstream device are different computing devices, and the intermediate device is the NIC of the downstream device; or, the upstream device, the intermediate device, and the downstream device are different physical components within the same computing device.

[0107] The functions of each functional unit of device 700 can be implemented with reference to the above description of the operations performed by the intermediate device, for example, the operations performed by the intermediate device 200 in the embodiment shown in Figure 6.

[0108] This application provides a computing device 800. As shown in FIG8, the computing device 800 includes a processor 810 and a memory 820. The memory 820 is used to store executable programs. The processor 810 is used to execute the executable programs stored in the memory 820, so that the computing device 800 can perform the operations performed by the intermediate device described above, for example, the operations performed by the intermediate device 200 in the embodiment shown in FIG6.

[0109] For example, processor 810 may include any one or more processors such as CPU, graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0110] For example, memory 820 may include volatile memory, such as random access memory (RAM). Memory 820 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive.

[0111] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computer device or stored on any usable medium. When the computer program product runs on a computer device, it causes the computer device to perform the operations performed by the intermediate device described above, for example, the operations performed by the intermediate device 200 in the embodiment shown in FIG6.

[0112] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computer device to perform the operations performed by the intermediate device described above, for example, the operations performed by the intermediate device 200 in the embodiment shown in FIG. 6.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: The intermediate device receives the first batch of data packets of the service from the previous level device; The intermediate device calculates intermediate parameters related to the service based on the first batch of data packets, caches the intermediate parameters, and sends the first batch of data packets to the next level device. The intermediate device receives subsequent batches of data packets for the service from the upper-level device. Each time the intermediate device receives a batch of data packets, it updates the cached intermediate parameters and sends this batch of data packets to the next-level device. The intermediate device receives the last batch of data packets for the service from the previous level device, updates the cached intermediate parameters to the result parameters of the service based on the last batch of data packets, and sends the last batch of data packets and the result parameters to the next level device.

2. The method according to claim 1, characterized in that, The method further includes: the intermediate device does not send the intermediate parameters to the next-level device.

3. The method according to claim 1 or 2, characterized in that, The intermediate parameter is verification data calculated by performing erasure coding (EC) on the data packets of the service received by the intermediate device.

4. The method according to any one of claims 1-3, characterized in that, The intermediate parameters for updating the cache specifically include: calculating new intermediate parameters based on the currently received batch of data packets and the cached intermediate parameters.

5. The method according to any one of claims 1-4, characterized in that, The next-level device is a storage device, and the method further includes: The storage device receives the data packets of the service sent by the intermediate device and persistently stores the data packets of the service. The storage device receives the result parameters of the service sent by the intermediate device and persistently stores the result parameters of the service.

6. The method according to any one of claims 1-5, characterized in that, The upstream device and the downstream device are different computing devices, and the intermediate device is a switch; Alternatively, the upstream device and the downstream device are different computing devices, and the intermediate device is the network card of the upstream device; Alternatively, the upstream device and the downstream device are different computing devices, and the intermediate device is the network card of the downstream device; Alternatively, the upstream device, the intermediate device, and the downstream device may be different physical components within the same computing device.

7. A data processing apparatus, characterized in that, The device is configured in an intermediate device, and the device includes: The receiving unit is used to receive the first batch of data packets of the service from the previous level device; The calculation unit is used to calculate intermediate parameters related to the service based on the first batch of data packets, cache the intermediate parameters, and send the first batch of data packets to the next level device. The first update unit is used to receive subsequent batches of data packets of the service from the upper-level device, wherein each time the intermediate device receives a batch of data packets, it updates the cached intermediate parameters and sends the batch of data packets to the next-level device. The second update unit is used to receive the last batch of data packets of the service from the previous level device, update the cached intermediate parameters to the result parameters of the service according to the last batch of data packets, and send the last batch of data packets and the result parameters to the next level device.

8. The apparatus according to claim 7, characterized in that, The device further includes a transmitting unit, configured not to transmit the intermediate parameters to the next-level device.

9. The apparatus according to claim 7 or 8, characterized in that, The intermediate parameter is verification data calculated by performing erasure coding (EC) on the data packets of the service received by the intermediate device.

10. The apparatus according to any one of claims 7-9, characterized in that, The first update unit is used to: calculate the new intermediate parameters based on the currently received batch of data packets and the cached intermediate parameters.

11. The apparatus according to any one of claims 7-10, characterized in that, The next-level device is a storage device, wherein the storage device is used for: Receive the data packets of the service sent by the intermediate device, and persistently store the data packets of the service; Upon receiving the result parameters of the service sent by the intermediate device, the result parameters of the service are persistently stored.

12. The apparatus according to any one of claims 7-11, characterized in that, The upstream device and the downstream device are different computing devices, and the intermediate device is a switch; Alternatively, the upstream device and the downstream device are different computing devices, and the intermediate device is the network card of the upstream device; Alternatively, the upstream device and the downstream device are different computing devices, and the intermediate device is the network card of the downstream device; Alternatively, the upstream device, the intermediate device, and the downstream device may be different physical components within the same computing device.

13. A computing device, characterized in that, include: Memory, used to store executable programs; A processor for performing the method as described in any one of claims 1-6 by running the executable program.

14. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computer device, perform the method as described in any one of claims 1-6.

15. A computer program product containing instructions, characterized in that, When the instructions are executed by a computer device, the computer device performs the method as described in any one of claims 1-6.

Images