Data processing methods, apparatuses, electronic devices, storage media, and computer program products

Abstract

The application provides a data processing method and device, electronic equipment, storage medium and computer program product, relates to the technical field of data transmission technology, and the method comprises the following steps: obtaining a plurality of container packages sent by a source end; wherein each container package comprises a plurality of data packages, and each data package comprises a container identifier of the container package and a corresponding data sequence identifier; sorting the plurality of container packages based on the container identifiers corresponding to the plurality of container packages and the continuity of the data sequence identifiers corresponding to the data packages in the plurality of container packages to obtain target data. The application embodiment can improve the processing efficiency of data.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] Unlike traditional data centers, intelligent computing centers primarily support artificial intelligence (AI) model training, characterized by a small number of flows and high bandwidth per flow. To avoid link congestion, packet-by-packet load balancing is more suitable for intelligent computing center traffic load balancing than flow-by-flow load balancing. When traffic passes through the source side, it is segmented, and the segmented packets are transmitted in the network packet by packet. Therefore, ensuring packet order at the destination is the final crucial step in traffic transmission. Currently, packet ordering is achieved by receiving packets sequentially at the destination. Only when a packet times out is a retransmission of the timed-out packet and subsequent packets requested. This introduces significant latency into data sorting at the destination, resulting in low data processing efficiency. Summary of the Invention

[0003] The data processing method, apparatus, electronic device, storage medium, and computer program product provided in this application embodiment can improve data processing efficiency.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a data processing method, including:

[0006] Obtain multiple container packets sent by the source: wherein each container packet includes multiple data packets, and each data packet includes a container identifier belonging to the container packet and a corresponding data sequence identifier;

[0007] Based on the container identifiers corresponding to the multiple container packets respectively, and the continuity of the data sequence identifiers corresponding to the data packets within the multiple container packets, the multiple container packets are sorted to obtain the target data.

[0008] In the above scheme, the step of sorting the multiple container packets to obtain the target data based on the continuity of the container identifiers corresponding to the multiple container packets and the data sequence identifiers corresponding to the data packets within the multiple container packets includes:

[0009] The continuity of multiple data sequence identifiers within multiple containers is detected to obtain a detection result;

[0010] If the detection result indicates that multiple data sequence identifiers within multiple container packages are consecutive and without missing values, then the multiple container packages are sorted according to the order in which the multiple container identifiers are represented to obtain the target data.

[0011] In the above scheme, after detecting the continuity of multiple data sequence identifiers within multiple containers and obtaining the detection result, the method further includes:

[0012] If the detection result indicates that multiple first data sequence identifiers within the first container are missing, a first lookup request is sent to the source based on the multiple first data sequence identifiers and their corresponding communication attribute information; wherein, the communication attribute information is used to characterize the communication address and memory-related information of the data packets within the multiple container packets;

[0013] The system receives the second container package as a response to the first lookup request from the source end, and uses the second container package to replace the first container package to update multiple container packages.

[0014] The target data is obtained by sorting the updated container packages according to the order of the container identifier representations.

[0015] In the above scheme, after detecting the continuity of multiple data sequence identifiers within multiple containers and obtaining the detection result, the method further includes:

[0016] If the detection result indicates that multiple second data sequence identifiers are missing from the third container, then multiple second data sequence identifiers are determined based on the data sequence identifiers in adjacent container packages; wherein, the adjacent container packages are the container packages corresponding to two container identifiers that are adjacent to the container identifier corresponding to the third container package;

[0017] Based on multiple second data sequence identifiers and communication attribute information, a second lookup request is sent to the source end;

[0018] The system receives the fourth container package as a response to the second lookup request from the source end, and uses the fourth container package to replace the third container package to update multiple container packages.

[0019] The target data is obtained by sorting the updated container packages according to the order of the container identifier representations.

[0020] In the above scheme, obtaining the multiple container packets sent by the source end includes:

[0021] Obtain the multiple initial container packets sent by the source end;

[0022] Among the multiple initial container packages, multiple container packages that include the same communication attribute information are identified.

[0023] In the above scheme, after sorting the multiple container packets to obtain the target data based on the continuity of the container identifiers corresponding to the multiple container packets and the data sequence identifiers corresponding to the data packets within the multiple container packets, the method further includes:

[0024] The communication attribute information is stored, and a task completion notification is sent back to the source.

[0025] This application also provides a data processing method, including:

[0026] The acquired data to be processed is segmented to obtain N data packets in a certain order; N is an integer greater than 1.

[0027] The N data packets are encapsulated into multiple container packets in sequence and sent to the destination; wherein each container packet includes multiple data packets, and each data packet includes a container identifier of the container packet to which it belongs and a corresponding data sequence identifier; the data sequence identifiers corresponding to the data packets in the multiple container packets are continuous.

[0028] In the above scheme, the N data packets are encapsulated into multiple container packets in sequence, including:

[0029] The data sequence identifier and communication attribute information are added sequentially to the extension header of each data packet; wherein the data sequence identifiers corresponding to the N data packets are consecutive.

[0030] The container identifier is added sequentially to the extended header of every M data packets; where M is an integer greater than 0 and less than N.

[0031] Each M data packets is encapsulated into a container packet until all N data packets are encapsulated, resulting in multiple container packets.

[0032] The method in the above scheme further includes:

[0033] Construct a correspondence between the original identifier of each data packet and the corresponding data sequence identifier and communication attribute information;

[0034] The correspondence is stored in a correspondence set; wherein the correspondence set includes the correspondence between the original identifier of different data packets and the corresponding data sequence identifier and communication attribute information.

[0035] In the above scheme, after encapsulating the N data packets into multiple container packets in sequence and sending them to the destination, the method further includes:

[0036] Receive a first lookup request sent by the destination terminal; wherein the first lookup request includes: multiple first data sequence identifiers and communication attribute information;

[0037] Based on the communication attribute information, a third data sequence identifier that is missing in multiple first data sequence identifiers is searched in the correspondence set;

[0038] After supplementing the multiple first data sequence identifiers with the third data sequence identifier, the multiple first original identifiers corresponding to the multiple first data sequence identifiers are searched in the correspondence set;

[0039] The data packets corresponding to the multiple first original identifiers are packaged into a second container packet in order and sent to the destination.

[0040] In the above scheme, after encapsulating the N data packets into multiple container packets in sequence and sending them to the destination, the method further includes:

[0041] Receive a second lookup request sent by the destination terminal; wherein the second lookup request includes: multiple second data sequence identifiers and communication attribute information;

[0042] Based on the communication attribute information, multiple second original identifiers corresponding to multiple second data sequence identifiers are searched in the correspondence set;

[0043] The data packets corresponding to the multiple second original identifiers are packaged into a fourth container packet in sequence and sent to the destination.

[0044] In the above scheme, before the step of segmenting the acquired data to be processed into N data packets of a certain order, the following steps are included:

[0045] The data to be processed, including the predetermined message header, is determined from multiple raw data sets.

[0046] This application also provides a data processing apparatus, including:

[0047] A data acquisition unit is used to acquire multiple container packets sent by the source end: wherein each container packet includes multiple data packets, and each data packet includes a container identifier belonging to the container packet and a corresponding data sequence identifier;

[0048] A data processing unit is used to sort multiple container packets to obtain target data based on the container identifiers corresponding to the multiple container packets respectively and the continuity of the data sequence identifiers corresponding to the data packets within the multiple container packets.

[0049] This application also provides a data processing apparatus, including:

[0050] The data segmentation unit is used to segment the acquired data to be processed into N data packets in a certain order; N is an integer greater than 1.

[0051] An encapsulation and transmission unit is used to encapsulate N data packets into multiple container packets in sequence and send them to the destination; wherein each container packet includes multiple data packets, and each data packet includes a container identifier of the container packet to which it belongs and a corresponding data sequence identifier; the data sequence identifiers corresponding to the data packets in the multiple container packets are continuous.

[0052] This application also provides an electronic device, including a first memory and a first processor. The first memory stores a computer program that can run on the first processor. When the first processor executes the computer program, it implements the steps in the method on the destination side.

[0053] This application also provides an electronic device, including a second memory and a second processor. The second memory stores a computer program that can run on the second processor. When the second processor executes the computer program, it implements the steps in the method on the source side.

[0054] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a first processor, implements the steps of the method on the destination side.

[0055] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a second processor, implements the steps of the method on the source side.

[0056] This application also provides a computer program product, including a computer program that, when executed by a first processor, implements the steps of the method on the destination side.

[0057] This application also provides a computer program product, including a computer program that, when executed by a second processor, implements the steps of the method on the source side.

[0058] In this embodiment, multiple container packets sent by the source are obtained. Each container packet includes multiple data packets, and each data packet includes a container identifier of its respective container packet and a corresponding data sequence identifier. Based on the continuity of the container identifiers corresponding to the multiple container packets and the data sequence identifiers corresponding to the data packets within the multiple container packets, the multiple container packets are sorted to obtain the target data. Thus, during the sorting of multiple container packets at the destination, the continuity of the corresponding data sequence identifiers can quickly determine whether there are any lost data packets. Compared to related technologies that rely on data timeouts to detect lost data packets, this method can quickly detect lost data packets, reduce latency, and improve data processing efficiency. Furthermore, this application relies on the container identifiers of each container packet for sorting, and each container packet includes multiple data packets. This sorting method can also save computing resources to some extent and improve data processing efficiency when dealing with large amounts of data. Attached Figure Description

[0059] Figure 1 An optional effect diagram of the related technology provided in the embodiments of this application;

[0060] Figure 2 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0061] Figure 3 An optional effect diagram of the data processing method provided in the embodiments of this application;

[0062] Figure 4 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0063] Figure 5 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0064] Figure 6 An optional effect diagram of the data processing method provided in the embodiments of this application;

[0065] Figure 7 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0066] Figure 8 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0067] Figure 9 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0068] Figure 10An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0069] Figure 11 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0070] Figure 12 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0071] Figure 13 An optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0072] Figure 14 An interactive schematic diagram of the data processing method provided in the embodiments of this application;

[0073] Figure 15 Schematic diagram of the data processing apparatus provided in the embodiments of this application Figure 1 ;

[0074] Figure 16 A hardware entity illustration of an electronic device provided in the embodiments of this application. Figure 1 ;

[0075] Figure 17 Schematic diagram of the data processing apparatus provided in the embodiments of this application Figure 2 ;

[0076] Figure 18 A hardware entity illustration of an electronic device provided in the embodiments of this application. Figure 2 . Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0078] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0079] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0081] Unlike traditional data centers, intelligent computing centers primarily support AI model training, and their communication traffic is characterized by a small number of streams but high bandwidth per stream. To avoid link congestion, packet-by-packet load balancing is more suitable for load balancing in intelligent computing centers than stream-by-stream load balancing. Figure 1 When the source sends multiple communication traffic to the destination through network device 4, the communication traffic sent by the source has the characteristic of large single-stream bandwidth. Therefore, when the processing capacity of network device 4 is limited, congestion is likely to occur at network device 4.

[0082] Out-of-order issues in current intelligent computing center networks are one of the key problems affecting the training efficiency of AI clusters. Although current network interface cards (NICs) support out-of-order reordering, existing solutions are insufficient to meet the out-of-order reordering requirements of intelligent computing centers based on packet-by-packet load balancing. Current solutions mainly suffer from the following problems:

[0083] Option 1: The destination hardware supports a Go-Back-N mechanism. The destination hardware receives packets sequentially, and if a packet times out, it retransmits the message information and subsequent packets corresponding to the processor sequence number (Personal Subscription Number, PSN). Therefore, this retransmission scheme ensures that the card inherently supports out-of-order reordering. While the Go-Back-N mechanism guarantees out-of-order reordering at the destination, it requires ensuring the sequential reception of packets during the sorting process. If a packet is missed, it will be retransmitted along with all subsequent packets. Therefore, this out-of-order reordering mechanism introduces significant latency to AI cluster training, impacting training effectiveness.

[0084] Option 2: The first packet received at the destination must be the first packet of the traffic, i.e., including the First RETH. If the first received packet is not the first packet, the sorting fails. When the first received packet is the first packet, subsequent received data packets will be ordered according to the message size and PSN offset in the RETH information. If a data packet with the corresponding PSN is not received after a timeout during reception, the data packet with the lost PSN will be retransmitted. Although this option can sort all data packets, it still has many problems. First, the network card needs to cache the data packet information for each QP and sort each data packet, which will put some pressure on the network card's cache. Second, traffic sorting cannot distinguish between Remote Direct Memory Access (RDMA) traffic and traditional Transmission Control Protocol (TCP) / User Datagram Protocol (UDP) traffic. Therefore, each type of traffic needs to be segmented and reordered, which puts a lot of pressure on the end-side reordering of large amounts of communication data in large model training scenarios.

[0085] This application provides a data processing method. Please refer to the embodiments provided. Figure 2 This is an optional flowchart illustrating a data processing method provided in an embodiment of this application, which will be combined with... Figure 2 The steps shown are explained.

[0086] S101. Obtain multiple container packets sent by the source: wherein each container packet includes multiple data packets, and each data packet includes a container identifier belonging to the container packet and a corresponding data sequence identifier.

[0087] In this embodiment, the source sends multiple container packets to the destination via a network device. Each container packet includes a data packet, and each data packet includes a container identifier of the container to which the data packet belongs, and a corresponding data sequence identifier.

[0088] In this embodiment, the source end can add a continuous data sequence identifier and a corresponding container identifier to the extended header of multiple data packets in the order of the data packets. Data packets belonging to the same container have the same container identifier. Data packets belonging to the same container are encapsulated into corresponding containers to form multiple container packets, and these multiple container packets are sent to the destination end.

[0089] The data sequence identifier can be a PSN number, and the extended header of the data packet can also include QP information and flags, etc. An exemplary combination... Figure 3This is a schematic diagram illustrating the structural composition of data packets 1 and 2. The extended header of data packet 1 may include: container identifier, quality procedure (QP) file, flag, and PSN number.

[0090] S102. Based on the container identifiers corresponding to the multiple container packets respectively, and the continuity of the data sequence identifiers corresponding to the data packets within the multiple container packets, the multiple container packets are sorted to obtain the target data.

[0091] In this embodiment, the destination end can arrange the multiple data sequence identifiers within each container packet in a sequential manner according to the order represented by the multiple container identifiers. If there are no missing data sequence identifiers in the arranged series of data sequence identifiers, it is determined that there are no data packets lost in the multiple container packets, and the multiple container packets are sorted to obtain the target data. If there are missing data sequence identifiers in the arranged series of data sequence identifiers, the destination end requests the missing data sequence identifiers from the source end to update the multiple container packets, and then sorts the updated multiple container packets to obtain the target data.

[0092] In some other embodiments, the destination end can also sort multiple container identifiers according to the order in which they represent. If no container identifier is missing in the sorted series of container identifiers, it checks whether there are any missing data sequence identifiers. Based on the detection result, it updates the corresponding data packets or sorts the multiple container packets. If a container identifier is missing in the sorted series of container identifiers, it requests multiple data packets corresponding to the missing container identifier from the source end to update the multiple container packets. The target data is then obtained by sorting the updated multiple container packets.

[0093] This application embodiment proposes adding Container ID, PSN, QP, and other information to the GSE extension header of the data packet, according to...<Container ID,PSN> This method determines whether packet loss has occurred during transmission without waiting for a timer to expire, alleviating the pressure on the network interface card (NIC) to reorder packets and improving the data processing efficiency at the destination. A packet container mechanism is introduced at the source end to ensure that packets within the container received at the destination are not out of order.

[0094] In this embodiment, multiple container packets sent by the source are obtained. Each container packet includes multiple data packets, and each data packet includes a container identifier of its respective container packet and a corresponding data sequence identifier. Based on the continuity of the container identifiers corresponding to the multiple container packets and the data sequence identifiers corresponding to the data packets within the multiple container packets, the multiple container packets are sorted to obtain the target data. Thus, during the sorting of multiple container packets at the destination, the continuity of the corresponding data sequence identifiers can quickly determine whether there are any lost data packets. Compared to related technologies that rely on data timeouts to detect lost data packets, this method can quickly detect lost data packets, reduce latency, and improve data processing efficiency. Furthermore, this application relies on the container identifiers of each container packet for sorting, and each container packet includes multiple data packets. This sorting method can also save computing resources to some extent and improve data processing efficiency when dealing with large amounts of data.

[0095] This application provides a data processing method. Please refer to the embodiments provided. Figure 4 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 2 S102 shown can also be implemented by S201 to S202, which will be explained in conjunction with the steps.

[0096] S201. Detect the continuity of multiple data sequence identifiers within multiple container packages to obtain a detection result.

[0097] In this embodiment of the application, the target end can sort the multiple data sequence identifiers included in the multiple containers based on the continuity of the multiple data sequence identifiers included in the multiple containers respectively, to obtain an identifier sequence, detect whether there are any missing data sequence identifiers in the identifier sequence, and obtain the detection result.

[0098] In this embodiment of the application, the destination end can sort the multiple data sequence identifiers in each container according to the order represented by the multiple container identifiers based on the corresponding continuity, to obtain an identifier sequence of a data sequence identifier, detect whether there is a missing data sequence identifier in the identifier sequence, and obtain the detection result.

[0099] The detection results are used to characterize the absence of one or more data sequence identifiers within the container.

[0100] S202. If the detection result indicates that the multiple data sequence identifiers within the multiple container packages are continuous and without missing values, then the multiple container packages are sorted according to the order in which the multiple container identifiers are represented to obtain the target data.

[0101] In this embodiment, if the detection results indicate that multiple data sequence identifiers within each container of multiple container packages are continuous and without missing values, then the multiple container packages are sorted according to the order in which the multiple container identifiers are represented to obtain the target data. After determining the target data, an AI model can be trained based on the target data.

[0102] In this embodiment, the continuity of multiple data sequence identifiers within multiple container packets is detected to obtain a detection result. If the detection result indicates that the multiple data sequence identifiers within the multiple container packets are continuous and without missing data, the multiple container packets are sorted according to the order in which the multiple container identifiers are represented to obtain the target data. In this way, sorting multiple container packets based on the continuity detection result of multiple data sequence identifiers can quickly detect lost data packets compared to the method of detecting lost data packets by relying on data timeout in related technologies, reducing latency and improving data processing efficiency.

[0103] This application provides a data processing method. Please refer to the embodiments provided. Figure 5 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 4 The implementation of S201 shown may include S301 to S303, which will be explained in conjunction with the steps.

[0104] S301. If the detection result indicates that multiple first data sequence identifiers within the first container are missing, a first lookup request is sent to the source based on the multiple first data sequence identifiers and their corresponding communication attribute information.

[0105] In this embodiment of the application, if the detection result indicates that multiple first data sequences within the first container are missing, a first supplement request is sent to the source end, carrying multiple first data sequence identifiers and communication attribute information.

[0106] The communication attribute information is used to characterize the communication address and memory-related information of the data packets within the multiple container packages; the communication attribute information can be QP information.

[0107] S302, Receive the second container package fed back by the source end in response to the first query request, and use the second container package to replace the first container package to update the multiple container packages.

[0108] In this embodiment, after receiving the first supplementation request, the source end responds by forming a second container packet that supplements and improves the multiple data packets within the first container packet. The source end sends the second container packet to the destination end, which replaces the first container packet with the second container packet to obtain updated multiple container packets.

[0109] S303. Sort the updated container packages according to the order of the container identifiers to obtain the target data.

[0110] In this embodiment of the application, the destination end sorts the updated container packages according to the order represented by the multiple container identifiers to obtain the target data.

[0111] For example, in combination Figure 6 The source sends container packet 1 and container packet 2 to the destination through the network device. The destination sorts container packet 1 and container packet 2 in order to obtain the target data QP1: container packet 2 + container packet 2 + ...

[0112] In this embodiment, after the destination card receives the container packet, it needs to determine whether there is packet loss. If packet loss exists, it will be quickly retransmitted; if no packet loss exists, it needs to be sorted. Because the multiple data packets inside the container packet are ordered, the destination checks whether there is packet loss inside the received container packet. If the PSN numbers of the received container packets are in the order of 21, 22, 23, 25..., then it is considered that the data packet with PSN=24 has been lost. After the destination detects the packet loss, it will send a data packet carrying the QP and PSN to the source. After receiving the information, the source looks up the correspondence table between QP and PSN according to the QP information and PSN data packet in the information, finds the Personal Subscription Number (IB) PSN, finds the data according to the IB PSN, encapsulates it, re-forwards it to the receiving network card, and sorts it to obtain a new container packet.

[0113] In this embodiment, if the detection result indicates that multiple first data sequence identifiers within the first container packet are missing, a first lookup request is sent to the source end based on the multiple first data sequence identifiers and their corresponding communication attribute information. The source end receives a second container packet in response to the first lookup request, and the second container packet is used to replace the first container packet to update the multiple container packets. Thus, compared to related technologies that rely on data packet timeouts to detect missing data packets, this approach eliminates the time delay of waiting for data packet timeouts, allowing for faster detection and retrieval of missing data packets, thereby improving data processing efficiency.

[0114] This application provides a data processing method. Please refer to the embodiments provided. Figure 7 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 4 The implementation may include S401 to S404 after S201 shown in the figure, which will be explained in conjunction with the steps.

[0115] S401. If the detection result indicates that multiple second data sequence identifiers are missing from the third container, then multiple second data sequence identifiers are determined based on the data sequence identifiers in adjacent containers.

[0116] In this embodiment of the application, if the detection result indicates that all data sequence identifiers in the third container are missing, that is, all data packets in the third container are missing, then the destination determines multiple missing second data sequence identifiers in the third container based on the data sequence identifiers in the adjacent container.

[0117] Among them, the adjacent container packs are the container packs corresponding to the two container identifiers that are adjacent to the container identifier of the third container pack.

[0118] In this embodiment of the application, the destination end can determine that the missing data sequence identifier in the middle of multiple data sequence identifiers in two adjacent containers is a multiple second data sequence identifier.

[0119] S402. Based on multiple second data sequence identifiers and communication attribute information, a second lookup request is sent to the source end.

[0120] In this embodiment of the application, the destination end carries multiple second data sequence identifiers and communication attribute information and sends a second lookup request to the source end.

[0121] Since the QP information corresponding to the first data packet and the third data packet is the same, the communication attribute information corresponding to the first data packet and the third data packet is also the same.

[0122] S403. Receive the fourth container package in response to the second lookup request from the source end, and use the fourth container package to replace the third container package to update the multiple container packages.

[0123] In this embodiment, the destination receives a fourth container packet from the source in response to the second missing data request. The fourth container packet includes multiple data packets that are missing from the third container packet. The destination replaces the third container packet with the fourth container packet, obtaining updated container packets.

[0124] S404. Sort the updated container packages according to the order of the container identifier representations to obtain the target data.

[0125] In this embodiment of the application, the destination end sorts the updated container packages according to the order represented by the multiple container identifiers to obtain the target data.

[0126] In this embodiment, if the detection result indicates the absence of multiple second data sequence identifiers within the third container packet, then multiple second data sequence identifiers are determined based on the data sequence identifiers within adjacent container packets. Based on the multiple second data sequence identifiers and communication attribute information, a second lookup request is sent to the source. A fourth container packet, responding to the second lookup request from the source, is received, and the fourth container packet is used to replace the third container packet to update the multiple container packets. Thus, compared to related technologies that rely on data packet timeouts to detect missing data packets, this approach eliminates the time delay of waiting for data packet timeouts, allowing for faster detection of missing data packets and consequently, faster retrieval of missing data packets, thereby improving data processing efficiency.

[0127] This application provides a data processing method. Please refer to the embodiments provided. Figure 8 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 2 The S101 shown can also be implemented by S501 to S502, which will be explained in conjunction with the steps.

[0128] S501. Obtain the multiple initial container packets sent by the source end.

[0129] In this embodiment, the destination end obtains multiple initial container packets sent by the source end. Each initial container packet contains data packets that include corresponding data attribute information (i.e., QP information).

[0130] S502. Determine multiple container packages that include the same communication attribute information among the multiple initial container packages.

[0131] In this embodiment, the destination determines multiple container packets that include the same communication attribute information from multiple initial container packets. That is, it determines multiple container packets that include the same QP information from multiple initial container packets.

[0132] In some other embodiments, after the target data is determined at the destination, the communication attribute information is stored, and a task completion notification is sent back to the source.

[0133] The destination end sorts each collected container according to its QP information. Since the data packets within a container are ordered and belong to the same QP, the destination end only needs to sort packets belonging to the same QP, greatly reducing the sorting pressure on the network interface card (NIC). After sorting according to each QP, the destination end writes each QP information into memory and sends it to the source end's CQE (Certified Quality Engineer) to notify the source end that the NIC task is complete.

[0134] In this embodiment, multiple container packets containing the same communication attribute information are identified from multiple initial container packets. Thus, when faced with a large number of initial container packets, sorting them according to their different communication attribute information saves data processing time and improves data processing efficiency compared to related technologies that involve packet segmentation and random reassembly for each type of traffic.

[0135] This application provides a data processing method. Please refer to the embodiments provided. Figure 9 This is an optional flowchart illustrating the data processing method provided in the embodiments of this application, which will be described in conjunction with the steps.

[0136] S601. The acquired data to be processed is segmented to obtain N data packets in a certain order.

[0137] In this embodiment, the source determines the data to be processed from multiple raw data sets, and then segments the data to be processed to obtain N data packets in a certain order. N is an integer greater than 1.

[0138] The source end can determine the data to be processed, including the predetermined message header, from multiple raw data sources.

[0139] For example, the data to be processed can be an RDMA data stream. The predefined message header can be an IB header. The source will first detect the traffic type. If it is RDMA traffic, it needs to be segmented and the flag field in the GSE header will be set to 1; if it is non-RDMA traffic, it does not need to be segmented and the flag will be set to 0.

[0140] S602. The N data packets are encapsulated into multiple container packets in sequence and sent to the destination; wherein each container packet includes: multiple data packets, each data packet includes a container identifier of the container packet to which it belongs and a corresponding data sequence identifier; the data sequence identifiers corresponding to the data packets in the multiple container packets are continuous.

[0141] In this embodiment, the source end encapsulates N data packets into multiple container packets in sequence and sends them to the destination end. Each container packet includes multiple data packets, and each data packet includes a container identifier belonging to its container packet and a corresponding data sequence identifier; the data sequence identifiers corresponding to the data packets within the multiple container packets are sequential.

[0142] In this embodiment, after determining the RDMA data, it needs to be segmented. The segmented data packets are divided into packet containers. Each packet within a container has its corresponding GSE header added sequentially. Each packet's GSE header includes a container ID, QP, a simple flag, and a PSN number. For each generated container packet, the container identifier is incremented by 1, ensuring that packets with the same QP are encapsulated. After the source end encapsulates multiple container packets, it sends them to the network device. Upon receiving the container packets, the network device performs packet spraying based on the container ID. Container packets with the same QP information are forwarded along the same path, while different QP information is hashed to different paths according to a hash rule.

[0143] In this way, during the sorting of multiple container packets at the destination, the existence of lost data packets can be quickly determined based on the continuity of the corresponding data sequence identifiers. Compared with related technologies that rely on data timeouts to detect lost data packets, this method can quickly detect lost data packets, reduce latency, and improve data processing efficiency. Furthermore, this application relies on the container identifiers of each container packet for sorting, and each container packet contains multiple data packets. This sorting method can also save computing resources to some extent and improve data processing efficiency when dealing with large volumes of data.

[0144] This application provides a data processing method. Please refer to the embodiments provided. Figure 10 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 9 The step S602 shown can also be implemented by steps S701 to S703, which will be explained in conjunction with the steps.

[0145] S701. Add the data sequence identifier and communication attribute information to the extension header of each data packet in sequence.

[0146] In this embodiment, after the data to be processed is segmented at the source end, N data packets with a certain order are obtained. The source end adds a corresponding data sequence identifier and communication attribute information to the extension header of each data packet in order.

[0147] The data sequence identifiers corresponding to the N data packets are consecutive. Communication attribute information can be determined based on the communication addresses and storage information corresponding to the source and destination ends, respectively.

[0148] S702. Add the container identifier to the extension header of every M data packets in sequence.

[0149] In this embodiment, the source adds a container identifier to the extension header of every M data packets in the order of N data packets. Here, M is an integer greater than 0 and less than N. The size of M can be determined based on the number of data packets that the container can hold.

[0150] The value of the container identifier within each M data packet increases sequentially.

[0151] S703. Encapsulate each M data packets into a container packet until all N data packets are encapsulated, thus obtaining multiple container packets.

[0152] In this embodiment of the application, the source end encapsulates every M data packets into a container packet until all N data packets are encapsulated to obtain multiple container packets.

[0153] In this embodiment, data sequence identifiers and communication attribute information are added sequentially to the extension header of each data packet. Container identifiers are added sequentially to the extension headers of every M data packets; each M data packets is encapsulated into a container packet, until all N data packets are encapsulated, resulting in multiple container packets. Because the data sequence identifiers within the N data packets have a certain continuity, after the destination receives multiple container packets, it can detect whether there are any missing data packets based on the data sequence identifiers of the multiple data packets within the multiple container packets. Missing data packets can be identified without waiting for data timeouts, saving time in identifying missing data packets and thus improving data processing efficiency.

[0154] This application provides a data processing method. Please refer to the embodiments provided. Figure 11 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 10 S703 shown may be followed by S801 to S802, which will be explained in conjunction with the steps.

[0155] S801. Construct the correspondence between the original identifier of each data packet and the corresponding data sequence identifier and communication attribute information.

[0156] In this embodiment, the source end constructs the correspondence between the original identifier of each data packet and the corresponding data sequence identifier and communication attribute information.

[0157] The original identifier can be the IB PSN number included in the IB header of the data packet.

[0158] For example, the correspondence between the original identifier of each data packet and the corresponding data sequence identifier and communication attribute information can be illustrated by Table 1.

[0159] Communication attribute information Data sequence identifier Original Identifier x000086 1 4139376 0x000086 2 4139377

[0160] Table 1

[0161] S802. Store the correspondence in a correspondence set; wherein the correspondence set includes the correspondence between the original identifier of different data packets and the corresponding data sequence identifier and the communication attribute information.

[0162] In this embodiment of the application, the source end can store the formed correspondence into a correspondence set, wherein the correspondence set includes the correspondence between the original identifier of different data packets and the corresponding data sequence identifier and the communication attribute information.

[0163] In this embodiment of the application, a table of correspondence between QP and PSN is maintained. When packet loss occurs at the destination end, the source end will look up the corresponding table based on the QP information and PSN information, thereby quickly finding the lost IB PSN number and retransmitting the container packet corresponding to the data.

[0164] This application provides a data processing method. Please refer to the embodiments provided. Figure 12 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 11 S802 shown may be followed by S803 to S806, which will be explained in conjunction with the steps.

[0165] S803. Receive a first lookup request sent by the destination terminal; wherein the first lookup request includes: multiple first data sequence identifiers and communication attribute information.

[0166] In this embodiment of the application, the source end receives a first lookup request sent by the destination end. The first lookup request includes multiple first data sequence identifiers and communication data information.

[0167] Among them, communication attribute information is also known as QP information.

[0168] S804. Based on the communication attribute information, search for the missing third data sequence identifier among the multiple first data sequence identifiers in the correspondence set.

[0169] In this embodiment of the application, the source end can search for the missing third data sequence identifier in the corresponding relationship set based on the communication attribute information.

[0170] For example, the source end can search for multiple consecutive data sequence identifiers in the corresponding relationship set based on communication attribute information. Then, it determines the missing third data sequence identifier based on the multiple first data sequence identifiers in the first lookup request.

[0171] S805. After supplementing the multiple first data sequence identifiers with the third data sequence identifier, search for the multiple first original identifiers corresponding to the multiple first data sequence identifiers in the correspondence set.

[0172] In this embodiment, the source end uses a third data sequence identifier to supplement multiple missing first data sequence identifiers, obtaining multiple supplemented first data sequence identifiers. The supplemented multiple first data sequence identifiers are then used to search for corresponding multiple original first identifiers in a correspondence set.

[0173] S806. Pack the data packets corresponding to the multiple first original identifiers into a second container packet in order, and send it to the destination.

[0174] In this embodiment, the source end can retrieve multiple data packets based on multiple first original identifiers, then package these multiple data packets into a second container packet according to their order, and send the second container packet to the destination end through a network device. The extended headers of each data packet within the second container packet are configured with corresponding first sequence identifiers, as well as corresponding data attribute information and a container identifier, in that order.

[0175] In this embodiment, based on communication attribute information, a missing third data sequence identifier is searched in a correspondence set to find the missing third data sequence identifier among multiple first data sequence identifiers. After supplementing the multiple first data sequence identifiers with the third data sequence identifier, multiple first original identifiers corresponding to the multiple first data sequence identifiers are searched in the correspondence set. The data packets corresponding to the multiple first original identifiers are packaged into a second container packet in sequence and sent to the destination. In this way, when the destination detects that the first container packet has missing data, the replacement packet for the first container packet can be determined at the source based on the multiple first data sequence identifiers and communication attribute information in the first data packet, without waiting for the timeout of the missing data packet, saving time in detecting and determining the lost data packet. In addition, in this application, a second container packet to replace the first container packet is determined based on the supplemented multiple first data sequence identifiers, without retransmitting the missing data packet and all data packets after the missing data packet, saving computing resources at the source.

[0176] This application provides a data processing method. Please refer to the embodiments provided. Figure 13 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application. Figure 11 S802 shown may be followed by S807 to S809, which will be explained in conjunction with the steps.

[0177] S807. Receive a second lookup request sent by the destination terminal; wherein the second lookup request includes: multiple second data sequence identifiers and communication attribute information.

[0178] In this embodiment of the application, the source end receives a second lookup request sent by the destination end. The second lookup request includes: multiple second data sequence identifiers and communication attribute information.

[0179] S808. Based on the communication attribute information, search for multiple second original identifiers corresponding to multiple second data sequence identifiers in the correspondence set.

[0180] In this embodiment of the application, the source end finds multiple consecutive data sequence identifiers in the corresponding relationship set based on communication attribute information, and finds multiple second original identifiers corresponding to multiple second data sequence identifiers according to the original identifiers corresponding to the multiple consecutive data sequence identifiers.

[0181] S809. Pack the data packets corresponding to the multiple second original identifiers into a fourth container packet in order, and send it to the destination.

[0182] In this embodiment, the source retrieves multiple data packets corresponding to the second original identifiers, packages these multiple data packets into a fourth data packet in sequence, and sends it to the destination via a network device. The extended headers of each data packet within the fourth container packet are configured with corresponding second sequence identifiers, as well as corresponding data attribute information and container identifiers in sequence.

[0183] In this embodiment, based on communication attribute information, multiple second original identifiers corresponding to multiple second data sequence identifiers are searched in the correspondence set. Data packets corresponding to the multiple second original identifiers are packaged into a fourth container packet in sequence and sent to the destination. Thus, when the destination detects that all data packets of the third container packet are missing, the replacement packet for the third container packet can be determined at the source based on the multiple second data sequence identifiers and communication attribute information in the third data packet, without waiting for the timeout of the missing data packets, saving time in detecting and determining the lost data packets. Furthermore, this application determines a replacement fourth container packet for the third container packet based on the supplemented multiple second data sequence identifiers, eliminating the need to resend the missing data packets and all data packets after the missing data packets, saving computational resources at the source.

[0184] This application provides a data processing method. Please refer to the embodiments provided. Figure 14 This is an interactive schematic diagram of the data processing method provided in the embodiments of this application.

[0185] S901, The source end performs data segmentation on the acquired data to be processed, resulting in N data packets in a certain order.

[0186] In this embodiment, the implementation process of S901 can refer to S601, and will not be described in detail here.

[0187] S902. The source end encapsulates the N data packets into multiple container packets in sequence and sends them to the destination end; wherein, each container packet includes: multiple data packets, and each data packet includes the container identifier of the container packet to which it belongs and the corresponding data sequence identifier.

[0188] In this embodiment, the implementation process of S902 can refer to S602, and will not be described in detail here.

[0189] S903. The destination end sorts the multiple container packets based on the container identifiers corresponding to the multiple container packets respectively, and the continuity of the data sequence identifiers corresponding to the data packets within the multiple container packets, to obtain the target data.

[0190] In this embodiment, the implementation process of S903 can be referred to S102, and will not be described in detail here.

[0191] Please see Figure 15 A schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. Figure 1 .

[0192] This application also provides a data processing device 600, including a data acquisition unit 601 and a data processing unit 602.

[0193] The data acquisition unit 601 is used to acquire multiple container packets sent by the source end: wherein each container packet includes multiple data packets, and each data packet includes a container identifier belonging to the container packet and a corresponding data sequence identifier;

[0194] The data processing unit 602 is used to sort the multiple container packets to obtain target data based on the container identifiers corresponding to the multiple container packets respectively and the continuity of the data sequence identifiers corresponding to the data packets within the multiple container packets.

[0195] In this embodiment of the application, the data processing unit 602 in the data processing device 600 is used to detect the continuity of multiple data sequence identifiers within multiple container packages and obtain a detection result; if the detection result indicates that multiple data sequence identifiers within multiple container packages are continuous and without missing information, then the multiple container packages are sorted according to the order in which the multiple container identifiers are characterized to obtain the target data.

[0196] In this embodiment of the application, the data processing unit 602 in the data processing device 600 is used to send a first lookup request to the source end based on the multiple first data sequence identifiers and the corresponding communication attribute information if the detection result indicates that multiple first data sequence identifiers in the first container are missing; wherein, the communication attribute information is used to characterize the communication address and memory-related information of the data packets in the multiple containers;

[0197] The system receives the second container package as a response to the first lookup request from the source end, and uses the second container package to replace the first container package to update multiple container packages.

[0198] The target data is obtained by sorting the updated container packages according to the order of the container identifier representations.

[0199] In this embodiment of the application, the data processing unit 602 in the data processing device 600 is used to determine multiple second data sequence identifiers based on the data sequence identifiers in adjacent container packages if the detection result indicates that multiple second data sequence identifiers are missing in the third container package; wherein, the adjacent container package is the container package corresponding to two container identifiers that are adjacent to the container identifier corresponding to the third container package;

[0200] Based on multiple second data sequence identifiers and communication attribute information, a second lookup request is sent to the source end;

[0201] The system receives the fourth container package as a response to the second lookup request from the source end, and uses the fourth container package to replace the third container package to update multiple container packages.

[0202] The target data is obtained by sorting the updated container packages according to the order of the container identifier representations.

[0203] In this embodiment of the application, the data acquisition unit 602 in the data processing device 600 is used to acquire multiple initial container packets sent by the source end;

[0204] Among the multiple initial container packages, multiple container packages that include the same communication attribute information are identified.

[0205] In this embodiment of the application, the data processing device 600 is used to store the communication attribute information and send a task completion notification back to the source end.

[0206] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a data processing device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0207] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the method on the destination side.

[0208] Correspondingly, this application also provides a computer program product, including a computer program that can be executed by a first processor 701 of an electronic device 700 to complete the steps in the method on the destination side.

[0209] Correspondingly, this application provides an electronic device 700, including a first memory 702 and a first processor 701. The memory 702 stores a computer program that can run on the first processor 701. When the first processor 701 executes the program, it implements the steps in the above method.

[0210] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0211] It should be noted that, Figure 16 A hardware entity illustration of an electronic device provided in the embodiments of this application. Figure 1 ,like Figure 16 As shown, the hardware entity of the electronic device 700 includes: a first processor 701 and a first memory 702, wherein;

[0212] The first processor 701 typically controls the overall operation of the electronic device 700.

[0213] The first memory 702 is configured to store instructions and applications executable by the first processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) of the processor 701 and various modules in the electronic device 700, which can be implemented by flash memory or random access memory (RAM).

[0214] Please see Figure 17 A schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. Figure 2 .

[0215] This application also provides a data processing device 800, including: a data segmentation unit 801 and an encapsulation and transmission unit 802.

[0216] The data segmentation unit 801 is used to segment the acquired data to be processed into N data packets in a certain order; N is an integer greater than 1.

[0217] The encapsulation and transmission unit 802 is used to encapsulate N data packets into multiple container packets in sequence and send them to the destination; wherein each container packet includes multiple data packets, and each data packet includes a container identifier of the container packet to which it belongs and a corresponding data sequence identifier.

[0218] In this embodiment of the application, the data segmentation unit 801 in the data processing device 800 is used to add the data sequence identifier and communication attribute information to the extension header of each data packet in sequence; wherein, the data sequence identifiers corresponding to the N data packets are consecutive;

[0219] The container identifier is added sequentially to the extended header of every M data packets; where M is an integer greater than 0 and less than N.

[0220] Each M data packets is encapsulated into a container packet until all N data packets are encapsulated, resulting in multiple container packets.

[0221] In this embodiment of the application, the data processing device 800 is used to construct a correspondence between the original identifier of each data packet and the corresponding data sequence identifier and the communication attribute information; and to store the correspondence in a correspondence set; wherein the correspondence set includes correspondences between the original identifier of different data packets and the corresponding data sequence identifier and the communication attribute information.

[0222] In this embodiment of the application, the data processing device 800 is used to receive a first lookup request sent by the destination terminal; wherein, the first lookup request includes: a plurality of first data sequence identifiers and communication attribute information;

[0223] Based on the communication attribute information, a third data sequence identifier that is missing in multiple first data sequence identifiers is searched in the correspondence set;

[0224] After supplementing the multiple first data sequence identifiers with the third data sequence identifier, the multiple first original identifiers corresponding to the multiple first data sequence identifiers are searched in the correspondence set;

[0225] The data packets corresponding to the multiple first original identifiers are packaged into a second container packet in order and sent to the destination.

[0226] In this embodiment of the application, the data processing device 800 is used to receive a second lookup request sent by the destination terminal; wherein, the second lookup request includes: a plurality of second data sequence identifiers and communication attribute information;

[0227] Based on the communication attribute information, multiple second original identifiers corresponding to multiple second data sequence identifiers are searched in the correspondence set;

[0228] The data packets corresponding to the multiple second original identifiers are packaged into a fourth container packet in sequence and sent to the destination.

[0229] In this embodiment of the application, the data segmentation unit 801 in the data processing device 800 is used to determine the data to be processed, which includes a predetermined message header, from a plurality of raw data.

[0230] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the method on the destination side.

[0231] Correspondingly, this application also provides a computer program product, including a computer program that can be executed by a second processor 901 of an electronic device 900 to complete the steps in the method on the destination side.

[0232] Correspondingly, this application provides an electronic device 900, including a second memory 902 and a second processor 901. The memory 902 stores a computer program that can run on the second processor 901. When the second processor 901 executes the program, it implements the steps in the above method.

[0233] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0234] It should be noted that, Figure 18 A hardware entity illustration of an electronic device provided in the embodiments of this application. Figure 2 ,like Figure 18 As shown, the hardware entity of the electronic device 900 includes: a second processor 901 and a second memory 902, wherein;

[0235] The second processor 901 typically controls the overall operation of the electronic device 900.

[0236] The second memory 902 is configured to store instructions and applications executable by the second processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) of the processor 901 and various modules in the electronic device 900. It can be implemented by flash memory or random access memory (RAM).

[0237] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0238] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0239] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0240] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0241] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0242] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0243] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0244] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a plurality of container packages sent by a source end, wherein each of the container packages comprises a plurality of data packages, and each of the data packages comprises a container identifier of the container package and a corresponding data sequence identifier; sorting the plurality of container packages based on the container identifiers corresponding to the plurality of container packages respectively and continuity of the data sequence identifiers corresponding to the data packages in the plurality of container packages to obtain target data.

2. The data processing method according to claim 1, characterized in that, The sorting the plurality of container packages based on the container identifiers corresponding to the plurality of container packages respectively and continuity of the data sequence identifiers corresponding to the data packages in the plurality of container packages to obtain target data comprises: detecting continuity of the data sequence identifiers in the plurality of container packages to obtain a detection result; if the detection result represents that the data sequence identifiers in the plurality of container packages are continuous without missing, sorting the plurality of container packages according to an order represented by the container identifiers to obtain the target data.

3. The data processing method according to claim 2, characterized in that, After the detecting continuity of the data sequence identifiers in the plurality of container packages to obtain a detection result, the method further comprises: if the detection result represents that a plurality of first data sequence identifiers in a first container package are missing, sending a first request for supplement to the source end based on the plurality of first data sequence identifiers and corresponding communication attribute information, wherein the communication attribute information is used to represent communication addresses and memory related information of the data packages in the plurality of container packages; receiving a second container package fed back by the source end in response to the first request for supplement, and updating the plurality of container packages by replacing the first container package with the second container package; sorting the updated plurality of container packages according to the order represented by the container identifiers to obtain the target data.

4. The data processing method according to claim 2, characterized in that, After the detecting continuity of the data sequence identifiers in the plurality of container packages to obtain a detection result, the method further comprises: if the detection result represents that a plurality of second data sequence identifiers in a third container package are missing, determining the plurality of second data sequence identifiers based on the data sequence identifiers in adjacent container packages, wherein the adjacent container packages are the container packages corresponding to two container identifiers adjacent to the container identifier corresponding to the third container package; sending a second request for supplement to the source end based on the plurality of second data sequence identifiers and communication attribute information; receiving a fourth container package fed back by the source end in response to the second request for supplement, and updating the plurality of container packages by replacing the third container package with the fourth container package; sorting the updated plurality of container packages according to the order represented by the container identifiers to obtain the target data.

5. The data processing method according to any one of claims 1 to 4, characterized in that, The obtaining the plurality of container packages sent by the source end comprises: obtaining a plurality of initial container packages sent by the source end; determining the plurality of container packages comprising the same communication attribute information in the plurality of initial container packages.

6. The data processing method according to claim 5, characterized in that, After the sorting the plurality of container packages based on the container identifiers corresponding to the plurality of container packages respectively and continuity of the data sequence identifiers corresponding to the data packages in the plurality of container packages to obtain target data, the method further comprises: Store the communication attribute information and feed back a task completion notification to the source end.

7. A data processing method, characterized by, Comprise: Data segmentation is carried out on the obtained to-be-processed data, and N data packets in a certain order are obtained; N is an integer greater than 1; N The data packets are encapsulated into a plurality of container packets in order, and sent to the destination end; wherein each container packet comprises: a plurality of data packets, each data packet comprises a container identifier of the container packet to which it belongs and a corresponding data sequence identifier; The data sequence identifiers corresponding to the data packets in the plurality of container packets have continuity.

8. The data processing method according to claim 7, characterized in that, Encapsulating N The data packets into a plurality of container packets in order, comprising: Add the data sequence identifier and the communication attribute information to the extension header of each data packet in order; wherein the data sequence identifiers corresponding to N The data packets are continuous; Add the container identifier to the extension header of each M The data packet in order; wherein M is an integer greater than 0 and less than N; Encapsulate every M The data packet into a container packet until N The data packets are all encapsulated to obtain a plurality of container packets.

9. The data processing method according to claim 8, characterized in that, The method further comprises: Build the correspondence between the original identifier of each data packet and the corresponding data sequence identifier and the communication attribute information; Store the correspondence in the correspondence set; wherein the correspondence set comprises the correspondence between the original identifier of different data packets and the corresponding data sequence identifier and the communication attribute information.

10. The data processing method according to claim 9, characterized in that, After N The data packets are encapsulated into a plurality of container packets in order and sent to the destination end, the method further comprises: Receive the first query request sent by the destination end; wherein the first query request comprises: a plurality of first data sequence identifiers and communication attribute information; Based on the communication attribute information, find the third data sequence identifier missing in the plurality of first data sequence identifiers in the correspondence set; After supplementing the plurality of first data sequence identifiers with the third data sequence identifier, find the plurality of first original identifiers corresponding to the plurality of first data sequence identifiers in the correspondence set; Pack the data packets corresponding to the plurality of first original identifiers into a second container packet in order and send to the destination end.

11. The data processing method according to claim 9, characterized in that, After N The data packets are encapsulated into a plurality of container packets in order and sent to the destination end, the method further comprises: Receive the second query request sent by the destination end; wherein the second query request comprises: a plurality of second data sequence identifiers and communication attribute information; Based on the communication attribute information, find the plurality of second original identifiers corresponding to the plurality of second data sequence identifiers in the correspondence set; Pack the data packets corresponding to the plurality of second original identifiers into a fourth container packet in order and send to the destination end.

12. The data processing method according to any one of claims 7 to 11, characterized in that, Before N The data packets are obtained by data segmentation on the obtained to-be-processed data in a certain order, comprising: Determine the to-be-processed data including the predetermined packet header in the plurality of original data.

13. A data processing apparatus, characterized by: Comprise: The data acquisition unit is configured to acquire a plurality of container packets sent by the source end, wherein each container packet comprises a plurality of data packets, and each data packet comprises a container identifier of the container packet and a corresponding data sequence identifier. The data processing unit is configured to sort the plurality of container packets based on the container identifiers corresponding to the plurality of container packets respectively and the continuity of the data sequence identifiers corresponding to the data packets in the plurality of container packets, to obtain target data.

14. A data processing apparatus, characterized by The method comprises: The data splitting unit is configured to perform data splitting on the acquired to-be-processed data to obtain N data packets in a certain order. N is an integer greater than 1. The encapsulation and sending unit is configured to encapsulate the N data packets into a plurality of container packets in the order and send the container packets to the destination end, wherein each container packet comprises a plurality of data packets, and each data packet comprises a container identifier of the container packet and a corresponding data sequence identifier; and the data sequence identifiers corresponding to the data packets in the plurality of container packets have continuity.

15. An electronic device, comprising: The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 6, or the steps in the method of any one of claims 7 to 12.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 6, or the steps in the method of any one of claims 7 to 12.

17. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 6, or the steps in the method of any one of claims 7 to 12.

Images