A data processing method and related device

Abstract

The embodiment of the application discloses a data processing method, and a receiving end can identify the identity of a first sending end of a first incomplete message in a first receiving queue according to the first incomplete message, thereby actively detecting the fault condition of the first sending end. Then, the active processing of the first incomplete message in the first receiving queue can be realized according to the fault condition of the first sending end, so that the receiving task corresponding to the first incomplete message in the first receiving queue can be ended, thereby the corresponding receiving resource can be released subsequently.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, specifically to a data processing method and related equipment. Background Technology

[0002] With the rapid development of applications such as high-performance computing, artificial intelligence, and big data, the data transmission needs between devices and the scenarios of network interconnection are becoming increasingly complex. In some communication methods, in order to facilitate the receiving end to receive request messages from one or more sending ends, the receiving queue in the receiving end can request receiving resources from multiple processes of the receiving end to receive request messages from the sending end for each process.

[0003] However, during data transmission, the sending end may experience a failure and be unable to send a complete request message. Since the failure message from the sending end cannot be notified to the receiving end, the receiving resources in the receiving queue of the receiving end, which are used to receive request messages, cannot complete the receiving task. As a result, the receiving resources cannot be reclaimed, and the receiving resources of the corresponding process of the receiving end are wasted. Summary of the Invention

[0004] This application provides a data processing method that solves the problem of wasted receiving resources at the receiving end due to the inability to reclaim receiving resources in the receiving queue of the receiving end caused by a failure at the sending end. This application also provides corresponding apparatus, devices, computer-readable storage media, and computer program products.

[0005] The first aspect of this application provides a data processing method, the method comprising: detecting a first incomplete message in a first receiving queue; upon detecting at least one first incomplete message in the first receiving queue, obtaining an identifier of each first sender among at least one first sender of the at least one first incomplete message; determining a fault condition of each first sender based on the identifier of each first sender; and processing the at least one first incomplete message based on the fault condition of each first sender.

[0006] In the first aspect, the receiving end can identify the identifier of the first sender of the first incomplete message based on the first incomplete message in the first receiving queue, thereby actively detecting the fault status of the first sender. Then, based on the fault status of the first sender, it can actively process the first incomplete message in the first receiving queue, so that the receiving task corresponding to the first incomplete message in the first receiving queue can be completed, thereby releasing the corresponding receiving resources in the future.

[0007] In one possible implementation of the first aspect, detecting a first incomplete message in a first receiving queue includes: if the first receiving queue receives a first message and an incomplete message is generated based on the first message, then a timer is started; after a first duration of timer start, if the incomplete message generated based on the first message is incomplete, then the incomplete message generated based on the first message is taken as the first incomplete message.

[0008] In this possible implementation, when an incomplete message is detected, message-level timing can be performed on the incomplete message to ensure the reliability of the detected incomplete message and achieve efficient and accurate detection of one or more incomplete messages in the receiving queue.

[0009] In one possible implementation of the first aspect, timing begins if the first receiving queue receives the first message and generates an incomplete message based on the first message, including: timing begins if the first receiving queue receives the first message and generates a new incomplete message based on the first message.

[0010] In this possible implementation, timing is only started when a new incomplete message is generated based on the message, which can avoid repeatedly timing the same incomplete message and thus reduce resource consumption.

[0011] In one possible implementation of the first aspect, determining the fault condition of each first sender based on the identifier of each first sender includes: obtaining the identifier of each second sender among at least one second sender, wherein the at least one second sender is the sender of at least one second incomplete message in the second receive queue; and determining the fault condition of each first sender based on the identifier of each second sender and the identifier of each first sender.

[0012] In this possible implementation, by comprehensively judging whether the sender is faulty based on the sender information of incomplete messages in multiple receiving queues, the situation of incomplete messages caused by network congestion can be better ruled out, thereby making a more accurate judgment on the fault status of the first sender.

[0013] In one possible implementation of the first aspect, determining the fault condition of each first sender based on the identifier of each second sender and the identifier of each first sender includes: designating the first sender that is the same as any second sender as the fault sender; and processing at least one first incomplete message based on the fault condition of each first sender includes: generating a first completion notification for the first incomplete message from the fault sender in a first receive queue, the first completion notification indicating that the first incomplete message from the fault sender has been received.

[0014] In this possible implementation, the failure status of the first sender can indicate whether the first sender is a faulty sender or a normal sender, and the processing method for the first incomplete message can be different under different failure statuses of the first sender. Specifically, if a certain first sender is the same as a certain second sender, it can be assumed that multiple receive queues cannot completely receive the message from that first sender. Therefore, the first sender that is the same as the second sender can be considered as a faulty sender.

[0015] In this way, a first completion notification can be generated for the first incomplete message from the faulty sender in the first receive queue, indicating that the first incomplete message from the faulty sender has been received. This allows the receiving resources in the first receive queue used to receive messages from the faulty sender to be reclaimed instead of being continuously occupied by the first receive queue, so that the receiving resources and the corresponding processes can be conveniently deregistered in the future.

[0016] In one possible implementation of the first aspect, the first completion notification indicates that the first incomplete message from the fault sender is an incomplete message.

[0017] In this possible implementation, the corresponding process receiving the first incomplete message at the receiving end can determine that the first incomplete message is an incomplete message, so as to facilitate subsequent processing, such as deletion or data processing of the first incomplete message.

[0018] In one possible implementation of the first aspect, determining the fault status of each first sender based on the identifier of each second sender and the identifier of each first sender includes: designating a first sender different from each second sender as a normal sender; and processing at least one first incomplete message based on the fault status of each first sender, including: waiting for missing packets in the first incomplete message from the normal sender in the first receive queue; and if the first incomplete message from the normal sender is incomplete after a waiting period of a second duration, generating a second completion notification for the first incomplete message from the normal sender, the second completion notification indicating that the first incomplete message from the normal sender has been successfully received.

[0019] In this possible implementation, if a first sender is different from any second sender, and other receiving queues can also receive messages from that first sender, it indicates that all receiving queues except the first receiving queue can completely receive messages from that first sender. This suggests that the first sender is not faulty; rather, the message was not received by the receiving end due to network congestion or other issues with the transmission connection corresponding to the first receiving queue, resulting in the first incomplete message. Therefore, in this possible implementation, a second waiting period can be initiated to await messages that were not received by the receiving end due to network congestion or other issues. If the first incomplete message from the normal sender is still incomplete after the second waiting period, the corresponding receiving resources can be reclaimed via a second completion notification, preventing the first receiving queue from continuously occupying them.

[0020] A second aspect of this application provides a data processing apparatus having the function of implementing the method of the first aspect or any possible implementation thereof. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function, such as a detection module and a processing module.

[0021] A third aspect of this application provides an electronic device including at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0022] The fourth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0023] The fifth aspect of this application provides a computer program product that stores one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0024] A sixth aspect of this application provides a chip system including a processor for supporting an electronic device in implementing the functions described in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing necessary program instructions and data for the electronic device. This chip system may be composed of chips or may include chips and other discrete devices.

[0025] The technical effects of the second to sixth aspects or any of their possible implementations can be found in the first aspect or the technical effects of its related possible implementations, and will not be repeated here. Attached Figure Description

[0026] Figure 1 This is an exemplary schematic diagram of the XRC technology provided in the embodiments of this application;

[0027] Figure 2 This is an exemplary schematic diagram of the data processing method provided in the embodiments of this application;

[0028] Figure 3 This is an exemplary schematic diagram of detecting incomplete messages based on a bitmap algorithm provided in an embodiment of this application;

[0029] Figure 4 This is an exemplary schematic diagram of the system architecture provided in the embodiments of this application;

[0030] Figure 5 This is an exemplary schematic diagram of the processing flow provided in the embodiments of this application;

[0031] Figure 6 This is an exemplary schematic diagram of the system architecture provided in the embodiments of this application;

[0032] Figure 7 This is a schematic diagram of an embodiment of the data processing apparatus provided in this application;

[0033] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0035] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0036] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of singular or plural items. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatus.

[0037] With the rapid development of applications such as high-performance computing, artificial intelligence, and big data, the data transmission needs between devices and the scenarios of network interconnection are becoming increasingly complex.

[0038] For example, in one exemplary scenario, to meet the requirements of various application scenarios, high-performance data processing capabilities can be provided through clusters of computing devices such as data center clusters.

[0039] As data center clusters and other clusters continue to grow in scale, their network performance significantly impacts overall cluster performance. To ensure high bandwidth and low latency network performance while minimizing resource consumption, various developers offer a range of network interconnection protocols, such as Infiniband, Remote Direct Memory Access over Converged Ethernet (RoCE), Scalable Reliable Datagram (SRD), Falcon, and NVLink.

[0040] In real-world applications, the network interconnection between computing devices in data center clusters and other computing equipment clusters is quite complex, and the possible abnormal scenarios are also quite diverse. For example, network packet loss, network timeout, network failure, node failure, and other issues may occur.

[0041] When the sending end fails, the fault information cannot be notified to the receiving end, so the receiving resources in the receiving queue of the receiving end used to receive request messages cannot complete the receiving task and cannot be reclaimed. As a result, the receiving resources of the corresponding process of the receiving end are wasted because they cannot be cancelled.

[0042] The following section uses the Extended Reliable Connection (XRC) communication model in the Infiniband protocol as an example.

[0043] The communication entities of the Infiniband protocol XRC technology are the XRC initiate queue pair (INIQP), the XRC target queue pair (TGT QP), and the XRC share receive queue (SRQ), which will be referred to as INI TP, TGT QP, and SRQ respectively.

[0044] like Figure 1 As shown, the sending host (Host0) includes processes process0 and process1, and the receiving host (also called the target) (HOST1) includes processes process0 and process1. A connection is established between the INIQP and the TGT QP, and one INIQP connects to only one TGT QP. For example, Figure 1 In the example shown, XRC INIQP000 is connected to XRC TGT QP 10, while XRC INIQP010 is connected to XRC TGT QP 11. The TGT QP has no send request capability; request messages are always sent by the INIQP and received by the TGT. The TGT obtains receive resources from the SRQ to receive request messages. The TGT QP can point to multiple SRQs, each obtaining receive resources from a different SRQ. For example, Figure 1 In the example shown, XRC TGT QP 10 can obtain receive resources from XRCSRQ100 and XRC SRQ110 respectively; multiple TGT QPs can point to the same SRQ and consume the receive resources of the same SRQ, for example, Figure 1 In the example shown, both XRC TGT QP 10 and XRC TGT QP11 can obtain reception resources from XRC SRQ100. Therefore, SRQ can be considered a shared reception resource of TGT QP.

[0045] And in Figure 1In the example shown, when the sending host Host0 fails, the receiving host Host1's TGTQP may receive an incomplete request message. However, since the sending host Host0 has failed, the missing packets in the incomplete request message cannot be retransmitted. As a result, the receiving host Host1's TGT QP cannot collect the missing packets, causing the consumed receive resources in the TGT QP to be unrecoverable, thus wasting receive resources. Furthermore, if this situation continues to accumulate, it may also lead to the risk of SRQ receive resources being exhausted but unrecoverable.

[0046] Furthermore, incomplete message receiving tasks in TGT QP keep the services on the receiving end running. Currently, when the service is running, users cannot issue SRQ deregistration commands, preventing them from managing the deregistration and resource release of abnormal SRQs in real time. This makes the deregistration of abnormal SRQs inflexible and reduces the flexibility of user operations.

[0047] Alternatively, users could be allowed to issue SRQ deregistration commands during service operation. However, since the InfiniBand protocol does not define how to report the completion of all SRQ receptions, users need to be configured not to rely on the complete reclamation of reception resources before releasing resources when deregistering SRQ resources. However, this would result in residual received data on unreclaimed reception resources, and subsequent reporting of reception completion would affect the reuse of SRQs.

[0048] It is evident that in current network interconnection scenarios based on the Infiniband protocol, a failure at the sending end will prevent the receiving resources in the receiving queue of the receiving end used for receiving request messages from being reclaimed, resulting in the waste of receiving resources at the receiving end. This affects the cancellation of SRQs, making the cancellation of abnormal SRQs at the receiving end inflexible and impacting the subsequent use of receiving resources.

[0049] It is understood that the examples of the Infiniband protocol described above are merely illustrative network interconnection scenarios and not limiting. The same issues exist in many other network interconnection scenarios as well.

[0050] For example, in some exemplary network interconnection scenarios, multiple transport connections can be established between the sending host and the receiving host. Each transport connection can correspond to a sending queue on the sending end and a receiving queue on the receiving end. The sending queue on the sending end can be connected to multiple virtual machines or processes, allowing these virtual machines or processes to share the transport connection, meaning they can all send request messages to the receiving end through this connection. Furthermore, the receiving queue on the receiving end can be connected to multiple processes or virtual machines on the receiving end, allowing request messages from the sending end to be transmitted to multiple processes or virtual machines on the receiving end. In other words, in this exemplary scenario, a single transport connection can enable message transmission between multiple virtual machines or processes on the sending end and multiple processes or virtual machines on the receiving end.

[0051] In this exemplary scenario, if a process or virtual machine on the sending end fails, it will prevent the failed process or virtual machine from sending a complete request message. This will prevent the receiving task in the receiving queue on the receiving end from being completed, thus preventing the process related to the receiving task from being deregistered and preventing the corresponding receiving resources from being reclaimed. In addition, it will also cause other normal services on the sending end to suffer from head-of-queue blocking and other normal services that transmit messages through the receiving queue from being executed normally.

[0052] It is evident that in various current network interconnection scenarios, during data transmission, the sending end may experience a failure and be unable to send a complete request message. However, since the failure message from the sending end cannot notify the receiving end, the receiving resources in the receiving queue of the receiving end, which are used to receive request messages, cannot complete the receiving task and therefore cannot be reclaimed. Consequently, the receiving resources of the corresponding process of the receiving end are wasted because they cannot be deregistered.

[0053] To address the aforementioned issues, this application provides a data processing method that allows the receiving end to actively detect faults in the sending end. Based on the fault status of the first sending end, the method can proactively process the first incomplete message in the first receiving queue, thereby releasing the corresponding receiving resources in the future, avoiding waste of receiving resources, and facilitating flexible cancellation of receiving resources.

[0054] The data processing method of this application embodiment can be applied to electronic devices, which can be terminal devices, single servers or server clusters, or virtual machines (VMs) or containers (Docker), etc.

[0055] This electronic device can act as a receiver in a communication system, receiving messages from the sender.

[0056] The specific architecture of the communication system and the network interconnection method used are not restricted here.

[0057] For example, in one embodiment, the communication system may include a data center cluster. The electronic device acting as the receiving end can be any server or server cluster within the data center cluster, or it can be a virtual machine within the servers of the data center cluster. The sending end, which sends messages to the electronic device, can be any server or server cluster within the data center cluster, or it can be a virtual machine within the servers of the data center cluster. The network interconnection between the sending and receiving ends can be existing protocols such as Infiniband, RoCE, SRD, Falcon, and NVLink, or it can be other network protocols that will be developed in the future.

[0058] In another example, the communication system can also be a communication system under other network architectures such as Ethernet, and the electronic device can also be a terminal device or other device. This application embodiment does not limit this.

[0059] like Figure 2 As shown, based on the above-mentioned electronic device, the data processing method of this application embodiment includes steps 201-204.

[0060] Step 201: Detect the first incomplete message in the first receive queue.

[0061] In this embodiment of the application, the first incomplete message can take many forms.

[0062] For example, in a pre-order connection, the first receive queue may fail to receive the next message that is missing from the current message, thus making the current message the first incomplete message. In a transmission connection with out-of-order reception capability, the message received by the first receive queue may cause a missing message in the current message, i.e., a gap, thus making the current message the first incomplete message.

[0063] The first incomplete message can be a request message, and the corresponding message is a response message. In other words, the response message is a response to the request message.

[0064] In this embodiment of the application, there may be various specific methods for detecting the first incomplete message, and no limitation is made here.

[0065] In one example, timing can be started if a missing message is detected in the message to detect incomplete messages.

[0066] Specifically, in some embodiments, step 201 above includes:

[0067] If the first receiving queue receives the first message and generates an incomplete message based on the first message, then the timer starts.

[0068] If, after the first duration of timing has begun, the incomplete message generated based on the first message is incomplete, then the incomplete message generated based on the first message will be regarded as the first incomplete message.

[0069] The following example illustrates the incomplete message detection method of this application embodiment.

[0070] like Figure 3 In the example shown, the transmission connection of the first receive queue supports out-of-order reception and implements message reordering based on a bitmap algorithm. It can be understood that this bitmap algorithm is an exemplary method for message reordering; in other examples, other methods may be used for recording and retrieving message sequences, which are not limited here.

[0071] In one exemplary scenario, the first receiving queue can receive message A1 via a transmission connection and, based on the bitmap algorithm, record 1 at the corresponding position of sequence number 1 in the sequence list corresponding to the bitmap algorithm, indicating that message with sequence number 1 has been received. After receiving message A1, the first receiving queue receives message A3. At this time, based on the bitmap algorithm, it can record 0 at the corresponding position of sequence number 2 and 1 at the corresponding position of sequence number 3 in the sequence list corresponding to the bitmap algorithm.

[0072] In this way, message reordering can be achieved through the sequence list corresponding to the bitmap algorithm, and a hole is generated in the first receiving queue based on message A3 and the sequence list. Then, based on the field information related to message A1, it can be determined that message A1 is not the end of the message. Therefore, it can be determined that the hole formed by message A1 and message A3 in the first receiving queue is a missing message in the corresponding message. That is to say, it can be determined that an incomplete message is generated based on message A3.

[0073] Similarly, after receiving message A3, the first receiving queue receives message A5. Then, in the sequence list corresponding to the bitmap algorithm, 0 is recorded at the corresponding position of sequence number 4, and 1 is recorded at the corresponding position of sequence number 5.

[0074] If it is determined based on the relevant field information of message A3 that message A3 is not the end of the message, then it can be determined that the gap formed by message A3 and message A5 in the first receiving queue is a missing message in the corresponding message. In other words, it can be determined that an incomplete message was generated based on message A5.

[0075] Taking A3 as an example, message A3 can be taken as the first message. If it is determined that an incomplete message is generated based on the first message, a timer can be started to count down the time. After the first duration of the timer, if message A2 is not received, that is, in the sequence list corresponding to the bitmap algorithm, the record at the corresponding position of sequence number 4 is still 0, then it can be determined that the incomplete message generated based on message A3 is incomplete. Thus, the incomplete message generated based on message A3 is taken as the first incomplete message.

[0076] As can be seen, in this embodiment of the application, when an incomplete message is detected, the incomplete message can be timed at the message level to ensure the reliability of the detected incomplete message and achieve efficient and accurate detection of one or more incomplete messages in the receiving queue.

[0077] In this example, the timing can start every time a message is detected and an incomplete message is generated based on the message, or it can start when a message is detected, an incomplete message is generated based on the message, and it is determined that the generated incomplete message meets the specified conditions. That is to say, the conditions for starting the timing can include, but are not limited to, the first receiving queue receiving the first message and generating an incomplete message based on the first message.

[0078] For example, in some embodiments, if the first receive queue receives a first message and generates an incomplete message based on the first message, then timing begins, including:

[0079] If the first receive queue receives the first message and generates a new incomplete message based on the first message, then the timer starts.

[0080] by Figure 3 For example, when timing an incomplete message generated based on message A3, if the first receiving queue receives message A5 after receiving message A3, and determines that message A3 is not the end of the message, then it can be determined that the gap formed by messages A3 and A5 in the first receiving queue is a missing message in the corresponding message. In other words, it can be determined that an incomplete message was generated based on message A5.

[0081] At this point, based on messages A3 and A5, it can be determined that the incomplete message generated based on message A5 and the incomplete message generated based on message A3 are the same incomplete message A. Therefore, it can be assumed that no new incomplete message was generated based on message A5, and no new timing needs to be started.

[0082] And in Figure 3In the example, if a new incomplete message is generated based on message B4, and this incomplete message is incomplete message B, that is, a new incomplete message, then a timer can be started for the new incomplete message B generated based on message B4. After the first duration of this timer, if the incomplete message generated based on message B4 is incomplete, then the incomplete message generated based on message B4 will be regarded as the first incomplete message.

[0083] It can be seen that, in Figure 3 In the example shown, there can be two incomplete messages: incomplete message A and incomplete message B.

[0084] In this embodiment, timing is only started when a new incomplete message is generated based on the message, which avoids repeatedly timing the same incomplete message and thus reduces resource consumption.

[0085] Step 202: If at least one first incomplete message is detected in the first receive queue, obtain the identifier of each first sender among at least one first sender of at least one first incomplete message.

[0086] In this embodiment of the application, the at least one first incomplete message may be an incomplete message detected from the first receiving queue within a specified time period, or it may be a first incomplete message whose quantity reaches a specified threshold when detected from the first receiving queue.

[0087] After detecting at least one first incomplete message, the identifier of the first sender of the at least one first incomplete message can be obtained.

[0088] The identifier of the first sender can take various forms and be obtained in various ways. For example, the identifier of the first sender may include one or more of the following information carried in the corresponding first incomplete message: the media access control (MAC) address, tuple information from the Transmission Control Protocol (TCP) or Internet Protocol (IP) (e.g., source IP address and source port number), virtual local area network (VLAN), etc., to uniquely identify the first sender. The specific form of the first sender is not limited here; for example, the first sender may be a host or a virtual machine.

[0089] The first senders corresponding to different first incomplete messages can be the same or different. If a certain first sender corresponds to multiple first incomplete messages, when obtaining the identifier of each first sender among at least one first sender of at least one first incomplete message, the identifier of the first sender corresponding to multiple first incomplete messages can be recorded only once, without needing to be recorded repeatedly.

[0090] Step 203: Determine the fault status of each first transmitter based on the identifier of each first transmitter.

[0091] In this embodiment of the application, the fault status of any first transmitter is used to indicate whether the corresponding first transmitter is faulty.

[0092] There are several ways to determine the fault condition of each first transmitter.

[0093] For example, in some cases, it can be determined whether the first sender is faulty based solely on the information from the sender corresponding to the first receive queue. For instance, the first duration can be set to a relatively long duration relative to the information transmission time. In this case, the first incomplete message is a message that has not been fully received by the first receive queue within a longer time period, thus ruling out network congestion and determining that the first sender corresponding to the first incomplete message has failed. Alternatively, if a first sender in the first receive queue corresponds to multiple first incomplete messages, and the reception times of these multiple first incomplete messages in the first receive queue are different, it can be assumed that the first receive queue is continuously receiving messages from the first sender, and the first sender is not faulty; rather, the existence of messages that have not yet been received by the first receive queue may be due to network congestion or other reasons.

[0094] In other examples, it can be determined whether the first sender of the first receive queue is faulty by combining the information of the first sender of the first receive queue with the information of the senders of other incomplete messages.

[0095] Specifically, in some embodiments, step 203 above includes:

[0096] Obtain the identifier of each of the at least one second sender, wherein the at least one second sender is the sender of at least one second incomplete message in the second receive queue;

[0097] The fault condition of each first transmitter is determined based on the identifier of each second transmitter and the identifier of each first transmitter.

[0098] The at least one second incomplete message and the at least one first incomplete message may be detected within the same time period or within different time periods. The detection method for the second incomplete message can refer to the detection method for the first incomplete message, and the identifier of the second sender can refer to the identifier of the first sender, which will not be elaborated further here.

[0099] The number of the second receiving queue can be one or more, and there is no limit to this.

[0100] In this embodiment of the application, the sending end can be comprehensively judged as to whether the sending end is faulty based on the sending end information of incomplete messages in multiple receiving queues.

[0101] For example, if there is an identical sender among the at least one second sender and at least one first sender, it can be considered that multiple receive queues cannot completely receive messages from the identical sender, and the identical sender can be considered to have failed. In other words, the first sender that is identical to any second sender can be regarded as the failed sender.

[0102] If a first sender is different from any second sender, and other receive queues can also receive messages from that first sender, it means that all receive queues except the first receive queue can receive messages from that first sender completely. This indicates that the first sender is not faulty, but rather that the message was not received by the receiver due to network congestion or other issues in the transmission connection corresponding to the first receive queue, resulting in the first incomplete message.

[0103] It is evident that by comprehensively judging whether the sender is faulty based on the sender information of incomplete messages in multiple receiving queues, the situation of incomplete messages caused by network congestion can be effectively ruled out, thus allowing for a more accurate determination of the fault status of the first sender.

[0104] Step 204: Process at least one first incomplete message according to the fault condition of each first sender.

[0105] In this embodiment, the electronic device acting as the receiving end can identify the identifier of the first sender of the first incomplete message based on the first incomplete message in the first receiving queue, thereby actively detecting the fault status of the first sender. In this way, based on the fault status of the first sender, the first incomplete message in the first receiving queue can be actively processed to release the corresponding receiving resources subsequently.

[0106] The fault status of the first sending end can indicate whether the first sending end is a faulty sending end or a normal sending end, and the processing method for the first incomplete message can be different under different fault statuses of the first sending end.

[0107] The following section provides examples of how to handle the first incomplete message from both faulty and normal senders, based on the scenario described above where the sender information from multiple incomplete message queues is used to determine whether the sender is faulty.

[0108] 1. The first sender is a faulty sender.

[0109] Specifically, in some embodiments, step 203 includes:

[0110] The first transmitter that is the same as any second transmitter is designated as the faulty transmitter.

[0111] Step 204 includes:

[0112] Generate a first completion notification for a first incomplete message from a faulty sender in a first receive queue. The first completion notification indicates that the reception of the first incomplete message from the faulty sender has been completed.

[0113] In this embodiment of the application, if a first sending end is the same as a second sending end, it can be considered that multiple receiving queues cannot completely receive the message from the first sending end. Therefore, the first sending end that is the same as the second sending end can be considered as the faulty sending end.

[0114] At this point, it can be assumed that the first receive queue is no longer able to receive the missing messages in the first incomplete message from the faulty sender. Therefore, a first completion notification can be generated regarding the first incomplete message from the faulty sender in the first receive queue, indicating that the reception of the first incomplete message from the faulty sender has been completed. Furthermore, when the first receive queue includes complete messages, a completion notification for the complete message can be reported to the corresponding processes at the receiving end according to the reporting order in the first receive queue. This allows the received resources consumed in the first receive queue (e.g., received resources used to receive messages from the faulty sender) to be reclaimed and not continuously occupied by the first receive queue. In this way, the received resources and the corresponding processes can be easily deregistered later.

[0115] The first completion notification can take various forms. For example, it can carry a first incomplete message from the faulty sender, allowing the first receive queue to indicate that the reception of the incomplete message has been completed by transmitting this message to processes. In some examples, the first completion notification can also indicate that the first incomplete message from the faulty sender is an incomplete message. This allows the corresponding process receiving the incomplete message at the receiving end to determine that it is indeed an incomplete message, facilitating subsequent processing, such as deletion or data processing.

[0116] In addition, in some examples, the first receiving queue may include the received complete message. After the complete message is received, a completion notification for the complete message may be generated, and the completion notification for the complete message may be reported to the corresponding process of the receiving end according to the reporting order in the first receiving queue.

[0117] In this example, the reporting process for the first incomplete message and the complete message in the first receive queue can be the same, allowing the receive resources consumed by the first receive queue to be reclaimed and avoiding waste. When reporting the first incomplete message to the corresponding process, a first type of reporting can be used to indicate that the first incomplete message is incomplete, while when reporting the complete message to the corresponding process, a second type of reporting can be used to indicate that it is a complete message.

[0118] 2. The first sender is a normal sender.

[0119] Specifically, in some embodiments, step 203 includes:

[0120] The first sender, which is different from each of the second senders, is taken as the normal sender;

[0121] Step 204 includes:

[0122] Waiting for the missing message in the first incomplete message from the normal sender in the first receive queue;

[0123] If the first incomplete message from the normal sender is incomplete after the waiting time reaches the second time limit, a second completion notification is generated regarding the first incomplete message from the normal sender. The second completion notification is used to indicate that the first incomplete message from the normal sender has been successfully received.

[0124] If a first sender is different from any second sender, and other receive queues can also receive messages from that first sender, it means that all receive queues except the first receive queue can receive messages from that first sender completely. This indicates that the first sender is not faulty, but rather that the message was not received by the receiver due to network congestion or other issues in the transmission connection corresponding to the first receive queue, resulting in the first incomplete message.

[0125] Therefore, in this embodiment of the application, a second duration can be waited for messages that have not been received by the receiving end due to network congestion or other reasons.

[0126] If a missing message from the first incomplete message from the normal sender is obtained within the second waiting period, the complete message from the normal sender can be obtained, and the complete message can be reported to the corresponding process at the receiving end.

[0127] If, after the second waiting period, the first incomplete message from the normal sender is still incomplete, a second completion notification is generated regarding the first incomplete message from the normal sender. This second completion notification indicates that the first incomplete message from the normal sender has been successfully received, allowing the corresponding receiving resources to be reclaimed instead of being continuously occupied by the first receiving queue. This allows for convenient subsequent cancellation operations on the receiving resources and the corresponding processes.

[0128] As can be seen, in this embodiment, the receiving end can identify the identifier of the first sender of the first incomplete message based on the first incomplete message in the first receiving queue, thereby actively detecting the fault status of the first sender. Then, based on the fault status of the first sender, it can actively process the first incomplete message in the first receiving queue, so that the receiving task corresponding to the first incomplete message in the first receiving queue can be completed, thereby releasing the corresponding receiving resources in the future.

[0129] As can be seen, the embodiments of this application can effectively solve the problem that the receiving task in the receiving queue cannot be completed due to the failure of the sending end, resulting in the inability to reclaim receiving resources. In addition, in a scenario where a single transmission connection can realize message transmission between multiple virtual machines or processes on the sending side and multiple processes or virtual machines on the receiving side, it can also avoid the problem of head-of-line blocking of normal services of other virtual machines on the sending side due to the failure of one virtual machine on the sending side.

[0130] For example, in Figure 4 In the example shown, the sending side Host0 includes virtual machines VM00 and VM01. Virtual machines VM00 and VM01 can act as different sending ends, uniquely identified by different identifiers (such as MAC addresses, TCP / IP protocol tuples, etc.).

[0131] The virtual machines VM00 and VM01 can send messages to the receiving queue TP10 of the receiving end Host1 through the sending queue TP00 of the transmission connection, and then pass the messages to the receiving virtual machine VM10 through the receiving queue TP10.

[0132] After adopting any of the above embodiments of this application, even if the virtual machine VM00 fails, the incomplete messages from the virtual machine VM00 and the complete messages from the virtual machine VM01 in the receive queue TP10 of the receiving end Host1 can be reported to the virtual machine VM10, so that the receive resources consumed by the receive queue TP10 from the virtual machine VM10 can be reclaimed. In addition, the complete messages transmitted by the sending side virtual machine VM01 through the receive queue TP10 will not be blocked in the receive queue TP10 by incomplete messages that cannot be processed, so that the message transmission of the normal business of the virtual machine VM01 is not affected by the failure of the virtual machine VM00.

[0133] The following specific example illustrates a particular implementation of an embodiment of this application.

[0134] like Figure 5 As shown in this example, the message receiving operation of the receive queue may include one or more of the following steps:

[0135] Message input, protocol verification, message discarding, message reordering, timing, sender fault diagnosis, receiver resource processing, payload writing, and generation of completion notification.

[0136] The following is about Figure 5 The steps shown are illustrated by way of example.

[0137] 1. Message Input

[0138] An electronic device acting as a receiver can receive messages from a sender via a transmission connection. These messages can include a header and a payload, and these two parts can be buffered independently.

[0139] 2. Protocol Verification

[0140] After receiving a message, the message can be verified using relevant protocols (such as transport layer protocols).

[0141] 3. Message discarding

[0142] If the protocol verification fails, the message can be discarded.

[0143] 4. Message reordering

[0144] If the protocol verification passes, the message can be further processed. In this example, out-of-order message transmission is supported, so message reordering can be performed using a bitmap.

[0145] After message reordering, parsing can be performed based on the bitmap. The parsing criteria include: whether the message is the first packet of the message, whether the parsing based on the bitmap results in an incomplete message, and whether the parsing based on the bitmap contains a complete message.

[0146] 5. Timing

[0147] After message reordering, if parsing based on the bitmap determines that an incomplete message was generated based on the message, then a timer is used to keep track of the time.

[0148] 6. Transmitter fault diagnosis

[0149] If the incomplete message is still incomplete after the first timeout period, a timeout is determined, and the timeout message (such as the number of the relevant transmission queue) is sent to the sender fault diagnosis module. The sender fault diagnosis module retrieves the identifier of the sender corresponding to each incomplete message based on the bitmap. When it is determined that the sender of incomplete messages in multiple receive queues has the same sender based on the identifier, the same sender is determined to be the faulty sender.

[0150] 7. Receiving and processing resources

[0151] If the parsed message is the first packet of a message, the receive queue can request receive resources and perform protocol verification and parsing related to the receive resources.

[0152] 8. Payload writing

[0153] After parsing the message as the first packet of the message, and after requesting receiving resources and performing related protocol verification and parsing, the payload of the message can be written into double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0154] 9. Generation complete notification

[0155] If multiple receive queues are found to have incomplete messages originating from the same sender, that same sender is identified as a faulty sender. The corresponding process at the receiving end can report both incomplete and complete messages from that same sender, causing the relevant receive queue to generate a completion notification indicating that the receiving task is complete.

[0156] The above steps can be applied to various system architectures.

[0157] For example, in Figure 6In the system architecture shown, after a message enters the transport layer, it can undergo processes such as protocol verification, message reordering, timing, sender fault diagnosis, receive resource processing, payload writing, and generation of completion notifications. Furthermore, in this system architecture, receive resources can reside outside the transport layer, for example... Figure 6 In the example, the transport layer's receive queue can request shared receive resources from the transaction layer. In other examples, this receive resource may also reside in the transport layer; for example, the receive resource could be an SRQ, etc.

[0158] The data processing method provided by the embodiments of this application has been described above from multiple aspects. The data processing apparatus provided by the embodiments of this application will now be described in conjunction with the accompanying drawings.

[0159] like Figure 7 As shown, this application embodiment provides a data processing apparatus 70, which includes:

[0160] Detection module 701 is used to: detect the first incomplete message in the first receive queue;

[0161] Processing module 702 is used for:

[0162] If at least one first incomplete message is detected in the first receive queue, obtain the identifier of each first sender among at least one first sender of at least one first incomplete message;

[0163] Based on the identifier of each first transmitter, determine the fault condition of each first transmitter;

[0164] At least one first incomplete message is processed based on the failure status of each first sender.

[0165] Optionally, the processing module 702 is used for:

[0166] If the first receiving queue receives the first message and generates an incomplete message based on the first message, then the timer starts.

[0167] If, after the first duration of timing has begun, the incomplete message generated based on the first message is incomplete, then the incomplete message generated based on the first message will be regarded as the first incomplete message.

[0168] Optionally, the processing module 702 is used for:

[0169] If the first receive queue receives the first message and generates a new incomplete message based on the first message, then the timer starts.

[0170] Optionally, the processing module 702 is used for:

[0171] Obtain the identifier of each of the at least one second sender, wherein the at least one second sender is the sender of at least one second incomplete message in the second receive queue;

[0172] The fault condition of each first transmitter is determined based on the identifier of each second transmitter and the identifier of each first transmitter.

[0173] Optionally, the processing module 702 is used for:

[0174] The first transmitter that is the same as any second transmitter is designated as the faulty transmitter.

[0175] Generate a first completion notification for a first incomplete message from a faulty sender in a first receive queue. The first completion notification indicates that the reception of the first incomplete message from the faulty sender has been completed.

[0176] Optionally, the first completion notification indicates that the first incomplete message from the fault sender is an incomplete message.

[0177] Optionally, the processing module 702 is used for:

[0178] The first sender, which is different from each of the second senders, is taken as the normal sender;

[0179] Waiting for the missing message in the first incomplete message from the normal sender in the first receive queue;

[0180] If the first incomplete message from the normal sender is incomplete after the waiting time reaches the second time limit, a second completion notification is generated regarding the first incomplete message from the normal sender. The second completion notification is used to indicate that the first incomplete message from the normal sender has been successfully received.

[0181] Figure 8 The diagram shown is a possible logical structure schematic of an electronic device 80 provided in an embodiment of this application. This electronic device 80 is used to implement the functions of the electronic devices involved in any of the above embodiments. The electronic device 80 includes: a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, processor 802, and communication interface 803 are interconnected via the bus 804.

[0182] The memory 801 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 801 may store a program, and when the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to perform one or more steps in the above-described data processing method embodiments.

[0183] Processor 802 can be a central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), digital signal processing (DSP), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any combination thereof, to execute relevant programs to achieve the functions required by the detection module and processing module in the data processing device of the above embodiments, or to execute one or more steps in the method embodiments of this application. The steps of the method disclosed in the embodiments of this application can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 801. Processor 802 reads information from memory 801 and, in conjunction with its hardware, executes one or more steps in the data processing method embodiments described above.

[0184] The communication interface 803 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the electronic device 80 and other devices or communication networks.

[0185] Bus 804 enables the transmission of information between various components of electronic device 80 (e.g., memory 801, processor 802, and communication interface 803). Bus 804 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation,Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0186] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figure 8 The steps performed by the processor in the process.

[0187] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium; when the processor of the device executes the computer-executable instructions, the device performs the above-described... Figure 8 The steps performed by the processor in the process.

[0188] In another embodiment of this application, a chip system is also provided, the chip system including a processor for implementing the above. Figure 8 The steps performed by the processor. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for data writing. The chip system may consist of chips or may include chips and other discrete components.

[0189] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0191] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data processing method, characterized in that, The method includes: Detect the first incomplete message in the first receive queue; If at least one first incomplete message is detected in the first receive queue, obtain the identifier of each of the at least one first senders of the at least one first incomplete message; Based on the identifier of each first transmitter, determine the fault condition of each first transmitter; The at least one first incomplete message is processed according to the fault condition of each of the first senders.

2. The method according to claim 1, characterized in that, The detection of the first incomplete message in the first receiving queue includes: If the first receiving queue receives the first message and generates an incomplete message based on the first message, then the timer starts. If, after the first duration of timing has begun, the incomplete message generated based on the first message is incomplete, then the incomplete message generated based on the first message will be used as the first incomplete message.

3. The method according to claim 2, characterized in that, The step of starting a timer if the first receiving queue receives a first message and generates an incomplete message based on the first message includes: If the first receiving queue receives the first message and generates a new incomplete message based on the first message, then the timer starts.

4. The method according to any one of claims 1-3, characterized in that, The step of determining the fault status of each first transmitter based on its identifier includes: Obtain the identifier of each of the at least one second sender, wherein the at least one second sender is the sender of at least one second incomplete message in the second receive queue; The fault condition of each first transmitter is determined based on the identifier of each second transmitter and the identifier of each first transmitter.

5. The method according to claim 4, characterized in that, The step of determining the fault status of each first transmitter based on the identifier of each second transmitter and the identifier of each first transmitter includes: The first transmitter that is the same as any of the second transmitters is designated as the faulty transmitter. The step of processing the at least one first incomplete message according to the fault condition of each first sender includes: Generate a first completion notification for a first incomplete message from the faulty sender in the first receive queue, the first completion notification indicating that the reception of the first incomplete message from the faulty sender has been completed.

6. The method according to claim 5, characterized in that, The first completion notification indicates that the first incomplete message from the fault sender is an incomplete message.

7. The method according to claim 4, characterized in that, The step of determining the fault status of each first transmitter based on the identifier of each second transmitter and the identifier of each first transmitter includes: The first sender, which is different from each of the second senders, is taken as the normal sender; The step of processing the at least one first incomplete message according to the fault condition of each first sender includes: Waiting for the missing message in the first incomplete message from the normal sending end in the first receiving queue; If the first incomplete message from the normal sending end is incomplete after the waiting time reaches the second time limit, a second completion notification is generated regarding the first incomplete message from the normal sending end. The second completion notification is used to indicate that the first incomplete message from the normal sending end has been successfully received.

8. A data processing apparatus, characterized in that, include: The detection module is used to: detect the first incomplete message in the first receive queue; Processing module, used for: If at least one first incomplete message is detected in the first receive queue, obtain the identifier of each of the at least one first senders of the at least one first incomplete message; Based on the identifier of each first transmitter, determine the fault condition of each first transmitter; The at least one first incomplete message is processed according to the fault condition of each of the first senders.

9. The apparatus according to claim 8, characterized in that, The processing module is used for: If the first receiving queue receives the first message and generates an incomplete message based on the first message, then the timer starts. If, after the first duration of timing has begun, the incomplete message generated based on the first message is incomplete, then the incomplete message generated based on the first message will be used as the first incomplete message.

10. The apparatus according to claim 9, characterized in that, The processing module is used for: If the first receiving queue receives the first message and generates a new incomplete message based on the first message, then the timer starts.

11. The apparatus according to any one of claims 8-10, characterized in that, The processing module is used for: Obtain the identifier of each of the at least one second sender, wherein the at least one second sender is the sender of at least one second incomplete message in the second receive queue; The fault condition of each first transmitter is determined based on the identifier of each second transmitter and the identifier of each first transmitter.

12. The apparatus according to claim 11, characterized in that, The processing module is used for: The first transmitter that is the same as any of the second transmitters is designated as the faulty transmitter. Generate a first completion notification for a first incomplete message from the faulty sender in the first receive queue, the first completion notification indicating that the reception of the first incomplete message from the faulty sender has been completed.

13. The apparatus according to claim 12, characterized in that, The first completion notification indicates that the first incomplete message from the fault sender is an incomplete message.

14. The apparatus according to claim 11, characterized in that, The processing module is used for: The first sender, which is different from each of the second senders, is taken as the normal sender; Waiting for the missing message in the first incomplete message from the normal sending end in the first receiving queue; If the first incomplete message from the normal sending end is incomplete after the waiting time reaches the second time limit, a second completion notification is generated regarding the first incomplete message from the normal sending end. The second completion notification is used to indicate that the first incomplete message from the normal sending end has been successfully received.

15. An electronic device, characterized in that, The electronic device includes a processor, a memory, and instructions stored in the memory and executable by the at least one processor, which, when executed by the processor, implement the method according to any one of claims 1-7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

17. A computer program product containing instructions, characterized in that, When the instructions are executed by the processor, the method described in any one of claims 1-7 is implemented.

Images