Data processing method and related equipment

Abstract

The embodiment of the invention discloses a data processing method, and a receiving end can identify an identifier of a first sending end of a first incomplete message according to the first incomplete message in a first receiving queue, thereby actively detecting a fault condition of the first sending end. And then, active processing of the first incomplete message in the first receiving queue can be realized according to the fault condition of the first sending end, so that a receiving task corresponding to the first incomplete message in the first receiving queue can be ended, and corresponding receiving resources can be released subsequently.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a data processing method and related equipment. Background Art

[0002] With the rapid development of applications such as high-performance computing, artificial intelligence, and big data, data transmission requirements and network interconnection scenarios between devices are becoming increasingly complex. In some communication methods, to facilitate the reception of request messages from one or more senders, the receive queue at the receiving end can request receive resources from multiple processes on the receiving end to receive request messages from the sender.

[0003] However, during the data transmission process, the sending end may fail and be unable to send a complete request message. However, since the failure message of the sending end cannot be notified to the receiving end, the receiving resources in the receiving queue of the receiving end used to receive the request message cannot complete the receiving task, and the receiving resources cannot be recovered, resulting in the receiving resources of the corresponding process of the receiving end being wasted. Summary of the Invention

[0004] The present application provides a data processing method that can address the problem of wasted receiving resources in a receiving queue at a receiving end, where the receiving resources cannot be recovered due to a failure at the sending end. The present application also provides corresponding apparatus, devices, computer-readable storage media, and computer program products.

[0005] The first aspect of the present application provides a data processing method, which includes: detecting a first incomplete message in a first receiving queue; when at least one first incomplete message is detected in the first receiving queue, obtaining an identifier of each first sending end of at least one first sending end of the at least one first incomplete message; determining a fault condition of each first sending end based on the identifier of each first sending end; and processing at least one first incomplete message based on the fault condition of each first sending end.

[0006] In a first aspect, a receiving end can identify the identifier of the first sender of the first incomplete message based on the first incomplete message in the first receiving queue, thereby proactively detecting a fault condition of the first sender. Then, based on the fault condition of the first sender, the receiving end can proactively process the first incomplete message in the first receiving queue, thereby completing the receiving task corresponding to the first incomplete message in the first receiving queue and subsequently releasing corresponding receiving resources.

[0007] In a possible implementation of the first aspect, detecting a first incomplete message in a first receiving queue includes: if the first receiving queue receives a first message and generates an incomplete message based on the first message, starting timing; after a first time period starts, if the incomplete message generated based on the first message is incomplete, treating the incomplete message generated based on the first message as the first incomplete message.

[0008] In this possible implementation, when an incomplete message is detected, the incomplete message can be timed at the message level to ensure the reliability of the detected incomplete message and achieve efficient and accurate detection of one or more incomplete messages in the receiving queue.

[0009] In a possible implementation of the first aspect, if the first receiving queue receives the first message and generates an incomplete message based on the first message, the timing is started, including: if the first receiving queue receives the first message and generates a new incomplete message based on the first message, the timing is started.

[0010] In this possible implementation, timing is performed only when it is detected that a new incomplete message is generated based on the message, which can avoid repeated timing of the same incomplete message, thereby reducing resource consumption.

[0011] In a possible implementation of the first aspect, the fault condition of each first sending end is determined based on the identifier of each first sending end, including: obtaining the identifier of each second sending end in at least one second sending end, at least one second sending end being the sender of at least one second incomplete message of the second receiving queue; determining the fault condition of each first sending end based on the identifier of each second sending end and the identifier of each first sending end.

[0012] In this possible implementation, based on the sender information of incomplete messages in multiple receiving queues, a comprehensive judgment is made as to whether the sender is faulty, which can better eliminate situations such as incomplete messages caused by network congestion, thereby more accurately judging the fault condition of the first sender.

[0013] In a possible implementation of the first aspect, the fault condition of each first transmitting end is determined based on the identifier of each second transmitting end and the identifier of each first transmitting end, including: treating the first transmitting end that is the same as any second transmitting end as the faulty transmitting end; and processing at least one first incomplete message based on the fault condition of each first transmitting end, including: generating a first completion notification regarding the first incomplete message from the faulty transmitting end in the first receiving queue, the first completion notification being used to indicate that the first incomplete message from the faulty transmitting end has been completed.

[0014] In this possible implementation, the failure condition of the first transmitter may indicate that the first transmitter is a faulty transmitter or a normal transmitter, and different failure conditions of the first transmitter may result in different processing of the first incomplete message. If a first transmitter is identical to a second transmitter, it may be considered that multiple receive queues are unable to completely receive messages from the first transmitter, and therefore, the first transmitter identical to the second transmitter may be considered a faulty transmitter.

[0015] In this way, a first completion notification of the first incomplete message from the faulty sending end in the first receiving queue can be generated to indicate that the first incomplete message from the faulty sending end has been received, so that the receiving resources in the first receiving queue used to receive messages from the faulty sending end can be recycled and will not be occupied by the first receiving queue all the time, so that the receiving resources and the corresponding processes can be conveniently logged out and other operations can be performed later.

[0016] In a possible implementation manner of the first aspect, the first completion notification indicates that the first incomplete message from the faulty sending end is an incomplete message.

[0017] In this possible implementation, the corresponding process receiving the first incomplete message in the receiving end can determine that the first incomplete message is an incomplete message, so as to facilitate subsequent processing, such as deletion or data processing of the first incomplete message.

[0018] In a possible implementation of the first aspect, the fault condition of each first transmitting end is determined based on the identifier of each second transmitting end and the identifier of each first transmitting end, including: treating a first transmitting end that is different from each second transmitting end as a normal transmitting end; processing at least one first incomplete message according to the fault condition of each first transmitting end, including: waiting for the missing message in the first incomplete message from the normal transmitting end in the first receiving queue; if the first incomplete message from the normal transmitting end is incomplete after the waiting time reaches a second time, generating a second completion notification about the first incomplete message from the normal transmitting end, the second completion notification is used to indicate that the first incomplete message from the normal transmitting end has been received.

[0019] In this possible implementation, if a first sending end is different from any second sending end, and other receiving queues can also receive messages from the first sending end, it means that except for the first receiving queue, other receiving queues can completely receive the message from the first sending end, which means that the first sending end is not faulty, but the message is not received by the receiving end due to network congestion or other conditions in the transmission connection corresponding to the first receiving queue, resulting in the first incomplete message. Therefore, in this possible implementation, you can continue to wait for the second time period to wait for the message that has not been received by the receiving end due to network congestion or other conditions. If the first incomplete message from the normal sending end is still incomplete after the waiting time reaches the second time period, the corresponding receiving resources can be recovered through the second completion notification, and will not be occupied by the first receiving queue.

[0020] A second aspect of the present application provides a data processing device that has the functionality to implement the method of the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functionality, such as a detection module and a processing module.

[0021] The third aspect of the present application provides an electronic device, which includes at least one processor, a memory, and computer-executable instructions stored in the memory and executable by the processor. When the computer-executable instructions are executed by the processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0022] The fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0023] The fifth aspect of the present application provides a computer program product that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0024] A sixth aspect of the present application provides a chip system, which includes a processor for supporting an electronic device in implementing the functions involved in the first aspect or any possible implementation of the first aspect. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the electronic device. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0025] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is an exemplary schematic diagram of the XRC technology provided in an embodiment of the present application;

[0027] Figure 2 This is an exemplary schematic diagram of the data processing method provided in an embodiment of the present application;

[0028] Figure 3 This is an exemplary schematic diagram of detecting incomplete messages based on a bitmap algorithm provided in an embodiment of the present application;

[0029] Figure 4 This is an exemplary schematic diagram of the system architecture provided by the embodiment of the present application;

[0030] Figure 5 This is an exemplary schematic diagram of the processing flow provided in the embodiment of the present application;

[0031] Figure 6 This is an exemplary schematic diagram of the system architecture provided by the embodiment of the present application;

[0032] Figure 7 1 is a schematic diagram of an embodiment of a data processing device provided in an embodiment of the present application;

[0033] Figure 8 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0035] Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0036] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate. This is merely a way of distinguishing objects with the same properties when describing them in the embodiments of this application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0037] With the rapid development of application scenarios such as high-performance computing, artificial intelligence, and big data, the data transmission requirements between devices and network interconnection scenarios are becoming increasingly complex.

[0038] For example, in an exemplary scenario, in order to meet the requirements of various application scenarios, high-performance data processing capabilities can be provided in the form of computing device clusters such as data center clusters.

[0039] As the scale of clusters, such as data center clusters, continues to grow, the network performance of these clusters will have a significant impact on their overall performance. To ensure high-bandwidth, low-latency network performance while reducing resource consumption, various developers currently offer a variety of network interconnection protocols, such as Infiniband, Remote Direct Memory Access over Converged Ethernet (RoCE), Scalable Reliable Datagram (SRD), Falcon, and Nvlink.

[0040] In actual application scenarios, the network interconnection between computing devices in computing device clusters such as data center clusters is relatively complex, and the possible abnormal scenarios are also relatively diverse. For example, network packet loss, network timeout, network failure, node failure, etc. may occur.

[0041] Among them, when a failure occurs at the sending end, the failure information of the sending end cannot be notified to the receiving end, so the receiving resources in the receiving queue of the receiving end used to receive the request message cannot complete the receiving task and cannot be recovered, resulting in the receiving resources of the corresponding process of the receiving end being wasted due to the inability to cancel.

[0042] The following takes the extended reliable connection (XRC) communication model in the Infiniband protocol as an example for introduction.

[0043] The communication entities of the Infiniband protocol XRC technology are the XRC initiate queue pair (INIQP), the XRC target queue pair (TGT QP) and the XRC shared receive queue (SRQ), which are subsequently referred to as INI TP, TGT QP and SRQ respectively.

[0044] like Figure 1 As shown, the sending host Host0 includes processes process0 and process1, and the receiving host (also called target) HOST1 includes processes process0 and process1. A connection is established between INIQP and TGT QP, and an INIQP is only connected to one TGT QP. For example, Figure 1 In the example shown, XRC INIQP000 is connected to XRC TGT QP 10, and XRC INIQP010 is connected to XRC TGT QP 11. TGT QP has no ability to send requests. It is fixed that INIQP sends request messages and TGT receives request messages. TGT obtains receiving resources from SRQ to receive request messages. TGT QP can point to multiple SRQs and obtain receiving resources from different SRQs. For example, Figure 1 In the example shown, XRC TGT QP 10 can obtain receive resources from XRCSRQ 100 and XRC SRQ 110 respectively; multiple TGT QPs can point to the same SRQ and consume the receive resources of the same SRQ, for example, Figure 1 In the example shown, both XRC TGT QP 10 and XRC TGT QP 11 can obtain reception resources from XRC SRQ 100. As can be seen, SRQ can be considered as a shared reception resource for TGT QPs.

[0045] And in Figure 1In the example shown, when the sending host Host0 fails, the TGTQP of the receiving host HOST1 may receive an incomplete request message. However, because the sending host Host0 has failed, the missing packets in the incomplete request message cannot be retransmitted. As a result, the TGTQP of the receiving host HOST1 cannot receive the missing packets. As a result, the consumed receive resources in the TGTQP cannot be recovered, resulting in wasted receive resources. Furthermore, if this situation continues to accumulate, there is a risk that the SRQ receive resources will be exhausted and unrecoverable.

[0046] Furthermore, if the message receiving task in the TGT QP is not completed, the service on the receiving end will remain in the running state. Currently, when the service is in the running state, the user cannot issue the SRQ deregistration command, resulting in the user being unable to deregister the abnormal SRQ and release resources in real time. This makes deregistration of abnormal SRQs inflexible and reduces user operational flexibility.

[0047] Alternatively, users can issue SRQ deregistration commands while services are running. However, because the Infiniband protocol doesn't define how to report all SRQ receive completions, users must configure SRQ resource deregistration to prevent them from relying on the recovery of all receive resources before releasing them. However, this can result in hardware-received data remaining in unreclaimed receive resources, and subsequent reporting of receive completions can affect subsequent SRQ reuse.

[0048] It can be seen that in the current network interconnection scenario based on the Infiniband protocol, a failure at the sending end will make it impossible to reclaim the receiving resources used to receive request messages in the receiving queue of the receiving end, resulting in the waste of the receiving resources at the receiving end, thereby affecting the cancellation of the SRQ, making the abnormal SRQ cancellation at the receiving end inflexible, and affecting the subsequent use of the receiving resources.

[0049] It is understandable that the above examples of the Infiniband protocol are only exemplary network interconnection scenarios and are not limiting. The above problems also exist in many other network interconnection scenarios.

[0050] For example, in some exemplary network interconnection scenarios, multiple transmission connections can be established between a sending host and a receiving host. One transmission connection can correspond to a sending queue at the sending end and a receiving queue at the receiving end. The sending queue at the sending end can be connected to multiple virtual machines or multiple processes, allowing multiple virtual machines or multiple processes to share the transmission connection, that is, all of them can send request messages to the receiving end through the transmission connection. In addition, the receiving queue at the receiving end of the transmission connection can be connected to multiple processes or multiple virtual machines at the receiving end, allowing request messages from the sending end to be transmitted to multiple processes or multiple virtual machines at the receiving end. In other words, in this exemplary scenario, one transmission connection can implement message transmission between multiple virtual machines or multiple processes at the sending end and multiple processes or multiple virtual machines at the receiving end.

[0051] In this exemplary scenario, if a process or virtual machine on the sending end fails, the failed process or virtual machine will be unable to send a complete request message, making it impossible to complete the receiving task in the receiving queue of the receiving end, thereby causing the related process of the receiving task to be unable to be deregistered, and the corresponding receiving resources to be unable to be recovered. In addition, other normal businesses on the sending end will suffer from head-of-line blocking and other problems, affecting the normal execution of other normal businesses that transmit messages through the receiving queue.

[0052] It can be seen that in various current network interconnection scenarios, during the data transmission process, the sending end may fail and fail to send a complete request message. However, since the failure message of the sending end cannot be notified to the receiving end, the receiving resources in the receiving queue of the receiving end used to receive the request message cannot complete the receiving task and cannot be recovered, resulting in the receiving resources of the corresponding process of the receiving end being wasted due to the inability to cancel.

[0053] To address the above-mentioned problem, an embodiment of the present application provides a data processing method, in which the receiving end can actively detect the fault condition of the sending end, thereby enabling active processing of the first incomplete message in the first receiving queue based on the fault condition of the first sending end, so as to subsequently release the corresponding receiving resources, avoid waste of receiving resources, and facilitate flexible cancellation of receiving resources and other operations.

[0054] The data processing method of the embodiment of the present application can be applied to an electronic device, which can be a terminal device, a single server or a server cluster, or a virtual machine (VM) or a container (Docker).

[0055] The electronic device can serve as a receiving end in a communication system to receive messages from a sending end.

[0056] The specific architecture of the communication system and the network interconnection method adopted are not limited here.

[0057] For example, in one example, the communication system may include a data center cluster, and the electronic device serving as the receiving end may be any server or server cluster in the data center cluster, or the electronic device may be a virtual machine in a server in the data center cluster. The sending end that sends a message to the electronic device may be any server or server cluster in the data center cluster, or a virtual machine in a server in the data center cluster. The network interconnection between the sending and receiving ends may be existing protocols such as Infiniband, RoCE, SRD, Falcon, and Nvlink, or other network protocols developed subsequently.

[0058] In another example, the communication system may also be a communication system under other network architectures such as Ethernet, and the electronic device may also be a terminal device or other device form, which is not limited in the embodiments of the present application.

[0059] like Figure 2 As shown, based on the above electronic device, the data processing method of the embodiment of the present application includes steps 201-204.

[0060] Step 201: Detect a first incomplete message in a first receiving queue.

[0061] In the embodiment of the present application, the specific form of the first incomplete message can be various.

[0062] For example, for an order-preserving connection, the first receive queue may fail to receive the next packet missing from the current message, resulting in the current message being the first incomplete message. For a transport connection with out-of-order reception capability, the first receive queue may receive a packet that causes a missing packet in the current message, i.e., a hole, resulting in the current message being the first incomplete message.

[0063] The first incomplete message may be a request message, and the response message corresponds to the request message, that is, the response message is a response to the request message.

[0064] In the embodiment of the present application, there may be many specific ways to detect the first incomplete message, which are not limited here.

[0065] In one example, timing may be performed upon detecting missing packets in a message to detect an incomplete message.

[0066] Specifically, in some embodiments, step 201 includes:

[0067] If the first receiving queue receives the first message and generates an incomplete message based on the first message, the timing starts;

[0068] After the first time period starts, if the incomplete message generated based on the first message is incomplete, the incomplete message generated based on the first message is used as the first incomplete message.

[0069] The following is an example to illustrate the incomplete message detection method of the embodiment of the present application.

[0070] like Figure 3 In the example shown, the transport connection of the first receive queue supports out-of-order reception and implements message reordering based on a bitmap algorithm. It is understood that the bitmap algorithm is only one exemplary method for message reordering. In other examples, other methods can also be used to record and retrieve message sequences, which are not limited here.

[0071] In an exemplary scenario, the first receive queue may receive message A1 through a transmission connection and, based on a bitmap algorithm, record 1 at the position corresponding to sequence number 1 in a sequence list corresponding to the bitmap algorithm, to indicate that the message with sequence number 1 has been received. After receiving message A1, the first receive queue receives message A3. At this time, based on the bitmap algorithm, 0 may be recorded at the position corresponding to sequence number 2 and 1 may be recorded at the position corresponding to sequence number 3 in the sequence list corresponding to the bitmap algorithm.

[0072] In this way, the message reordering can be achieved through the sequence list corresponding to the bitmap algorithm, and a hole can be generated in the first receiving queue based on message A3 and the sequence list. Then, based on the field information related to message A1, it can be determined that message A1 is not the end of the message. Then, it can be determined that the hole formed by messages A1 and A3 in the first receiving queue is the missing message in the corresponding message. In other words, it can be determined that an incomplete message is generated based on message A3.

[0073] Similarly, after receiving message A3, the first receiving queue receives message A5. Then, in the sequence list corresponding to the bitmap algorithm, 0 can be recorded at the corresponding position of sequence number 4, and 1 can be recorded at the corresponding position of sequence number 5.

[0074] If it is determined based on the relevant field information of message A3 that message A3 is not the end of the message, then it can be determined that the hole formed by message A3 and message A5 in the first receiving queue is the message missing in the corresponding message, that is, it can be determined that an incomplete message is generated based on message A5.

[0075] Taking A3 as an example, message A3 can be taken as the first message, and it is determined that an incomplete message is generated based on the first message. Then, timing can be started through a timer, and after the first period of time is started, it is determined that message A2 is not received, that is, in the sequence list corresponding to the bitmap algorithm, the corresponding position record of sequence number 4 is still 0, then it can be determined that the incomplete message generated based on message A3 is incomplete, and the incomplete message generated based on message A3 is the first incomplete message.

[0076] It can be seen that in the embodiment of the present application, when an incomplete message is detected, the incomplete message can be timed at the message level to ensure the reliability of the detected incomplete message and achieve efficient and accurate detection of one or more incomplete messages in the receiving queue.

[0077] In this example, the timing may start each time a message is detected and an incomplete message is generated based on the message, or the timing may start when a message is detected and an incomplete message is generated based on the message, and the timing starts only when it is determined that the generated incomplete message meets specified conditions. That is, the conditions for starting the timing may include but are not limited to the first receiving queue receiving the first message and generating an incomplete message based on the first message.

[0078] For example, in some embodiments, if the first receiving queue receives the first message and generates an incomplete message based on the first message, then timing is started, including:

[0079] If the first receiving queue receives the first message and generates a new incomplete message based on the first message, the timer starts.

[0080] by Figure 3 For example, when timing an incomplete message generated based on message A3, the first receiving queue receives message A5 after receiving message A3, and determines that message A3 is not the end of the message. Then, it can be determined that the hole formed by messages A3 and A5 in the first receiving queue is the missing message in the corresponding message, that is, it can be determined that an incomplete message is generated based on message A5.

[0081] At this time, based on messages A3 and A5, it can be determined that the incomplete message generated based on message A5 and the incomplete message generated based on message A3 are the same incomplete message A. Then it can be considered that no new incomplete message is generated based on message A5, and there is no need to start a new timing.

[0082] And in Figure 3In the example, if a new incomplete message is generated based on message B4, and the incomplete message is an incomplete message B, that is, a new incomplete message, then the timing of the new incomplete message B generated based on message B4 can be started, and after the first period of the timing, if the incomplete message generated based on message B4 is incomplete, the incomplete message generated based on message B4 will be used as the first incomplete message.

[0083] It can be seen that in Figure 3 In the example shown, there may be two first incomplete messages, namely incomplete message A and incomplete message B.

[0084] In this embodiment, timing is performed only when it is detected that a new incomplete message is generated based on the message, thereby avoiding repeated timing of the same incomplete message, thereby reducing resource consumption.

[0085] Step 202: When at least one first incomplete message is detected in the first receiving queue, obtain an identifier of each first sending end of at least one first sending end of the at least one first incomplete message.

[0086] In an embodiment of the present application, the at least one first incomplete message may be an incomplete message detected from the first receiving queue within a specified time period, or, when the number of first incomplete messages reaching a specified threshold is detected from the first receiving queue, the first incomplete messages reaching the specified threshold are used as the at least one first incomplete message.

[0087] After detecting the at least one first incomplete message, an identifier of a first sender of the at least one first incomplete message may be obtained.

[0088] The identifier of the first sending end can have multiple forms and acquisition methods. Exemplarily, the identifier of the first sending end can include one or more of the following information: media access control (MAC) address, multi-tuple information in the Transmission Control Protocol (TCP) or Internet Protocol (IP) (such as source IP address and source port number), virtual local area network (VLAN), etc., carried in the message of the corresponding first incomplete message, so as to uniquely identify the first sending end. The specific form of the first sending end is not limited here. For example, the first sending end can be a host or a virtual machine.

[0089] The first senders corresponding to different first incomplete messages may be the same or different. If a first sender corresponds to multiple first incomplete messages, when obtaining the identifier of each of the at least one first sender of at least one first incomplete message, the identifier of the first sender corresponding to the multiple first incomplete messages may be recorded only once, without having to record it repeatedly.

[0090] Step 203: Determine the fault condition of each first transmitting end according to the identifier of each first transmitting end.

[0091] In the embodiment of the present application, the fault condition of any first transmitting end is used to indicate whether the corresponding first transmitting end is faulty.

[0092] There are many ways to determine the fault condition of each first transmitting end.

[0093] For example, in some examples, it is possible to determine whether the first sending end is faulty based solely on the information of the sending end corresponding to the first receiving queue. For example, the first duration can be set to a longer duration relative to the information transmission. In this case, the first incomplete message is a message that has not been completely received by the first receiving queue within a longer period of time. Then, network congestion and other conditions can be excluded, and it can be determined that the first sending end corresponding to the first incomplete message is faulty. Alternatively, if a first sending end in the first receiving queue corresponds to multiple first incomplete messages, and the receiving times of the messages received by the multiple first incomplete messages in the first receiving queue are different, it can be considered that the first receiving queue is continuously receiving messages from the first sending end, and the first sending end is not faulty, but there may be messages that have not been received by the first receiving queue due to network congestion and other conditions.

[0094] In other examples, whether the first sending end of the first receiving queue is faulty can be comprehensively determined based on the information of the first sending end of the first receiving queue and the information of the sending ends corresponding to other incomplete messages received.

[0095] Specifically, in some embodiments, the above step 203 includes:

[0096] Obtaining an identifier of each second sending end of at least one second sending end, where the at least one second sending end is a sending end of at least one second incomplete message of the second receiving queue;

[0097] The fault condition of each first transmitting end is determined according to the identifier of each second transmitting end and the identifier of each first transmitting end.

[0098] The at least one second incomplete message and the at least one first incomplete message may be detected in the same time period or in different time periods. The detection method for the second incomplete message may refer to the detection method for the first incomplete message, and the identifier of the second sending end may also refer to the identifier of the first sending end, which will not be repeated here.

[0099] The number of the second receiving queues can be one or more, which is not limited here.

[0100] In an embodiment of the present application, it is possible to comprehensively determine whether the sending end is faulty based on the sending end information of incomplete messages in multiple receiving queues.

[0101] For example, if the at least one second sender and the at least one first sender have the same sender, it can be considered that multiple receiving queues cannot completely receive the message from the same sender, and it can be considered that the same sender has failed. In other words, the first sender that is the same as any second sender can be regarded as a failed sender.

[0102] If a first sending end is different from any second sending end, and other receiving queues can also receive messages from the first sending end, it means that except for the first receiving queue, other receiving queues can completely receive the message from the first sending end, which means that the first sending end is not faulty, but due to network congestion or other conditions in the transmission connection corresponding to the first receiving queue, the message was not received by the receiving end, resulting in the first incomplete message.

[0103] It can be seen that by comprehensively judging whether the sending end is faulty based on the sending end information of incomplete messages in multiple receiving queues, it is possible to better eliminate situations such as incomplete messages caused by network congestion, thereby more accurately judging the fault condition of the first sending end.

[0104] Step 204: Process at least one first incomplete message according to the fault condition of each first sending end.

[0105] In an embodiment of the present application, the electronic device serving as the receiving end can identify the identifier of the first sender of the first incomplete message based on the first incomplete message in the first receiving queue, thereby proactively detecting a fault condition of the first sender. In this way, proactive processing of the first incomplete message in the first receiving queue can be achieved based on the fault condition of the first sender, thereby subsequently releasing corresponding receiving resources.

[0106] The fault condition of the first transmitting end may indicate that the first transmitting end is a faulty transmitting end or a normal transmitting end, and the first incomplete message may be processed differently in different fault conditions of the first transmitting end.

[0107] In the following, combined with the above scenario of comprehensively judging whether the sending end is faulty based on the sending end information of incomplete messages in multiple receiving queues, examples are given below to introduce how to handle the first incomplete information corresponding to the faulty sending end and the normal sending end respectively.

[0108] 1. The first sending end is a faulty sending end.

[0109] Specifically, in some embodiments, step 203 includes:

[0110] The first transmitting end that is the same as any second transmitting end is regarded as a faulty transmitting end;

[0111] Step 204 includes:

[0112] A first completion notification regarding a first incomplete message from a faulty sending end in a first receiving queue is generated, where the first completion notification is used to indicate that reception of the first incomplete message from the faulty sending end is complete.

[0113] In an embodiment of the present application, if a first sending end is the same as a second sending end, it can be considered that multiple receiving queues cannot completely receive the message from the first sending end. Therefore, the first sending end that is the same as the second sending end can be considered as a faulty sending end.

[0114] At this point, it can be considered that the first receiving queue is no longer able to receive the missing message in the first incomplete message from the faulty sending end, so a first completion notification of the first incomplete message from the faulty sending end in the first receiving queue can be generated to indicate that the first incomplete message from the faulty sending end has been received. And when the first receiving queue includes a complete message, the completion notification of the complete message can also be reported to the corresponding process of the receiving end according to the reporting order in the first receiving queue, so that the receiving resources consumed in the first receiving queue (for example, the receiving resources used to receive messages from the faulty sending end) can be recycled and will not be occupied by the first receiving queue all the time. In this way, the receiving resources and the corresponding processes can be easily deregistered and other operations can be performed later.

[0115] The first completion notification can take various forms. For example, the first completion notification can carry a first incomplete message from the faulty sending end. In this way, the first receiving queue can pass the first incomplete message to a process, etc., to indicate that the first incomplete message has been received. In some examples, the first completion notification can also indicate that the first incomplete message from the faulty sending end is an incomplete message. In this way, the corresponding process on the receiving end that receives the first incomplete message can determine that the first incomplete message is an incomplete message, facilitating subsequent processing, such as deletion or data processing of the first incomplete message.

[0116] In addition, in some examples, the first receiving queue may include a complete message received, and after the reception of the complete message is completed, a completion notification of the complete message may be generated, and the completion notification of the complete message may be reported to the corresponding process of the receiving end, etc. according to the reporting order in the first receiving queue.

[0117] In this example, the reporting completion process for the first incomplete message in the first receive queue and the complete message in the first receive queue can be the same, so that the receive resources consumed by the first receive queue can be recycled, avoiding waste of receive resources. When reporting the first incomplete message to the corresponding process, etc., it can be reported using the first type to indicate that the first incomplete message is an incomplete message, and when reporting the complete message to the corresponding process, etc., it can be reported using the second type to indicate that it is a complete message.

[0118] 2. The first sending end is a normal sending end.

[0119] Specifically, in some embodiments, step 203 includes:

[0120] taking a first transmitting end that is different from each second transmitting end as a normal transmitting end;

[0121] Step 204 includes:

[0122] Waiting for the missing message in the first incomplete message from the normal sending end in the first receiving queue;

[0123] If the first incomplete message from the normal sending end is incomplete after the waiting time reaches the second time, a second completion notification of the first incomplete message from the normal sending end is generated, and the second completion notification is used to indicate that the first incomplete message from the normal sending end has been received.

[0124] If a first sending end is different from any second sending end, and other receiving queues can also receive messages from the first sending end, it means that except for the first receiving queue, other receiving queues can completely receive the message from the first sending end, which means that the first sending end is not faulty, but due to network congestion or other conditions in the transmission connection corresponding to the first receiving queue, the message was not received by the receiving end, resulting in the first incomplete message.

[0125] Therefore, in the embodiment of the present application, you can continue to wait for the second period of time to wait for the message that has not been received by the receiving end due to network congestion or other conditions.

[0126] If the missing message in the first incomplete message from the normal sending end is obtained within the second waiting period, the complete message from the normal sending end can be obtained, and the complete message can be reported to the corresponding process of the receiving end.

[0127] If the first incomplete message from the normal sender is not complete after the waiting time reaches the second time, a second completion notification for the first incomplete message from the normal sender is generated. The second completion notification is used to indicate that the first incomplete message from the normal sender has been received, so that the corresponding receiving resources can be reclaimed and will not be continuously occupied by the first receiving queue. In this way, the receiving resources and the corresponding processes can be easily deregistered later.

[0128] As can be seen, in the embodiment of the present application, the receiving end can identify the identifier of the first sender of the first incomplete message based on the first incomplete message in the first receiving queue, thereby actively detecting the fault condition of the first sender. Then, based on the fault condition of the first sender, the first incomplete message in the first receiving queue can be actively processed to complete the receiving task corresponding to the first incomplete message in the first receiving queue, thereby releasing the corresponding receiving resources later.

[0129] It can be seen that through the embodiments of the present application, it is possible to effectively solve the problem that the receiving task of the receiving queue cannot be completed due to reasons such as failure of the sending end, resulting in the inability to recycle receiving resources. In addition, in a scenario where a transmission connection can realize message transmission between multiple virtual machines or multiple processes on the sending side and multiple processes or multiple virtual machines on the receiving side, it can also avoid the head-of-line blocking problem in the receiving queue caused by the failure of a virtual machine on the sending side.

[0130] For example, in Figure 4 In the example shown, the sending side Host0 includes virtual machines VM00 and VM01. Virtual machines VM00 and VM01 can serve as different sending ends, each uniquely represented by a different identifier (eg, MAC address, TCP / IP protocol tuple, etc.).

[0131] The virtual machines VM00 and VM01 can send messages to the receiving queue TP10 of the receiving end Host1 through the sending queue TP00 of the transmission connection, and transmit the messages to the virtual machine VM10 at the receiving end through the receiving queue TP10.

[0132] After adopting any of the above embodiments of the present application, even if the virtual machine VM00 fails, the incomplete messages from the virtual machine VM00 and the complete messages from the virtual machine VM01 in the receiving queue TP10 of the receiving end Host1 can be reported to the virtual machine VM10, so that the receiving resources consumed by the receiving queue TP10 from the virtual machine VM10 can be recovered, and the complete messages transmitted by the virtual machine VM01 on the sending side through the receiving queue TP10 will not be blocked in the receiving queue TP10 by the incomplete messages that cannot be processed, so that the normal business message transmission of the virtual machine VM01 is not affected by the failure of the virtual machine VM00.

[0133] A specific implementation method of the embodiment of the present application is described below through a specific example.

[0134] like Figure 5 As shown, in this example, the message receiving operation of receiving the queue may include one or more of the following steps:

[0135] Message input, protocol verification, message discard, message reordering, timing, sender fault diagnosis, receive resource processing, payload writing, and generation completion notification.

[0136] Below Figure 5 The various steps shown are introduced as examples.

[0137] 1. Message input

[0138] The electronic device as the receiving end can receive a message from the sending end through a transmission connection. The message may include a header and a payload, and these two parts can be cached independently.

[0139] 2. Protocol Verification

[0140] After receiving the message, the message may be verified according to relevant protocols (eg, transport layer related protocols, etc.).

[0141] 3. Packet discard

[0142] If the protocol verification fails, the message can be discarded.

[0143] 4. Message reordering

[0144] If the protocol verification passes, the message can be processed. In this example, out-of-order message transmission can be supported, and message reordering can be performed using bitmap.

[0145] After the message is reordered, it can be parsed based on the bitmap. The parsing matters include: whether the message is the first packet of the message, whether an incomplete message is generated based on the message based on the bitmap parsing, and whether a complete message exists based on the bitmap parsing.

[0146] 5. Timing

[0147] After the message is reordered, if it is determined through bitmap parsing that an incomplete message is generated based on the message, a timer is used to count the number of times.

[0148] 6. Transmitter fault diagnosis

[0149] If the incomplete message is still incomplete after the timing reaches the first time length, a timeout is determined, and the timeout message (such as the number of the relevant transmission queue, etc.) is sent to the sending end fault diagnosis module, so that the sending end fault diagnosis module can retrieve the identifier of the sending end corresponding to each incomplete message based on the bitmap. When it is determined according to the identifier that the sending end of the incomplete messages of multiple receiving queues has the same sending end, the same sending end is determined to be a faulty sending end.

[0150] 7. Receiving resource processing

[0151] If the parsed message is the first packet of a message, the receiving queue can apply for receiving resources and perform protocol verification and parsing related to the receiving resources.

[0152] 8. Payload Writing

[0153] After parsing the message as the first packet of the message, applying for receiving resources, and performing protocol verification and parsing related to the receiving resources, the payload of the message can be written to a double data rate synchronous dynamic random access memory (DDR SDRAM) or the like.

[0154] 9. Generate completion notification

[0155] When it is determined that the senders of incomplete messages in multiple receiving queues have the same sender, the same sender is determined to be a faulty sender. The incomplete messages and complete messages from the same sender can be reported to the corresponding process of the receiving end, so that the corresponding receiving queue generates a corresponding completion notification indicating that the receiving task is completed.

[0156] The above steps can be applied to various system architectures.

[0157] For example, in Figure 6In the system architecture shown, after the message enters the transport layer, protocol verification, message reordering, timing, sender fault diagnosis, receiving resource processing, payload writing, and completion notification generation can be performed. In addition, in this system architecture, the receiving resources can be located outside the transport layer, for example Figure 6 In the example, the receiving queue of the transport layer can apply for shared receiving resources from the transaction layer. In some other examples, the receiving resource can also be located in the transport layer, for example, the receiving resource can be an SRQ.

[0158] The data processing method provided in the embodiments of the present application has been described above from multiple aspects. The data processing device provided in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0159] like Figure 7 As shown, an embodiment of the present application provides a data processing device 70, which includes:

[0160] The detection module 701 is configured to: detect a first incomplete message in a first receiving queue;

[0161] The processing module 702 is configured to:

[0162] Upon detecting at least one first incomplete message in the first receiving queue, obtaining an identifier of each first sending end of at least one first sending end of the at least one first incomplete message;

[0163] determining a fault condition of each first transmitting end according to an identifier of each first transmitting end;

[0164] At least one first incomplete message is processed according to a fault condition of each first sending end.

[0165] Optionally, the processing module 702 is configured to:

[0166] If the first receiving queue receives the first message and generates an incomplete message based on the first message, the timing starts;

[0167] After the first time period starts, if the incomplete message generated based on the first message is incomplete, the incomplete message generated based on the first message is used as the first incomplete message.

[0168] Optionally, the processing module 702 is configured to:

[0169] If the first receiving queue receives the first message and generates a new incomplete message based on the first message, the timer starts.

[0170] Optionally, the processing module 702 is configured to:

[0171] Obtaining an identifier of each second sending end of at least one second sending end, where the at least one second sending end is a sending end of at least one second incomplete message of the second receiving queue;

[0172] The fault condition of each first transmitting end is determined according to the identifier of each second transmitting end and the identifier of each first transmitting end.

[0173] Optionally, the processing module 702 is configured to:

[0174] The first transmitting end that is the same as any second transmitting end is regarded as a faulty transmitting end;

[0175] A first completion notification regarding a first incomplete message from a faulty sending end in a first receiving queue is generated, where the first completion notification is used to indicate that reception of the first incomplete message from the faulty sending end is complete.

[0176] Optionally, the first completion notification indicates that the first incomplete message from the faulty sending end is an incomplete message.

[0177] Optionally, the processing module 702 is configured to:

[0178] taking a first transmitting end that is different from each second transmitting end as a normal transmitting end;

[0179] Waiting for the missing message in the first incomplete message from the normal sending end in the first receiving queue;

[0180] If the first incomplete message from the normal sending end is incomplete after the waiting time reaches the second time, a second completion notification of the first incomplete message from the normal sending end is generated, and the second completion notification is used to indicate that the first incomplete message from the normal sending end has been received.

[0181] Figure 8 80 is a schematic diagram of a possible logical structure of an electronic device 80 provided in an embodiment of the present application. The electronic device 80 is used to implement the functions of the electronic device involved in any of the above embodiments. The electronic device 80 includes: a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, processor 802, and communication interface 803 are connected to each other via the bus 804.

[0182] The memory 801 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 801 may store programs. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to perform one or more steps in the above-described data processing method embodiment.

[0183] The processor 802 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component or any combination thereof, used to execute relevant programs to implement the functions required to be executed by the detection module and the processing module in the data processing device in the above embodiment, or to execute one or more steps in the embodiment of the method of the present application. The steps of the method disclosed in conjunction with the embodiment of the present application can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801 and executes one or more steps in the above-mentioned data processing method embodiment in combination with its hardware.

[0184] The communication interface 803 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the electronic device 80 and other devices or a communication network.

[0185] The bus 804 can implement a path for transmitting information between the various components of the electronic device 80 (for example, the memory 801, the processor 802, and the communication interface 803). The bus 804 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0186] In another embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the above-mentioned Figure 8 The steps performed by the processor in .

[0187] In another embodiment of the present application, a computer program product is further provided. The computer program product includes computer-executable instructions stored in a computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the device executes the above-mentioned Figure 8 The steps performed by the processor in .

[0188] In another embodiment of the present application, a chip system is further provided, the chip system including a processor, the processor being configured to implement the above Figure 8 In one possible design, the chip system may also include a memory for storing program instructions and data necessary for data writing. The chip system may consist of a chip or may include a chip and other discrete devices.

[0189] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0191] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A data processing method, characterized in that: The method comprises: detecting a first incomplete message in a first receive queue; In a case where at least one first incomplete message is detected in the first receiving queue, obtaining an identifier of each of the at least one first sending end of the at least one first incomplete message; determining a fault condition of each of the first transmitting ends according to the identifier of each of the first transmitting ends; The at least one first incomplete message is processed according to a fault condition of each of the first sending ends.

2. The method according to claim 1, characterized in that The detecting a first incomplete message in the first receiving queue includes: If the first receiving queue receives a first message and generates an incomplete message based on the first message, start timing; After the first time period starts, if the incomplete message generated based on the first message is incomplete, the incomplete message generated based on the first message is used as the first incomplete message.

3. The method according to claim 2, characterized in that If the first receiving queue receives the first message and generates an incomplete message based on the first message, starting timing includes: If the first receiving queue receives a first message and generates a new incomplete message based on the first message, the timing starts.

4. The method according to any one of claims 1 to 3, characterized in that The determining, according to the identifier of each first transmitting end, a fault condition of each first transmitting end includes: Obtaining an identifier of each of at least one second sending end, where the at least one second sending end is a sending end of at least one second incomplete message of the second receiving queue; A fault condition of each of the first transmitting ends is determined according to the identifier of each of the second transmitting ends and the identifier of each of the first transmitting ends.

5. The method according to claim 4, characterized in that The determining, according to the identifier of each second transmitting end and the identifier of each first transmitting end, a fault condition of each first transmitting end includes: taking the first transmitting end that is the same as any of the second transmitting ends as a faulty transmitting end; The processing of the at least one first incomplete message according to the fault condition of each first sending end includes: A first completion notification regarding a first incomplete message from the faulty sending end in the first receiving queue is generated, where the first completion notification is used to indicate that reception of the first incomplete message from the faulty sending end is complete.

6. The method according to claim 5, characterized in that The first completion notification indicates that the first incomplete message from the faulty sending end is an incomplete message.

7. The method according to claim 4, characterized in that The determining, according to the identifier of each second transmitting end and the identifier of each first transmitting end, a fault condition of each first transmitting end includes: taking a first transmitting end different from each of the second transmitting ends as a normal transmitting end; The processing of the at least one first incomplete message according to the fault condition of each first sending end includes: Waiting for a missing message in the first incomplete message from the normal sending end in the first receiving queue; If the first incomplete message from the normal sending end is incomplete after the waiting time reaches the second time, a second completion notification of the first incomplete message from the normal sending end is generated, and the second completion notification is used to indicate that the first incomplete message from the normal sending end has been received.

8. A data processing device, characterized in that: include: A detection module, configured to: detect a first incomplete message in a first receiving queue; Processing module for: In a case where at least one first incomplete message is detected in the first receiving queue, obtaining an identifier of each of the at least one first sending end of the at least one first incomplete message; determining a fault condition of each of the first transmitting ends according to the identifier of each of the first transmitting ends; The at least one first incomplete message is processed according to a fault condition of each of the first sending ends.

9. The device according to claim 8, characterized in that The processing module is used for: If the first receiving queue receives a first message and generates an incomplete message based on the first message, start timing; After the first time period starts, if the incomplete message generated based on the first message is incomplete, the incomplete message generated based on the first message is used as the first incomplete message.

10. The device according to claim 9, characterized in that The processing module is used for: If the first receiving queue receives a first message and generates a new incomplete message based on the first message, the timing starts.

11. The device according to any one of claims 8 to 10, characterized in that: The processing module is used for: Obtaining an identifier of each of at least one second sending end, where the at least one second sending end is a sending end of at least one second incomplete message of the second receiving queue; A fault condition of each of the first transmitting ends is determined according to the identifier of each of the second transmitting ends and the identifier of each of the first transmitting ends.

12. The device according to claim 11, characterized in that The processing module is used for: taking the first transmitting end that is the same as any of the second transmitting ends as a faulty transmitting end; A first completion notification regarding a first incomplete message from the faulty sending end in the first receiving queue is generated, where the first completion notification is used to indicate that reception of the first incomplete message from the faulty sending end is complete.

13. The device according to claim 12, characterized in that The first completion notification indicates that the first incomplete message from the faulty sending end is an incomplete message.

14. The device according to claim 11, characterized in that The processing module is used for: taking a first transmitting end different from each of the second transmitting ends as a normal transmitting end; Waiting for a missing message in the first incomplete message from the normal sending end in the first receiving queue; If the first incomplete message from the normal sending end is incomplete after the waiting time reaches the second time, a second completion notification of the first incomplete message from the normal sending end is generated, and the second completion notification is used to indicate that the first incomplete message from the normal sending end has been received.

15. An electronic device, characterized in that: The electronic device includes a processor, a memory, and instructions stored in the memory and executable by the at least one processor, and when the instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

17. A computer program product comprising instructions, characterized in that When the instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images