Control method for retry of distributed system, electronic device and distributed system

Abstract

The application discloses a control method for retry of a distributed system, an electronic device and the distributed system. The method is applied to a first node and comprises the following steps: receiving second feedback information sent by a second node; updating a first information table of the first node according to the second feedback information, and updating a remaining retry number of a first request; the remaining retry number is a remaining number of times that all nodes in the distributed system can re-process the first request; in response to failure of the first node in processing the first request, re-executing a local task corresponding to the first request when the remaining re-processing number of the first request is greater than zero; updating the remaining retry number of the first request in the first information table, sending the first request to the second node, so that the second node updates a second information table according to the received first request and processes the first request; and sending first feedback information to a third node when the remaining re-processing number of the first request is equal to zero.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a control method and device for retry of a distributed system and electronic equipment. BACKGROUND

[0002] In a distributed system, due to a long request processing link, a node often fails to process a request, and thus the request needs to be processed again, that is, retry is performed.

[0003] However, there are many nodes participating in the distributed system, and each upstream node performs retry, which easily causes great pressure on downstream nodes, a retry storm, exceeds the maximum processing capacity of the downstream nodes, and causes the downstream nodes to fail.

[0004] Therefore, a technical solution for controlling retry in a distributed system is urgently needed. SUMMARY

[0005] In view of the above problems, the present application provides a control method for retry of a distributed system, electronic equipment and a distributed system, to control the retry of each node in the distributed system, so as to avoid a retry storm and reduce the processing pressure of the nodes. The specific solution is as follows:

[0006] The first aspect of the present application provides a control method for retry of a distributed system, applied to a first node, the first node being any node in the distributed system, and the method comprising:

[0007] receiving second feedback information sent by a second node; the second node being a downstream node adjacent to the first node in the distributed system; and the second feedback information containing the total number of retries of the first request by the second node and downstream nodes thereof;

[0008] updating a first information table of the first node according to the second feedback information, the first information table containing at least the remaining number of retries of the first request; and the remaining number of retries being the remaining number of times that all nodes in the distributed system can process the first request again;

[0009] in response to the first node failing to process the first request, determining whether the remaining number of retries of the first request in the first information table is greater than or equal to zero;

[0010] in case that the remaining retry times of the first request is greater than zero, re-executing a local task corresponding to the first request, updating the remaining retry times of the first request in the first information table and sending the first request to the second node; the first request at least carrying the remaining retry times of the first request, so that the second node updates a second information table of the second node according to the received first request and processes the first request, the second information table at least containing the remaining retry times of the first request;

[0011] in case that the remaining retry times of the first request is equal to zero, sending first feedback information to a third node, the first feedback information containing total retry times of the first request by the first node and downstream nodes of the first node; the third node being an adjacent upstream node of the first node in the distributed system.

[0012] In a possible implementation, the method further comprises: receiving a second request sent by the third node, the second request at least carrying remaining retry times of the second request; updating the remaining retry times of the second request in the first information table according to the remaining retry times of the second request; processing the second request.

[0013] In a possible implementation, in case that the first request is different from the second request, an execution order between the first request and the second request is determined based on a first priority of the first request and a second priority of the second request; the first priority of the first request being related to the remaining retry times of the first request; the second priority of the second request being related to the remaining retry times of the second request.

[0014] In a possible implementation, the first information table further contains retried times of the first request and retried times of the second request; the retried times of the first request being a number of times that all nodes in the distributed system have retried processing the first request; the retried times of the second request being a number of times that all nodes in the distributed system have retried processing the second request; wherein the first priority of the first request is obtained based on the remaining retry times and the retried times of the first request, and the second priority of the second request is obtained based on the remaining retry times and the retried times of the second request.

[0015] In a possible implementation, the first priority of the first request is carried in the first request; the second priority of the second request is carried in the second request;

[0016] or,

[0017] The first priority of the first request is obtained based on the remaining retry times of the first request in the first information table; and the second priority of the second request is obtained based on the remaining retry times of the second request in the first information table.

[0018] In a possible implementation, the second request also carries at least one retry parameter of the second request; and the retry parameter includes a retry timeout timestamp, a retry condition, and a retry interval duration.

[0019] The first node prohibits retry processing the second request in a case where a current time is greater than the retry timeout timestamp.

[0020] The first node retries processing the second request at least in a case where the retry condition is met.

[0021] The first node retries processing the second request after the retry interval duration.

[0022] In a possible implementation, before sending the first request to the second node, the method further includes:

[0023] According to a request type of the first request, it is determined whether the second node can perform retry for the first request.

[0024] If the second node can perform retry, a retry attribute of the second node for the first request is set as a first attribute; the first attribute is used to instruct the second node to retry processing the first request in a case where processing the first request fails.

[0025] If the second node cannot perform retry, the retry attribute of the second node for the first request is set as a second attribute; the second attribute is used to instruct the second node not to retry processing the first request in a case where processing the first request fails.

[0026] In a possible implementation, updating the remaining retry times of the first request in the first information table includes:

[0027] The remaining retry times of the first request in the first information table are reduced by a number of times of retry processing the first request by the first node.

[0028] The second aspect of the present application provides a distributed system, including at least a first node, a second node, and a third node; wherein:

[0029] The first node receives second feedback information sent by the second node; the second node is a downstream node adjacent to the first node in the distributed nodes; the second feedback information includes a total number of retries of the first request by the second node and downstream nodes thereof;

[0030] The first node updates a first information table of the first node according to the second feedback information, the first information table including at least a remaining number of retries of the first request; the remaining number of retries is a remaining number of times that all nodes in the distributed system can re-process the first request;

[0031] The first node determines whether the remaining number of retries of the first request in the first information table is greater than or equal to zero in response to a failure of the first node to process the first request;

[0032] In a case where the remaining number of retries of the first request is greater than zero, the first node re-executes a local task corresponding to the first request, updates the remaining number of retries of the first request in the first information table, and sends the first request to the second node; the first request carries at least the remaining number of retries of the first request, so that the second node updates a second information table of the second node according to the received first request and processes the first request, the second information table including at least the remaining number of retries of the first request;

[0033] In a case where the remaining number of retries of the first request is equal to zero, the first node sends first feedback information to a third node, the first feedback information including a total number of retries of the first request by the first node and downstream nodes thereof; the third node is an upstream node adjacent to the first node in the distributed system.

[0034] The third aspect of the present application provides a computer program product, including computer readable instructions, when the computer readable instructions run on an electronic device, the electronic device implements the control method of the distributed system for retrying according to the first aspect or any implementation manner of the first aspect.

[0035] The fourth aspect of the present application provides an electronic device, including at least one processor and a memory connected with the processor, wherein:

[0036] The memory is used to store a computer program;

[0037] The processor is used to execute the computer program, so that the electronic device can implement the control method of the distributed system for retrying according to the first aspect or any implementation manner of the first aspect.

[0038] The fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can make the distributed system of the first aspect or any implementation manner of the first aspect execute the control method of retry.

[0039] By the above technical solution, the present application provides a control method of retry of a distributed system, an electronic device and a distributed system. The total retry number fed back by a downstream node is received on any node, so as to update the remaining retry number for a request in the local information table. If the downstream node feeds back a processing failure, it is determined whether to retry based on whether the remaining retry number is greater than zero, that is, if the remaining retry number is greater than zero, the node is allowed to retry, that is, to re-execute the task corresponding to the request and update the remaining retry number to the downstream node, and if the remaining retry number reaches zero, the retry is not executed and the total retry number is fed back to the upstream node, so that the upstream node updates the remaining retry number in the same way. It can be seen that the present application updates the remaining retry number based on the total retry number fed back by the downstream node on each node of the distributed system and determines whether to retry accordingly, and feeds back the total retry number to the upstream node. Thus, the remaining retry number is continuously updated by the upstream and downstream nodes, so as to control the retry number and avoid retry storm on the downstream node, thereby reducing the processing pressure of the downstream node. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 A system architecture diagram disclosed by the present application is shown in the figure.

[0042] Figure 2 A hardware structure diagram of the terminal 100 in the present application is shown in the figure.

[0043] Figure 3 A hardware structure diagram of the server 200 in the present application is shown in the figure.

[0044] Figure 4 Another system architecture diagram disclosed by the present application is shown in the figure.

[0045] Figure 5 A flow diagram of the control method of retry of a distributed system provided by the embodiment of the present application is shown in the figure.

[0046] Figure 6Another flowchart of a control method for performing retry of a distributed system according to an embodiment of the present application is provided.

[0047] Figure 7 Another flowchart of a control method for performing retry of a distributed system according to an embodiment of the present application is provided.

[0048] Figure 8 A structure diagram of a control device for performing retry of a distributed system according to an embodiment of the present application is provided.

[0049] Figure 9 A structure diagram of an electronic device suitable for implementing an embodiment of the present application is provided.

[0050] Figure 10 A structure diagram of a distributed system suitable for implementing an embodiment of the present application is provided.

[0051] Figure 11 A flowchart of a process of implementing retry control by modules of a distributed system according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiments section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0053] The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those of ordinary skill in the art that, as technology develops and new scenarios emerge, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0054] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms used in this way can be interchanged as appropriate, and this is only a way of distinguishing objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device that includes a series of units does not necessarily have to be limited to those units, but can include other units not clearly listed or inherent to the process, method, product or device.

[0055] The present application can be applied in the field of data processing. The following will take the processing of video data as an example to introduce a plurality of application scenarios that fall into products.

[0056] Referring to Figure 1 , Figure 1A system architecture diagram is shown. The system can include terminals 100 and servers 200. Each of the servers 200 constitutes a node of a distributed system. The server 200 can include one or more servers (as shown by an example of one server). Figure 1 The server 200 can provide the method provided by the embodiments of the present application for one or more terminals 100.

[0057] The terminal 100 can install a video application. The application can provide an interface. The terminal 100 can receive an operation input by a user on the interface and generate a corresponding request, and send the request to the server 200. The server 200 can obtain a processing result based on the received instruction, and return the processing result to the terminal 100. For example, the terminal 100 generates a video acquisition request, sends the video acquisition request to a server 200 in the distributed system, the server 200 and other servers 200 jointly process the request through cascading to obtain video data, and returns the video data as a processing result to the terminal 100.

[0058] Next, the product form of the terminal 100 is described. Figure 1

[0059] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not make any limitation in this regard.

[0060] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0061] Referring to Figure 2 As shown, the terminal 100 can include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190, etc. Those skilled in the art can understand that Figure 2 The terminal or multi-functional device is only an example and does not constitute a limitation on the terminal or multi-functional device, which can include more or less components, or combine certain components, or different components. ​

[0062] The input unit 130 can be configured to receive inputted digital or character information, and to generate key signal inputs related to user settings of the portable multifunctional device and control of functions. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations of a user thereon or proximate thereto (such as operations of the user using a finger, a knuckle, a stylus, or any suitable object on or proximate to the touch screen), and drive corresponding connected devices according to a pre-set program. The touch screen can detect touch actions of the user on the touch screen, convert the touch actions into touch signals and send the touch signals to the processor 170, and can receive commands from the processor 170 and execute the commands; the touch signals at least include touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc.

[0063] The input device 132 can receive inputted data, etc.

[0064] The display unit 140 can be configured to display information inputted by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playing of any kind of multimedia files. In the embodiments of the present application, the display unit 140 can be configured to display interfaces of video applications, processing results such as video data, etc.

[0065] The storage 120 can be configured to store instructions and data. The storage 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc.; the storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc., or their subsets, expanded sets. It can also include a non-volatile random access memory; provide the processor 170 with software and applications that include management of hardware, software, and data resources in the computing processing device, support control. It is also used for storage of multimedia files, and storage of running programs and applications.

[0066] The processor 170 is the control center of the terminal 100, connects each part of the terminal 100 by various interfaces and lines, executes various functions of the terminal 100 and processes data by running or executing instructions stored in the memory 120 and calling data stored in the memory 120, and thus controls the terminal device as a whole. Optionally, the processor 170 can include one or more processing units; preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips respectively. The processor 170 can also be used to generate corresponding operation control signals to send to corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein executes corresponding functions, and thus controls the corresponding components to act according to the requirements of the instructions.

[0067] The memory 120 can be used to store software codes related to the video application, and the processor 170 can execute the execution steps of the video application, or can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to realize corresponding functions.

[0068] The RF unit 110 (optional) can be used to receive and send signals in the process of information or communication, for example, receiving the downlink information of the base station, and processing by the processor 170. In addition, the uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0069] In the embodiments of the present application, the RF unit 110 can send a request to the server 200 and receive the processing result sent by the server 200.

[0070] It should be understood that the RF unit 110 is optional, which can be replaced by other communication interfaces, for example, a network interface.

[0071] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system.

[0072] The terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector. The external interface 180 can be used to connect the terminal 100 with other devices for communication, and can also be used to connect a charger for charging the terminal 100.

[0073] Although not shown, the terminal 100 can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described here. Some or all of the methods described below can be applied in the terminal 100 as shown. Figure 2

[0074] ​Next, the product form of the server 200 is described. Figure 1 The product form of the server 200 is described.

[0075] Figure 3 A structural diagram of the server 200 is provided, as shown in the figure. Figure 3 The server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201.

[0076] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 In the figure, only one thick line is used, but it does not mean that there is only one bus or one type of bus.

[0077] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0078] The memory 204 can include a volatile memory, such as a random access memory (RAM). The memory 204 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0079] The memory 204 can be used to store software code related to the control method for retry of the distributed system, and the processor 202 can execute the steps of the control method for retry of the chip of the distributed system, or can schedule other units to realize the corresponding functions.

[0080] It should be understood that the terminal 100 and the server 200 described above can be centralized or distributed devices, and the processors (for example, the processor 170 and the processor 202) in the terminal 100 and the server 200 can be hardware circuits (for example, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processing (DSP), a microprocessor, a microcontroller, or the like) or a combination of the hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, or the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, or the like, or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0081] Reference Figure 4 , Figure 4 Another system architecture diagram is shown, which can include a terminal 100 and a server 200. The server 200 has multiple microservices deployed therein, and each microservice serves as a node to form a distributed system. Figure 4 The specific structure of the terminal 100 and the server 200 in the above system can refer to the content in the foregoing.

[0082] For example, a distributed system for video data processing is deployed on a server in the cloud. The distributed system includes multiple nodes, each node is implemented by a microservice, and each microservice is used to perform a corresponding local task. Based on this, multiple microservices form a cascading call to implement video data processing, such as video data filtering or playlist generation.

[0083] For example, a video acquisition request is generated on the terminal 100, and the video acquisition request is sent to a microservice in the distributed system on the server 200. The microservice and other microservices jointly process the request through cascading to obtain video data, and return the video data to the terminal 100 as a processing result.

[0084] Based on the above system architecture, the technical solutions of the present application are described as follows:

[0085] In the distributed system, due to the long request processing link, the situation that a node fails to process a request often occurs, and then the request needs to be processed again. However, there are many nodes participating in the distributed system, and if each upstream node retries, it is easy to cause great pressure on downstream nodes, a retry storm occurs, exceeds the maximum processing capacity of the downstream nodes, and causes the downstream nodes to fail.

[0086] For example, if a request passes through A->B->C->D 4 nodes, each node is configured with 3 times of retry number, if D node is overloaded, an error occurs when processing the request, then C node will fail after retry, return the failure information to B, B will also fail in retry, return the failure information to A, A continues to retry, so A, B and C nodes will all retry, then D node will finally receive 3x3x3=27 requests, if the maximum processing capacity of D is 1000 requests per second, and the number of failed requests per second is 100, then 2700 retries per second will be performed, which far exceeds the processing capacity of D!

[0087] To solve the above problems, the embodiment of the application provides a control method for performing retry of a distributed system. The control method for performing retry of a distributed system of the embodiment of the application will be described in detail below with reference to the accompanying drawings.

[0088] Reference Figure 5 , Figure 5 A flowchart of a control method for performing retry of a distributed system provided by the embodiment of the application is shown in the figure. The method can be applied to a first node in a distributed system. The first node is any node in the distributed system, such as any server shown in Figure 1 or any microservice shown in Figure 4 . Specifically, the method in the embodiment can include the following steps:

[0089] Step 501: The first node receives second feedback information sent by a second node.

[0090] The second node is a downstream node adjacent to the first node in the distributed system. The second feedback information contains the total number of retries of the first request by the second node and its downstream nodes.

[0091] For example, the distributed system contains four nodes: A, B, C, and D. The execution order is A->B->C->D. B is the downstream node of A, and B is the upstream node of C. Taking node B as the first node, node B receives the second feedback information sent by node C, which contains at least the sum of the number of retries of the first request by node C and node D, i.e., the total number of retries.

[0092] It should be noted that the first request is sent by a request end to a starting node of the distributed system. For example, the terminal 100 sends a video acquisition request to node A in the distributed system.

[0093] Step 502: The first node updates a first information table of the first node according to the second feedback information.

[0094] The first information table includes at least the remaining retry number of the first request, and the remaining retry number is the remaining number of times that all nodes in the distributed system can retry processing the first request.

[0095] It should be noted that the distributed system is configured with a maximum retry number, and the remaining retry number is less than or equal to the maximum retry number, that is, the initial value of the remaining retry number is the maximum retry number. After any node retries processing the first request, the remaining retry number is decremented by one.

[0096] Based on this, the first node records the remaining retry number of the first request through the first information table. After receiving the second feedback information of the downstream node such as the second node each time, the remaining retry number in the first information table is updated according to the total retry number of the downstream node in the second feedback information.

[0097] Specifically, the first node can subtract the total retry number in the second feedback information from the remaining retry number of the first request in the first information table, and take the obtained number of times as the updated remaining retry number of the first request in the first information table.

[0098] The total retry number in the second feedback information can be obtained in the following manner:

[0099] The number of times that the downstream node of the second node retries processing the first request is added to the number of times that the second node retries processing the first request.

[0100] It should be noted that the number of times that the second node retries processing the first request refers to the number of times that the second node retries processing the first request after receiving the message of processing the first request failed fed back by the downstream node for the first time. After the second node sends the second feedback information to the first node, the number of times that the second node retries processing the first request is cleared. Subsequently, if the first node retries processing the first request causes the second node to receive the first request again and execute the task corresponding to the first request, and if the message of processing the first request failed fed back by the downstream node is received, the second node starts to retry processing the first request for the first time, and the number of retries is 1. If the message of processing the first request failed fed back by the downstream node is received again, the second node starts to retry processing the first request for the second time, and the number of retries is 2. In this way, the number of retries is incremented by one each time.

[0101] For example, taking the distributed system containing four nodes: A, B, C, D as an example, node B receives the second feedback information sent by node C, the second feedback information contains: after node B sends the video acquisition request to node C, the total retry number of node C and node D retry processing the first request is 5, node B deducts 5 from the remaining retry number 10 of the video acquisition request in the local information table of node B, finally, the remaining retry number of the video acquisition request in the local information table of node B is 5, which represents that the opportunity of each node in the subsequent distributed system to retry processing the video acquisition request is only 5 times. And, node C sets its total retry number to 0. If node B retries processing the video acquisition request, the remaining retry number is 4, during which node C needs to be called to send the video acquisition request to node C, then node C processes the first request by calling node D after receiving the video acquisition request, if node C fails to execute the task of the video acquisition request (for example, node C receives the feedback information that node D fails to process the video acquisition request) and receives the feedback information that node D retries 0 times, then the remaining retry number in the local information table of node C does not change (because the retry number of D is 0) but retries processing the video acquisition request, at this time, the total retry number of node C is 1 (the number of times of retry of node C) plus 0 (the number of times of retry of node D), after the end of retry processing the video acquisition request, node C feeds back the total retry number 1 and the request processing result of node C retry processing the video acquisition request to node B, and then node C clears the total retry number and node B updates the remaining retry number to 3.

[0102] Step 503: determining whether the first node fails to process the first request, if not, i.e. the first node succeeds in processing the first request, executing step 504, if yes, in response to the first node failing to process the first request, executing step 505;

[0103] The first node processing the first request includes: the first node executing the local task corresponding to the first request, and the first node calling the second node to process the first request by the second node. Correspondingly, the first node failing to process the first request includes: the first node failing to execute the local task corresponding to the first request, and / or the second feedback information representing the second node failing to process the first request. Correspondingly, the second node failing to process the first request includes: the second node failing to execute the local task corresponding to the first request, and / or the feedback information sent by the downstream node of the second node representing the downstream node failing to process the first request.

[0104] Step 504: sending third feedback information to the third node.

[0105] The third feedback information includes the total retry times of the first node and its downstream nodes for the first request and a request processing result of the first node processing the first request. The request processing result of the first node processing the first request is obtained by the first node according to the request processing result in the second feedback information for continuing to execute the local task corresponding to the first request. The request processing result in the second feedback information is obtained by the second node according to the request processing result in the feedback information sent by its downstream nodes for executing the local task corresponding to the first request.

[0106] It should be noted that after receiving the third feedback information sent by the first node, the third node continues to execute the local task corresponding to the first request according to the request processing result in the third feedback information, and so on, until the starting node in the distributed system successfully processes the first request and sends the corresponding request processing result to the request end of the first request.

[0107] For example, taking a distributed system including four nodes A, B, C and D as an example, the node B receives the second feedback information sent by the node C, and the second feedback information includes the total retry times of the node C and its downstream nodes for the video acquisition request. The node B updates the remaining retry times of the video acquisition request in the local information table of the node B according to the total retry times in the second feedback information. If the second feedback information indicates that the second node successfully processes the first request, the node B continues to execute the local task corresponding to the video acquisition request according to the request processing result in the second feedback information. If the node B successfully executes the local task, it can be determined that the node B successfully processes the video acquisition request. At this time, the node B generates the third feedback information and adds the total retry times of the node B and its downstream nodes for the video acquisition request and the request processing result of the node B processing the video acquisition request in the third feedback information. Therefore, after receiving the third feedback information, the node A continues to execute the local task corresponding to the video acquisition request according to the request processing result in the third feedback information. If the node A successfully executes the local task, the node A can obtain the request processing result of the video acquisition request such as the acquired video data and sends the request processing result to the terminal 100 corresponding to the video acquisition request.

[0108] Step 505: Determine whether the remaining retry times of the first request in the first information table are greater than or equal to zero. If equal to zero, execute step 506; if greater than zero, execute step 507.

[0109] Step 506: Send the first feedback information to the third node.

[0110] The first feedback information includes the total retry times of the first node and its downstream nodes for the first request. The third node is the adjacent upstream node of the first node in the distributed system.

[0111] That is, if the remaining retry number of the first request is zero, it means that the first node cannot retry processing the first request any more, at this time, the first node feeds back to the upstream node, i.e., the third node, the total retry number of the first request by the first node and its downstream nodes, and the third node updates the remaining retry number of the first request in the third information table of the third node based on the total retry number.

[0112] For example, taking the distributed system containing four nodes: A, B, C, and D as an example, node B receives the second feedback information sent by node C, the second feedback information contains the total retry number of the video acquisition request by node C and its downstream nodes, and node B updates the remaining retry number of the video acquisition request in the local information table of node B according to the total retry number in the second feedback information. If the second feedback information indicates that the second node successfully processes the first request but node B fails to execute the local task corresponding to the video acquisition request according to the request processing result in the second feedback information, or if the second feedback information indicates that the second node fails to process the first request, then node B is ready to retry processing the first request, at this time, it is first determined whether the remaining retry number of the video acquisition request in the local information table is greater than zero or equal to zero. If the remaining retry number is equal to zero, it means that node B cannot retry processing the first request any more, at this time, node B sends the first feedback information to node A, the first feedback information contains the total retry number of the video acquisition request by node B and its downstream nodes, and node A updates the remaining retry number of the video acquisition request in the local information table of node A according to the total retry number in the first feedback information. Since node B fails to process the video acquisition request, node A determines that there is no opportunity to retry processing the first request based on the remaining retry number, at this time, the processing of the video acquisition request is ended, and the information of processing failure is fed back to the terminal 100.

[0113] It should be noted that the third node can be the starting node for processing the first request in the distributed system.

[0114] If the first node is the starting node for processing the first request in the distributed system, then the first node does not execute step 506 in the case where it is determined that the remaining retry number is equal to zero, because the first node has no upstream node.

[0115] Step 507: re-executes the local task corresponding to the first request.

[0116] Step 508: updates the remaining retry number of the first request in the first information table.

[0117] For example, the remaining retry number of the first request in the first information table is reduced by the number of times of retrying processing the first request by the first node. For example, after re-executing the local task corresponding to the first request each time, the remaining retry number of the first request in the first information table is reduced by one.

[0118] Step 509: sending the first request to the second node.

[0119] The first request sent to the second node carries at least the remaining retry times of the first request. That is, after updating the remaining retry times of the first request in the first information table, the remaining retry times of the first request in the first information table are added to the first request, and then the first request is sent to the second node, so that the second node updates the second information table according to the received first request and processes the first request.

[0120] The second information table contains at least the remaining retry times of the first request. Specifically, the second node replaces the remaining retry times of the first request in the second information table with the remaining retry times of the first request carried by the received first request.

[0121] For example, in a distributed system containing four nodes: A, B, C, and D, node B receives the second feedback information sent by node C, and the second feedback information contains the total retry times of the video acquisition request by node C and its downstream nodes. Node B updates the remaining retry times of the video acquisition request in the local information table of node B according to the total retry times in the second feedback information. If the second feedback information indicates that the second node successfully processes the first request but node B fails to execute the local task corresponding to the video acquisition request according to the request processing result in the second feedback information, or if the second feedback information indicates that the second node fails to process the first request, then node B prepares to retry processing the first request. At this time, it is first determined whether the remaining retry times of the video acquisition request in the local information table are greater than zero or equal to zero. If the remaining retry times are greater than zero, it means that node B can still retry processing the video acquisition request. At this time, node B re-executes the local task of the video acquisition request, reduces the remaining retry times of the video acquisition request in the local information table by one, and sends the video acquisition request carrying the remaining retry times to call node C and its downstream nodes to process the video acquisition request. Node C updates the remaining retry times of the video acquisition request in the local information table according to the received video acquisition request. The remaining retry times of the video acquisition request in the local information table are used for retry judgment by node C when processing the video acquisition request fails. Until node C successfully processes the video acquisition request and sends the second feedback information to node B, or until node C fails to retry processing the video acquisition request and the remaining retry times are zero.

[0122] By the technical solution, the control method for retry of the distributed system provided by the embodiment of the application receives the total retry times fed back by the downstream node on any node, so as to update the remaining retry times of the request in the local information table, if the downstream node feeds back a processing failure, it is determined whether to retry based on whether the remaining retry times are greater than zero, that is, if the remaining retry times are greater than zero, the node is allowed to retry, that is, to re-execute the task corresponding to the request and to update the remaining retry times to the downstream node, and if the remaining retry times reach zero, the retry is not executed any more and the total retry times are fed back to the upstream node, so that the upstream node updates the remaining retry times in the same way. It can be seen that the embodiment updates the remaining retry times based on the total retry times fed back by the downstream node on each node in the distributed system and determines whether to retry according to the remaining retry times, and the total retry times are fed back to the upstream node, so that the upstream node and the downstream node continuously update the remaining retry times, so as to control the retry times, avoid the retry storm on the downstream node, and reduce the processing pressure of the downstream node.

[0123] In an implementation manner, the first node can further have the following processing, as shown in the following: Figure 6

[0124] Step 510: The first node receives the second request sent by the third node.

[0125] The second request carries at least the remaining retry times of the second request. The second request can be the same as the first request, or the second request can not be the same as the first request.

[0126] It should be noted that the second request can be the request sent by the third node to the first node for the first time to process the second request, or the request sent by the third node to the first request to retry processing the second request.

[0127] Step 511: The first node updates the remaining retry times of the second request in the first information table according to the remaining retry times of the second request.

[0128] Specifically, the first node replaces the remaining retry times of the second request in the first information table with the remaining retry times of the second request carried by the received second request.

[0129] Step 512: The first node processes the second request.

[0130] The first node processing the second request includes that the first node executes the local task corresponding to the second request, and the first node calls the second node to process the second request.

[0131] ​For example, taking the example of a distributed system containing four nodes: A, B, C, and D, node B receives a video acquisition request sent by node A, node B updates the remaining retry times of the video acquisition request in the local information table according to the remaining retry times carried by the received video acquisition request, and processes the video acquisition request, and the remaining retry times of the video acquisition request in the local information table are used for retry judgment by node B when the video acquisition request fails to be processed, until node B successfully processes the video acquisition request and sends corresponding feedback information to node A, or until the number of times that node B retries to process the video acquisition request fails such that the remaining retry times are zero.

[0132] For example, taking the example of a distributed system containing four nodes: A, B, C, and D, node B receives a video acquisition request sent by node A, node B updates the remaining retry times of the video acquisition request in the local information table according to the remaining retry times carried by the received video acquisition request, and processes the video acquisition request, and the remaining retry times of the video acquisition request in the local information table are used for retry judgment by node B when the video acquisition request fails to be processed, until node B successfully processes the video acquisition request and sends corresponding feedback information to node A, or until the number of times that node B retries to process the video acquisition request fails such that the remaining retry times are zero.

[0133] Based on the above implementation, in the case that the first request and the second request are different, the execution order between the first request and the second request is determined based on the first priority of the first request and the second priority of the second request.

[0134] The first priority of the first request is related to the remaining retry times of the first request; and the second priority of the second request is related to the remaining retry times of the second request.

[0135] That is, in the case that there are multiple requests to be processed on the first node, the execution order of each request can be determined according to the priority of each request. The request with a higher priority is preferentially processed by the first node.

[0136] Specifically, the first information table further contains the number of times that the first request has been retried and the number of times that the second request has been retried. The number of times that the first request has been retried is the number of times that all nodes in the distributed system have retried to process the first request; and the number of times that the second request has been retried is the number of times that all nodes in the distributed system have retried to process the second request.

[0137] For example, the first node updates the first request's retried number of times in the first information table in addition to updating the first request's remaining number of times of retries in the first information table after receiving the second feedback information sent by the second node. For example, the first request's remaining number of times of retries in the first information table is deducted by the total number of times of retries in the second feedback information, and the first request's retried number of times in the first information table is increased by the total number of times of retries in the second feedback information. Similarly, the first request's retried number of times in the first information table is updated in the same way.

[0138] It should be noted that the first node updates the first request's retried number of times in the first information table in addition to updating the first request's remaining number of times of retries in the first information table when the first request's remaining number of times of retries is greater than zero, and the first request sent by the first node to the second node carries the first request's retried number of times in addition to the first request's remaining number of times of retries, so that the second node updates the first request's remaining number of times of retries and the first request's retried number of times in the second information table according to the received first request, and if the second node retries the first request, the second node updates the first request's remaining number of times of retries and the first request's retried number of times in the second information table, and then carries the updated first request's remaining number of times of retries and the first request's retried number of times to a downstream node of the second node, and so on, until the first request is successfully processed or the remaining number of times of retries is zero.

[0139] The second node updates the first request's remaining number of times of retries and the first request's retried number of times in the second information table, specifically: deducting the number of times of retries of the first request by the second node from the first request's remaining number of times of retries, and adding the number of times of retries of the first request by the second node to the first request's retried number of times.

[0140] Based on this, the first priority of the first request is obtained based on the first request's remaining number of times of retries and the first request's retried number of times, and the second priority of the second request is obtained based on the second request's remaining number of times of retries and the second request's retried number of times.

[0141] For example, the first request's remaining number of times of retries and the first request's retried number of times are added to obtain the maximum number of times of retries of the first request, and the first request's remaining number of times of retries is divided by the maximum number of times of retries to obtain the first ratio of the first request; the second request's remaining number of times of retries and the second request's retried number of times are added to obtain the maximum number of times of retries of the second request, and the second request's remaining number of times of retries is divided by the maximum number of times of retries to obtain the second ratio of the second request; based on this, if the first ratio is greater than the second ratio, the first priority is higher than the second priority; if the first ratio is less than the second ratio, the second priority is higher than the first priority. That is, the greater the proportion of the remaining number of times of retries in the maximum number of times of retries, the higher the priority of the request.

[0142] For example, taking the example of a distributed system containing four nodes: A, B, C, and D, node B receives second feedback information sent by node C, the second feedback information containing the total number of retries of node C and its downstream nodes for the video acquisition request, and node B updates the remaining number of retries and the number of retries of the video acquisition request in the local information table according to the total number of retries in the second feedback information. If the second feedback information indicates that the second node successfully processes the first request but node B fails to execute the local task of the video acquisition request according to the request processing result in the second feedback information, or if the second feedback information indicates that the second node fails to process the first request, then node B prepares to retry processing the first request. At this time, it is first determined whether the remaining number of retries of the video acquisition request in the local information table is greater than zero or equal to zero. If the remaining number of retries is greater than zero, it indicates that node B can still retry processing the video acquisition request. At this time, node B re-executes the local task of the video acquisition request, reduces the remaining number of retries of the video acquisition request in the local information table by one and increases the number of retries of the video acquisition request in the local information table by one, and sends a video acquisition request carrying the remaining number of retries and the number of retries to call node C and its downstream nodes to process the video acquisition request. Node C updates the remaining number of retries and the number of retries of the video acquisition request in the local information table according to the received video acquisition request. The remaining number of retries of the video acquisition request in the local information table is used for retry judgment by node C when processing the video acquisition request fails. This process continues until node C successfully processes the video acquisition request and sends second feedback information to node B (node B updates the local information table again), or until the number of retries of node C for processing the video acquisition request fails to zero.

[0143] In an implementation manner, the first priority of the first request can be carried in the first request, and the first node can obtain the first priority of the first request from the first request after receiving the first request. The second priority of the second request is carried in the second request, and the first node can obtain the second priority of the second request from the second request after receiving the second request.

[0144] Further, when the first node sends the first request to the second node, the first priority of the first request is determined by updating the remaining number of retries of the first request in the first information table, the first priority is carried in the first request and sent to the second node. When the first node sends the second request to the second node, the second priority of the second request is determined by updating the remaining number of retries of the second request in the first information table, and the second priority is carried in the second request and sent to the second node.

[0145] In another implementation, the first priority of the first request is obtained based on the remaining retry times of the first request in the first information table, and the second priority of the second request is obtained based on the remaining retry times of the second request in the first information table. For example, after the first node updates the remaining retry times and the retry times of the first request in the first information table, the first node calculates a first ratio using the remaining retry times and the retry times of the first request in the first information table, and after the first node updates the remaining retry times and the retry times of the second request in the first information table, the first node calculates a second ratio using the remaining retry times and the retry times of the second request in the first information table, thereby determining the first priority of the first request and the second priority of the second request.

[0146] In an implementation, the second request further carries at least one retry parameter of the second request; the retry parameter includes: a retry timeout timestamp, a retry condition, a retry interval duration.

[0147] In the case that the current time is greater than the retry timeout timestamp, the first node prohibits retry processing of the second request. Specifically, after the first node receives the second request, the first node updates the retry timeout timestamp carried in the second request before sending the second request to the second node. Specifically, the second node deducts the duration for which the first node processes the second request from the retry timeout timestamp in the second request, so as to prompt the second node to process the second request before the received retry timeout timestamp. If the second node finds that the current time has exceeded the retry timeout timestamp when retry processing the second request, it means that the processing has timed out, and the second request is no longer retry processed.

[0148] The first node retries processing of the second request at least in the case that the retry condition is met. The retry condition can be that the error code of the second request is a specific error code, or the error level of the second request is a specific error level. For example, the retry condition is that the HTTP return code corresponding to the second request is 503 or 504.

[0149] The first node retries processing of the second request after an interval of the retry interval duration. For example, when the first node retries processing of the second request, it does not immediately start retrying, but waits for an interval of the retry interval duration before retrying processing of the second request.

[0150] In an implementation, before step 509, the first node can perform the following processing, as shown in step 508: Figure 7

[0151] Step 513: Determine whether the second node can perform retry for the first request according to the request type of the first request. If the second node can perform retry for the first request, perform step 514; if the second node cannot perform retry for the first request, perform step 515. ​

[0152] The request type of the first request is determined based on a service function implemented by the first request.

[0153] Step 514: Set the retry attribute of the second node for the first request as the first attribute.

[0154] The first attribute is used to instruct the second node to retry processing the first request in the case of failure in processing the first request. Specifically, the second node retries processing the first request according to the remaining retry times of the first request in the updated second information table.

[0155] Specifically, the retry attribute is configured in the first request, and thus the first node carries the first attribute in the first request to provide to the second node.

[0156] Step 515: Set the retry attribute of the second node for the first request as the second attribute.

[0157] The second attribute is used to instruct the second node not to retry processing the first request in the case of failure in processing the first request. That is, even if the second node fails in processing the first request, the second node does not retry processing the first request, but sends corresponding feedback information to the upstream node, i.e., the first node, and the first node retries processing the first request.

[0158] Specifically, the retry attribute is configured in the first request, and thus the first node carries the second attribute in the first request to provide to the second node.

[0159] That is, the first node can instruct the downstream second node whether to retry processing the first request based on the request type of the first request, thereby meeting the actual needs of the service function.

[0160] It should be noted that when the first node processes the second request, in the process of sending the second request to the second node to call the second node to process the second request, before the first node sends the second request to the second node, the first node can also determine whether the second node can perform retry for the second request according to the request type of the second request. If the second node can perform retry for the second request, the first attribute is set for the retry attribute of the second node for the second request, and the first attribute indicates that the second node retries processing the second request in the case of failure of processing the second request. If the second node cannot perform retry for the second request, the second attribute is set for the retry attribute of the second node for the second request, and the second attribute indicates that the second node does not retry processing the second request in the case of failure of processing the second request. Based on this, the first attribute or the second attribute is carried in the second request, and after the second request is sent to the second node, the second node retries processing the second request according to the first attribute in the case of failure of processing the second request, or the second node does not retry processing the second request according to the second attribute in the case of failure of processing the second request.

[0161] That is, whether it is the case of retrying processing the request or the scene of processing the request for the first time, the retry attribute of the downstream node for the request is set before the request is sent to the downstream node, to indicate whether the downstream node performs retry for the request if the downstream node fails to process the request, to meet the actual demand of the business function corresponding to the request.

[0162] In another implementation mode, before step 509, the first node can also determine whether the second node can perform retry for the first request according to the request type of the first request. If the second node can perform retry for the first request, step 509 is performed, and if the second node cannot perform retry for the first request, the remaining retry times of the first request carried in the first request are set to zero, to notify all downstream nodes to update the remaining retry times of the first request in the local information table to zero, so that the downstream nodes do not perform retry for the first request.

[0163] That is, the first node can indicate whether all downstream nodes retry processing the first request based on the request type of the first request, to meet the actual demand of the business function.

[0164] The above introduces a control method for performing retry of a distributed system provided by an embodiment of the application, and the following introduces an apparatus for performing the control method for performing retry of the distributed system.

[0165] Please refer to Figure 8 , Figure 8A structure diagram of a control device for retry of a distributed system is provided in the embodiments of the present application. The device is deployed on a first node of the distributed system, and the first node is any node in the distributed system. The device can include the following units:

[0166] A feedback receiving unit 801 is configured to receive second feedback information sent by a second node. The second node is a downstream node adjacent to the first node in the distributed system. The second feedback information includes a total retry number of the first request by the second node and downstream nodes thereof.

[0167] An information updating unit 802 is configured to update a first information table of the first node according to the second feedback information. The first information table includes at least a remaining retry number of the first request. The remaining retry number is a remaining number of times that all nodes in the distributed system can re-process the first request.

[0168] A remaining judging unit 803 is configured to judge whether the remaining retry number of the first request in the first information table is greater than or equal to zero in response to a failure of the first node to process the first request. In a case where the remaining retry number of the first request is greater than zero, a retry processing unit 804 is triggered. In a case where the remaining retry number of the first request is equal to zero, an upward feedback unit 805 is triggered.

[0169] The retry processing unit 804 is configured to re-execute a local task corresponding to the first request, update the remaining retry number of the first request in the first information table, and send the first request to the second node. The first request carries at least the remaining retry number of the first request, so that the second node updates a second information table of the second node according to the received first request and processes the first request. The second information table includes at least the remaining retry number of the first request.

[0170] The upward feedback unit 805 is configured to send first feedback information to a third node. The first feedback information includes a total retry number of the first request by the first node and downstream nodes thereof. The third node is an upstream node adjacent to the first node in the distributed system.

[0171] It can be seen that, in the embodiments, the remaining retry number is updated based on the total retry number fed back by a downstream node, and it is determined whether to retry according to the remaining retry number. The total retry number is also fed back to an upstream node. Thus, the remaining retry number is continuously updated through coordination of the upstream and downstream nodes, so that the control of the retry number is realized, the retry storm on the downstream node is avoided, and the processing pressure of the downstream node is reduced.

[0172] In an implementation manner, the feedback receiving unit 801 is further configured to receive a second request sent by the third node, wherein the second request carries at least a remaining retry number of the second request.

[0173] The information updating unit 802 is further configured to update the remaining retry number of the second request in the first information table according to the remaining retry number of the second request.

[0174] The retry processing unit 804 is further configured to process the second request.

[0175] In an implementation manner, in a case where the first request is different from the second request, an execution order between the first request and the second request is determined based on a first priority of the first request and a second priority of the second request.

[0176] The first priority of the first request is related to the remaining retry number of the first request, and the second priority of the second request is related to the remaining retry number of the second request.

[0177] In an implementation manner, the first information table further comprises a retried number of the first request and a retried number of the second request; the retried number of the first request is a number of times that all nodes in the distributed system have retried processing the first request; and the retried number of the second request is a number of times that all nodes in the distributed system have retried processing the second request.

[0178] The first priority of the first request is obtained based on the remaining retry number and the retried number of the first request, and the second priority of the second request is obtained based on the remaining retry number and the retried number of the second request.

[0179] In an implementation manner, the first priority of the first request is carried in the first request, and the second priority of the second request is carried in the second request.

[0180] Or,

[0181] The first priority of the first request is obtained based on the remaining retry number of the first request in the first information table, and the second priority of the second request is obtained based on the remaining retry number of the second request in the first information table.

[0182] In an implementation manner, the second request further carries at least one retry parameter of the second request; the retry parameter comprises a retry timeout timestamp, a retry condition, and a retry interval duration.

[0183] The first node prohibits retrying processing the second request in a case that a current time is greater than the retry timeout timestamp.

[0184] The first node retries processing the second request at least in a case that the retry condition is met.

[0185] The first node retries processing the second request after the retry interval time duration.

[0186] In an implementation manner, before the retry processing unit 804 sends the first request to the second node, the retry processing unit 804 is further configured to: determine whether the second node can perform retry for the first request according to a request type of the first request; if the second node can perform retry, set a retry attribute of the second node for the first request as a first attribute; the first attribute is used to instruct the second node to retry processing the first request in a case that processing the first request fails; if the second node cannot perform retry, set the retry attribute of the second node for the first request as a second attribute; the second attribute is used to instruct the second node not to retry processing the first request in a case that processing the first request fails.

[0187] In an implementation manner, when the retry processing unit 804 updates the remaining retry times of the first request in the first information table, the retry processing unit 804 is specifically configured to: decrease the remaining retry times of the first request in the first information table by one.

[0188] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the foregoing, which will not be described in detail here.

[0189] In the embodiments of this application, an electronic device is also provided. Referring to FIG. 8, Figure 9 The electronic device shown in FIG. 8 only shows a structure diagram suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application can include but is not limited to a fixed terminal such as a desktop computer or a server, etc. Figure 9 The electronic device shown in FIG. 8 is only an example, and should not bring any limitation to the functions and use range of the embodiments of this application.

[0190] As shown in FIG. 8, Figure 9As shown, the electronic device can include a processing device (e.g., a central processor, a graphics processor, etc.) 901 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage device 908 into a random access memory (RAM) 903. In a state where the electronic device is powered on, various programs and data required for operation of the electronic device are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0191] Generally, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a memory card, a hard disk, etc.; and communication devices 909. The communication devices 909 can allow the electronic device to communicate wirelessly or wired with other devices to exchange data. Although Figure 9 The electronic device is shown with various devices, but it should be understood that all of the shown devices are not required to be implemented or possessed. More or fewer devices can alternatively be implemented or possessed.

[0192] Figure 9 The electronic device shown can be deployed as a node in a distributed system, and the electronic device as a first node executes a computer program in the memory by the processor to implement any one of the control methods for retrying of a distributed system provided by the embodiments of the present application.

[0193] The embodiments of the present application also provide a computer program product including computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any one of the control methods for retrying of a distributed system provided by the embodiments of the present application.

[0194] The embodiments of the present application also provide a computer storage medium carrying one or more computer programs, which, when executed on an electronic device, can cause the electronic device to implement any one of the control methods for retrying of a distributed system provided by the embodiments of the present application.

[0195] The embodiments of the present application also provide a distributed system, which is described with reference to Figure 10 As shown, the distributed system at least includes a first node 1001, a second node 1002, and a third node 1003, and each node can be Figure 9 As shown, the electronic device (such as Figure 1 The server 200 in the distributed system shown, or each node can beFigure 8 The apparatus (e.g. Figure 4 The server 200 shown in the microservice in the distributed system). Specifically:

[0196] The first node 1001 receives second feedback information sent by the second node 1002; the second node 1002 is a downstream node adjacent to the first node 1001 in the distributed system; the second feedback information contains the total number of retries of the first request by the second node 1002 and its downstream nodes;

[0197] The first node 1001 updates the first information table of the first node 1001 according to the second feedback information, and the first information table at least contains the remaining number of retries of the first request; the remaining number of retries is the remaining number of times that all nodes in the distributed system can re-process the first request;

[0198] The first node 1001 judges whether the remaining number of retries of the first request in the first information table is greater than or equal to zero in response to the failure of the first node 1001 to process the first request;

[0199] In the case where the remaining number of retries of the first request is greater than zero, the first node 1001 re-executes the local task corresponding to the first request; updates the remaining number of retries of the first request in the first information table and sends the first request to the second node 1002; the first request at least carries the remaining number of retries of the first request, so that the second node 1002 updates the second information table of the second node 1002 according to the received first request and processes the first request, and the second information table at least contains the remaining number of retries of the first request;

[0200] In the case where the remaining number of retries of the first request is equal to zero, the first node 1001 sends first feedback information to the third node 1003, and the first feedback information contains the total number of retries of the first request by the first node 1001 and its downstream nodes; the third node 1003 is an adjacent upstream node of the first node 1001 in the distributed system.

[0201] The following illustrates the distributed system provided by the embodiments of the present application:

[0202] First, in the process of processing requests in the distributed system, the node carries some retry strategy information in the meta information of the request, each node retries according to the retry strategy information, and the retry result (i.e. feedback information) fed back by the downstream node is corrected again, and the last retry result (i.e. feedback information, which can also have request processing result) is fed back to the upstream node, so as to ensure that the entire link meets the request retry strategy.

[0203] The retry strategy information in the request meta-information includes: retryable times (i.e., the remaining retry times in the foregoing), retry timeout timestamp, retry condition, retry interval duration, retry priority factor (i.e., the priority in the foregoing), and the like. Details are as follows:

[0204] The retryable times refer to how many times of retries are allowed. The starting node initiating the request sets the initial first retryable times (i.e., the maximum retry times), and the retryable times of the entire link (i.e., the sum of the retryable times of all nodes in the entire link) must be less than or equal to the first retryable times (set when the request is initiated).

[0205] The retry condition refers to that retries can be performed only when certain error codes or error levels are present;

[0206] The retry timeout timestamp refers to that retries must be completed before this timestamp, otherwise retries are not performed;

[0207] The retry priority factor is specifically: the retryable times divided by the maximum retry times. Of course, it can also be adjusted by the node, and the retry priority factor decreases with the increase of the retryable times. The greater the factor, the higher the retry priority level. Based on this, in the case of high node load, requests with a greater factor are preferentially retried, and the factor decreases with the increase of the retryable times.

[0208] Secondly, each node creates a retry information table (i.e., the local information table in the foregoing) of the request in its memory, and this table stores the retry information of the request, including: retryable times, retried times, retry condition, retry timeout timestamp, and the like.

[0209] Initialization of the retry information in the retry information table: the starting node sets the retry strategy information described above in the request meta-information, and sets different retry attributes of downstream nodes according to different request types to indicate whether the downstream nodes can perform retries.

[0210] Retry result in the response (request result): the node carries the retried times (here, the retried times refer to the total retry times in the foregoing, i.e., the sum of the retryable times of all downstream nodes) in the result meta-information of the request. The node feeds back the retry result to the upstream node after processing the request regardless of success or failure. For example, if the node has retried 2 times, it should tell the upstream node that it has retried 2 times in the result (i.e., the feedback information in the foregoing).

[0211] Offloading information: the node parses the retry strategy information from the request meta-information after receiving the request, and stores it in the request retry information table, which is referred to as offloading information.

[0212] Retry policy information is passed to downstream nodes: when the current node calls downstream nodes, the requested retry policy information is carried in the request and passed to downstream nodes.

[0213] If the current node wants the downstream node to retry: the retryable times in the request at this time is the updated retryable times based on the total retry times. This corrected retryable times is passed to the downstream node. In order to try to retry as close to the failure node as possible, this can reduce the transmission of feedback information and reduce the length of the retry link. Therefore, the retry policy information needs to be passed to each node of the link.

[0214] If the current node does not want the downstream node to retry, but the current node itself retries, the retryable times sent to the downstream node is set to 0, such as setting the retry attribute of the downstream node as the second attribute to indicate that the downstream node does not perform retry for the request, but the local retry information table needs to be updated in order to pass the retry policy information to its downstream node in the future.

[0215] Feedback information of retry result: after the node processes the request, whether it succeeds or fails, or retries, it needs to return the request processing result to the upstream node. The feedback information contains the total retry times and the request processing result, such as the video data or video list obtained.

[0216] The upstream node corrects the retry policy information of the current node for this request according to the feedback information of the downstream node: updates the retryable times of the request in the retry information table, the retryable times in the table is the retryable times in the table minus the total retry times in the feedback information, the already retried times in the table is the already retried times in the table plus the total retry times of the downstream. If the result of the retryable times minus the total retry times is greater than 0, the current node continues to initiate retry to the downstream node, and at the same time, the updated retryable times is passed to the downstream; if the retryable times minus the total retry times is equal to 0, the current result does not retry any more, and the updated total retry times (the retry times of the current node plus the retry times of the downstream node) is returned to the upstream node, and the upstream node makes a judgment again.

[0217] Priority: if the node load is high, different requests can be prioritized according to the retry priority factor of the request, and the request with higher priority is processed first. If the request is retried multiple times, the request priority factor will be reduced after each retry. If a request fails multiple times, the probability of its successful retry will also gradually decrease. Therefore, for extreme cases, requests with fewer retries should be prioritized to ensure that such requests are retried first.

[0218] It can be seen that the embodiment can control the retry strategy through the forward call chain of the request, and adjust the retry strategy in combination with the chain returned by the request. Meanwhile, in the end case, the retry priority factor can improve the success rate of the request in the case of protecting the node. The pre-post coordination control scheme controls the number of retries and time by the upstream node, and corrects in combination with the feedback of the downstream node, solves the problem that the nodes cannot coordinate the strategy, and avoids that the downstream node exceeds its maximum processing capacity.

[0219] Reference Figure 11 As shown in the figure, the distributed system includes M1 module, M2 module, M3 module and M4 module. It is assumed that the maximum number of retries set for a certain request is 10 times, the retry timeout timestamp is 20240513101010, and the retry condition is to retry when the HTTP return code is 503 and 504, and the retry interval duration is 1 second.

[0220] The M1 module is the initiator of the request, belongs to the starting node, and is responsible for initializing the retry strategy information. A field with the prefix "mg-x-retry-" is added to the meta information of the request to describe the retry strategy information of the request, including the initial retryable number such as 10 times, the retry timeout timestamp, the retry condition, the retry interval duration, the retry priority factor and the like.

[0221] The M2 module unloads these information, that is, parses the retry strategy information from the request meta information and stores it in the request retry information table. The retry information table includes:

[0222] The retryable number is 10 times;

[0223] The number of retries is 0 times (assuming that it has not been retried yet);

[0224] The retry condition is to retry when the request error code is 503 and 504.

[0225] The retry timeout is 1 second. If the current node needs some additional time to process the error request, the retry timeout time passed down by the upstream is reduced by 1 second.

[0226] The retry priority factor is 10 / 10=1 if no retry has been performed, and the priority is 9 / 10 if it has been retried once. The denominator is reduced by 1 each time the retry is performed. The smaller the value, the lower the priority. If the node load is high, the request with higher priority will be selected for retry.

[0227] M3 module loads retry information to be passed to downstream nodes, encapsulates retry strategy information into meta information of request sent to downstream nodes. At this time, the retryable times should be equal to the retryable times in the request retry table, and the retryable times in the table are determined based on the updated retryable times of the upstream node. The updated retryable times of the upstream node are: the original retryable times in the table of the upstream node minus the retryable times of the upstream node (or the maximum retryable times minus the updated retryable times in the table). If the node does not want the downstream node to retry, the retryable times are set to 0, indicating that the downstream node cannot retry.

[0228] M4 module: After the downstream node processes the request, it will feedback the retry result information. After the current node receives the information, it will correct the information in the request retry information table. For example, the downstream node has retried 3 times, so the request retry times in the table are added by 3 times. If the maximum retryable times minus the retryable times in the table at this time are greater than 0, and the request error code returned by the downstream node is equal to 503 or 504, the retry is continued.

[0229] The following takes a request through A, B, C, and D4 nodes as an example:

[0230] Suppose that the maximum retryable times are set to 5 times when the request is initialized.

[0231] Based on the link of A->B->C->D, the D node does not allow retry, and the retryable times of the request retry table of each node in the A, B, and C nodes are 5, and the retryable times are 0.

[0232] Suppose that the D node fails, and returns 503 error to the node C.

[0233] The node C request will fail, and the node C will continuously retry 5 times. If all of them fail, the node C will return the retry result of total retry 5 times to the node B.

[0234] The node B receives the total retry 5 times, queries the retryable 5 times in the table, updates the retryable times to 0, i.e. 5 minus 5 is 0, so the node B does not retry any more. The B feeds back the information of 5 times of retry to the A.

[0235] The node A receives the total retry 5 times, queries the retryable 5 times in the table, updates the retryable times to 0, i.e. 5 minus 5 is 0, so the node A does not retry any more. The node A returns the result of request processing failure to the request initiation end (i.e. request end, such as terminal 100).

[0236] If node A does not want the downstream nodes to retry, the number of retryable times of the request is set to 0 when the first request is transmitted to node B. Then, node B and node C will not retry, and when the request to node D fails, node C transmits the failure information to node B, and node B continues to transmit to node A. After node A receives the failure information of the request, it checks the number of retryable times in the table, which is 5 times, and the number of retried times, which is 0. Then, node A retries, and the number of retried times in the table is incremented by 1.

[0237] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the apparatus embodiments provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0238] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0239] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0240] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A control method for retry execution by a distributed system, characterized by, The method is applied to a first node, and the first node is any node in the distributed system, and the method comprises the following steps: receiving second feedback information sent by a second node; the second node is a downstream node adjacent to the first node in the distributed system; the second feedback information comprises a total number of retries of the first request by the second node and downstream nodes of the second node; updating a first information table of the first node according to the second feedback information; the first information table comprises at least a remaining number of retries of the first request; the remaining number of retries is a remaining number of times that all nodes in the distributed system can re-process the first request; in response to a failure of the first node to process the first request, determining whether the remaining number of retries of the first request in the first information table is greater than or equal to zero; in a case where the remaining number of retries of the first request is greater than zero, re-executing a local task corresponding to the first request; updating the remaining number of retries of the first request in the first information table and sending the first request to the second node; the first request carries at least the remaining number of retries of the first request, so that the second node updates a second information table of the second node according to the received first request and processes the first request; the second information table comprises at least the remaining number of retries of the first request; in a case where the remaining number of retries of the first request is equal to zero, sending first feedback information to a third node; the first feedback information comprises a total number of retries of the first request by the first node and downstream nodes of the first node; the third node is an upstream node adjacent to the first node in the distributed system.

2. The method of claim 1, wherein, The method further comprises the following steps: receiving a second request sent by the third node; the second request carries at least a remaining number of retries of the second request; updating the remaining number of retries of the second request in the first information table according to the remaining number of retries of the second request; processing the second request.

3. The method of claim 2, wherein, in a case where the first request and the second request are different, an execution order between the first request and the second request is determined based on a first priority of the first request and a second priority of the second request; the first priority of the first request is related to the remaining number of retries of the first request; the second priority of the second request is related to the remaining number of retries of the second request.

4. The method of claim 3, wherein, the first information table further comprises a number of retries of the first request and a number of retries of the second request; the number of retries of the first request is a number of times that all nodes in the distributed system have retried to process the first request; the number of retries of the second request is a number of times that all nodes in the distributed system have retried to process the second request; wherein the first priority of the first request is obtained based on the remaining number of retries and the number of retries of the first request, and the second priority of the second request is obtained based on the remaining number of retries and the number of retries of the second request.

5. The method of claim 3, wherein, The first priority of the first request is carried in the first request; and the second priority of the second request is carried in the second request. Or, The first priority of the first request is obtained based on the remaining retry times of the first request in the first information table; and the second priority of the second request is obtained based on the remaining retry times of the second request in the first information table.

6. The method of claim 2, wherein, The second request further carries at least one retry parameter of the second request; the retry parameter comprises a retry timeout timestamp, a retry condition, and a retry interval duration. The first node prohibits retry processing the second request in a case where a current time is greater than the retry timeout timestamp. The first node retries processing the second request in a case where at least the retry condition is met. The first node retries processing the second request after the retry interval duration.

7. The method according to claim 1 or 2, characterized in that, Before sending the first request to the second node, the method further comprises: determining whether the second node can perform retry for the first request according to a request type of the first request; setting a retry attribute of the second node for the first request as a first attribute if the second node can perform retry; the first attribute is used to instruct the second node to retry processing the first request in a case where processing the first request fails; setting the retry attribute of the second node for the first request as a second attribute if the second node cannot perform retry; the second attribute is used to instruct the second node not to retry processing the first request in a case where processing the first request fails.

8. The method of claim 1 or 2, wherein, updating the remaining retry times of the first request in the first information table comprises: subtracting the number of times of retry processing the first request by the first node from the remaining retry times of the first request in the first information table.

9. A distributed system, characterized by At least comprising: a first node, a second node and a third node; wherein: The first node receives second feedback information sent by the second node; the second node is a downstream node adjacent to the first node in the distributed nodes; the second feedback information contains the total retry times of the first request by the second node and its downstream nodes; The first node updates a first information table of the first node according to the second feedback information, the first information table at least containing the remaining retry times of the first request; the remaining retry times are the remaining times that all nodes in the distributed system can re-process the first request; The first node judges whether the remaining retry times of the first request in the first information table are greater than or equal to zero in response to the first node failing to process the first request. In a case that the remaining retry times of the first request is greater than zero, the first node re-executes the local task corresponding to the first request; updates the remaining retry times of the first request in the first information table and sends the first request to the second node; the first request carries at least the remaining retry times of the first request, so that the second node updates a second information table of the second node according to the received first request and processes the first request, the second information table at least contains the remaining retry times of the first request; In a case that the remaining retry times of the first request is equal to zero, the first node sends first feedback information to a third node, the first feedback information contains the total retry times of the first request by the first node and downstream nodes of the first node; the third node is an adjacent upstream node of the first node in the distributed system.

10. An electronic device, deployed as a first node in a distributed system, characterized in that, The electronic device comprises at least one processor and a memory connected with the processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program, so that the electronic device can implement the control method for performing retry in the distributed system according to any one of claims 1 to 8.

Images