System for requesting data retries

Abstract

Embodiments of the present disclosure disclose a system for requesting data retry. A specific implementation of the system includes an application server, a retry service container, a retry service manager and a target system, wherein: the application server is configured to collect service data request information and request execution status information; the application server is configured to send the service data request information to the retry service container; the retry service container is configured to send the service data request information to the retry service manager; the retry service manager is configured to send a retry request command to the retry service container in response to initiating a retry of the service data request information; the retry service container is configured to control the application server to initiate the retry to the target system; the target system is configured to send a retry result to the application server; and the application server is configured to send the retry result to the retry service container for storage. The implementation can ensure consistency of data among various systems in a distributed system.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of data retry technology, and more specifically to a data retry request system. Background Technology

[0002] Since network instability or jitter often causes API requests to fail, retrying the request can ensure success in such cases. Currently, retries in distributed scenarios can be implemented by calling new threads, creating a failure log table, or using some mainstream distributed transaction frameworks.

[0003] However, the following technical problems often arise when using the above method:

[0004] First, because the status of retry calls is unknown, data loss may occur when the system restarts, or continuous calls may occur even when the call cannot be successfully made, resulting in the consumption of application server resources.

[0005] Second, it is difficult to maintain the normal operation of the system if the application server is unable to send a retry request or if the retry fails.

[0006] Third, because the retry result is determined manually, retry cannot be initiated in a timely manner. At the same time, the possibility of false alarms is relatively high in manual determination, which makes it impossible to accurately determine the retry result. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure propose a request data retry system to address one or more of the technical problems mentioned in the background section above.

[0009] Some embodiments of this disclosure provide a request data retry system, which includes: an application server, a retry service container, a retry service manager, and a target system. The application server is configured to collect service data request information and request execution status information corresponding to the service data request information, wherein the request execution status information includes successful execution and failed execution. The application server is communicatively connected to the retry service container, and is configured to asynchronously send the service data request information to the retry service container in response to the request execution status information indicating failure. The retry service container is communicatively connected to the retry service manager, and is configured to store the received service data request information. The system stores and displays the received service data request information and sends it to the retry service manager. The retry service manager is configured to determine whether to retry the service data request information, and in response to retrying the service data request information, sends a retry request command for the service data request information to the retry service container. The target system is communicatively connected to the application server, wherein the retry service container is configured to, in response to receiving the retry request command, control the application server to initiate a retry for the service data request information to the target system. The target system is configured to send the generated retry result to the application server. The application server is configured to send the received retry result to the retry service container for storage.

[0010] In some embodiments, the system further includes a program hosting server, wherein:

[0011] The aforementioned program hosting server is communicatively connected to the aforementioned retry service container, wherein the aforementioned retry service container is configured to send retry results to the aforementioned program hosting server, wherein the aforementioned retry results include retry success and retry failure;

[0012] The retry service container is configured to send the service data request information to the program hosting server in response to determining that the retry result is a retry failure.

[0013] The aforementioned program hosting server is configured to retry the received service data request information.

[0014] In some embodiments, the application server includes a communication interface and a listener installed at the target network layer of the communication interface, wherein:

[0015] The aforementioned communication interface is configured to collect service data request information and request execution status information corresponding to the aforementioned service data request information;

[0016] The listener is connected to the retry service container. The listener is configured to asynchronously send the service data request information to the retry service container in response to the request execution status information being deemed as failure.

[0017] In some embodiments, the listener includes a filter, wherein the filter is configured to intercept the service data request information in response to the request execution status information being an execution failure, and to asynchronously send the service data request information to the retry service container.

[0018] In some embodiments, the retry service container includes a first predetermined number of databases storing data in key-value pairs and a second predetermined number of database statement parsers, wherein:

[0019] The aforementioned first predetermined number of databases storing data in key-value pairs are configured to support information insertion, querying, and updating based on the primary key;

[0020] The aforementioned second predetermined number of database statement parsers are configured to perform queries based on pre-acquired query information.

[0021] In some embodiments, the retry service manager is further configured to update the received service data request information to obtain updated service data request information.

[0022] The retry service manager is configured to determine whether to retry the updated service data request information. In response to retrying the updated service data request information, the retry service manager sends a retry request command for the updated service data request information to the retry service container.

[0023] In some embodiments, the retry service manager includes a pre-trained detector, wherein the retry service manager has functions for scheduled retries, retry result verification, and retry effect analysis, and the pre-trained detector is configured to detect abnormal retry states.

[0024] In some embodiments, the pre-trained detector described above is obtained in the following manner:

[0025] Obtain the retry frequency set and data characteristics, whereby the data characteristics include: date, time, network status, database load, and data processing scheme;

[0026] Clustering the above set of retry frequencies yields the classification results;

[0027] The retry frequency corresponding to the classification that meets the preset conditions in the above classification results is determined as the negative sample set.

[0028] Each negative sample in the above negative sample set is labeled to obtain a labeled negative sample set.

[0029] The negative samples that meet the preset conditions in the above negative sample set are determined as the target negative sample set;

[0030] Data augmentation is performed on the aforementioned target negative sample set to obtain the augmented sample set;

[0031] Based on the enhanced sample set and the data features described above, the initial detector is trained to obtain a fully trained detector.

[0032] In some embodiments, the pre-trained detector includes a model training module, wherein the model training module is configured to train an initial detector based on the enhanced sample set and the data features.

[0033] In some embodiments, the above-mentioned model training module is further configured to:

[0034] Random sampling is performed from the aforementioned enhanced sample set to generate a sub-data set;

[0035] Based on the above sub-data sets, construct multiple decision trees;

[0036] Each sub-data set and the data features mentioned above are input into the aforementioned decision trees for training.

[0037] The above embodiments of this disclosure have the following beneficial effects: By retrying failed requests for data through the request data retry system of some embodiments of this disclosure, the data consistency of various systems in a distributed system can be maintained. Specifically, the reason why system data is difficult to maintain consistency is that, due to the unknown status of the retry call, data loss occurs when the system restarts. Furthermore, continuous calls without successful execution consume application server resources. Based on this, in the request data retry system of some embodiments of this disclosure, the application server can collect service data request information and request execution status information corresponding to the service data request information. In response to the request execution status information indicating execution failure, the application server can asynchronously send the service data request information to the retry service container. The retry service container can store and display the received service data request information and send it to the retry service manager. The retry service manager can determine whether to initiate a retry for the service data request information, and in response to initiating a retry for the service data request information, send a retry request command for the service data request information to the retry service container. The retry service container, upon receiving the retry request command, controls the application server to initiate a retrieval to the target system for the service data request information. The target system can send the generated retry result to the application server. The application server can then send the received retry result to the retry service container for storage. This allows for timely acquisition of the request execution status information corresponding to the service data request information, enabling retries of failed data. Even during system restarts, the retry status can be determined based on the request execution status information, effectively preventing data loss and ensuring data consistency across systems. Attached Figure Description

[0038] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0039] Figure 1 This is an architectural diagram of some embodiments of the present disclosure that can be configured as exemplary systems therein;

[0040] Figure 2 This is a schematic diagram of the structure of one embodiment of a request data retry system according to some embodiments of the present disclosure;

[0041] Figure 3 This is a flowchart of a request data retry system according to some embodiments of the present disclosure;

[0042] Figure 4 This is a schematic diagram of the structure of yet another embodiment of the request data retry system according to some embodiments of the present disclosure;

[0043] Figure 5 This is a schematic diagram illustrating the operation of an application server according to some embodiments of this disclosure;

[0044] Figure 6 This is a schematic diagram of the structure of a retry service manager according to some embodiments of the present disclosure. Detailed Implementation

[0045] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0046] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0047] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0048] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0049] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0050] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] Figure 1 An exemplary system architecture 100 for a request data retry system to which some embodiments of the present disclosure may be applied is shown.

[0052] like Figure 1As shown, system architecture 100 may include application server 101, retry service container 102, retry service manager 103, target system 104, application hosting server 105, network 106, network 107, and network 108. Network 106 serves as the medium for providing a communication link between application server 101 and retry service container 102. Network 107 serves as the medium for providing a communication link between retry service container 102 and retry service manager 103, and between retry service container 102 and application hosting server 105. Network 108 serves as the medium for providing a communication link between application server 101 and target system 104. Networks 106, 107, and 108 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0053] Application server 101 can interact with retry service container 102 via network 106 to asynchronously send the aforementioned service data request information to the retry service container.

[0054] The retry service container 102 can interact with the program hosting server 105 via network 106 to send the retry result to the program hosting server.

[0055] The retry service container 102 can interact with the retry service manager 103 via network 107 to send the received service data request information to the retry service manager.

[0056] Application server 101 can interact with target system 104 via network 108 to initiate a retry to the target system.

[0057] It should be understood that Figure 1 The number of application servers, retry service containers, retry service managers, target systems, application hosting servers, and networks shown in the diagram is merely illustrative. Depending on implementation needs, any number of application servers, retry service containers, retry service managers, target systems, application hosting servers, and networks can be included.

[0058] Continue to refer to Figure 2 This illustrates a structural diagram 200 of an embodiment of the request data retry system provided in this disclosure. Figure 2 As shown, the request data retry system in this embodiment may include: application server 1, retry service container 2, retry service manager 3, and target system 4.

[0059] In some embodiments, the application server 1 described above may be a device capable of sending, receiving, and providing application services. The application server may include, but is not limited to, a microcontroller, a microcomputer, a chip, etc. Here, the application server is configured to collect service data request information and request execution status information corresponding to the service data request information. The request execution status information includes successful execution and execution failure. The application server is configured to receive retry results sent by the target system and send the received retry results to the retry service container for storage.

[0060] In some embodiments, the retry service container 2 can be a high-capacity, distributed compressed data storage container. Here, the application server is communicatively connected to the retry service container, and the application server is configured to asynchronously send the service data request information to the retry service container in response to the request execution status information indicating execution failure. The retry service container is communicatively connected to the retry service manager, and the retry service container is configured to store and display the received service data request information, and send the received service data request information to the retry service manager.

[0061] In some embodiments, the retry service manager 3 described above can be a management engine with simple algorithm analysis capabilities. The retry service manager may include, but is not limited to, a microcontroller, a microcomputer, a chip, etc. Here, the retry service manager is configured to determine whether to initiate a retry for the service data request information, and in response to initiating a retry for the service data request information, to send a retry request command for the service data request information to the retry service container.

[0062] In some embodiments, the target system 4 can be a system in a distributed scenario, capable of interacting with other systems. Here, the target system is communicatively connected to the application server, wherein the retry service container is configured to, in response to receiving the retry request command, control the application server to initiate a retry to the target system for the service data request information. The target system is configured to send the generated retry result to the application server. A flowchart of the request data retry system is shown below. Figure 3 As shown.

[0063] The above embodiments of this disclosure have the following beneficial effects: By retrying failed requests for data through the request data retry system of some embodiments of this disclosure, the data consistency of various systems in a distributed system can be maintained. Specifically, the reason why system data is difficult to maintain consistency is that, due to the unknown status of the retry call, data loss occurs when the system restarts. Furthermore, continuous calls without successful execution consume application server resources. Based on this, in the request data retry system of some embodiments of this disclosure, the application server can collect service data request information and request execution status information corresponding to the service data request information. In response to the request execution status information indicating execution failure, the application server can asynchronously send the service data request information to the retry service container. The retry service container can store and display the received service data request information and send it to the retry service manager. The retry service manager can determine whether to initiate a retry for the service data request information, and in response to initiating a retry for the service data request information, send a retry request command for the service data request information to the retry service container. The retry service container, upon receiving the retry request command, controls the application server to initiate a retrieval to the target system for the service data request information. The target system can send the generated retry result to the application server. The application server can then send the received retry result to the retry service container for storage. This allows for timely acquisition of the request execution status information corresponding to the service data request information, enabling retries of failed data. Even during system restarts, the retry status can be determined based on the request execution status information, effectively preventing data loss and ensuring data consistency across systems.

[0064] Further reference Figure 4 This illustrates a structural schematic 300 of an embodiment of the request data retry system provided in this disclosure. Figure 2 Similar to the request data retry system in this embodiment, the request data retry system in this embodiment can also include an application server 1, a retry service container 2, a retry service manager 3, and a target system 4. For detailed structural relationships, please refer to [link to relevant documentation]. Figure 2 The relevant descriptions in the embodiments will not be repeated here.

[0065] and Figure 2Unlike the request data retry system in this embodiment, the system in this embodiment also includes a program hosting server 5. The hosting server can be a device that allows the retry process to continue running normally when the host application crashes. The program hosting server is communicatively connected to the retry service container, wherein the retry service container is configured to send retry results to the program hosting server. The retry results include retry success and retry failure. The retry service container is configured to send the service data request information to the program hosting server in response to determining that the retry result is a retry failure; the program hosting server is configured to initiate a retry on the received service data request information.

[0066] and Figure 2 Unlike the request data retry system in this embodiment, the application server in this embodiment includes a communication interface 11 and a listener 12 installed at the target network layer of the communication interface. The communication interface is configured to collect service data request information and corresponding request execution status information. The listener is communicatively connected to the retry service container. The listener is configured to asynchronously send the service data request information to the retry service container in response to a request execution status indicating failure.

[0067] Currently, data transmission between applications often uses RESTful, Dubbo, and HTTP interfaces, with the application server adding a listener at the lowest level of the interface. RESTful interfaces represent a design style and development approach for web applications. Dubbo interfaces are designed to provide high-performance and transparent RPC remote service calls. HTTP interfaces are based on the Hypertext Transfer Protocol. When the listener detects a system anomaly or alarm, it collects the service request information from the requesting party, compresses it, and transmits it to the retry service container. A schematic diagram of the application server's operation is shown below. Figure 5 As shown. The unique feature of this algorithm is its use of aspect-oriented programming. Listeners are added to Dubbo, RESTful, and HTTP interfaces to automatically collect service data request information that the application needs to upload and send it to the retry service container. This eliminates the need for the application to call the service separately, resulting in low invasiveness.

[0068] and Figure 2 Unlike the request data retry system in this embodiment, the listener in this embodiment includes a filter, wherein the filter is configured to intercept the service data request information in response to the request execution status information being an execution failure, and asynchronously send the service data request information to the retry service container.

[0069] and Figure 2Unlike the data retry system in this embodiment, the retry service container in this embodiment includes a first predetermined number of databases 21 storing data in key-value pairs and a second predetermined number of database statement parsers 22. The first predetermined number of databases storing data in key-value pairs are configured to support information insertion, querying, and updating based on primary keys. The second predetermined number of database statement parsers are configured to perform queries based on pre-acquired query information.

[0070] and Figure 2 Unlike the request data retry system in this embodiment, the retry service manager 3 in this embodiment is further configured to update the received service data request information to obtain updated service data request information. The retry service manager is configured to determine whether to initiate a retry for the updated service data request information. In response to initiating a retry for the updated service data request information, the retry service manager sends a retry request command for the updated service data request information to the retry service container.

[0071] The structural diagram of the retry service manager is as follows: Figure 6 As shown, in existing distributed systems, both the number and content of requests involve massive amounts of data. The retry service needs to support system-wide retries and requires a structure capable of supporting massive concurrency and storage. This structure does not have high requirements for data consistency and transaction consistency, but it needs to provide massive storage and throughput. Therefore, in this application, the data container used by the retry service container will be a partitioned database based on key-value pairs. This type of database supports fast insertion, querying, and updating by primary key, making it suitable for this high-concurrency scenario. Furthermore, based on this key-value database, assuming an SQL parser, the system can perform SQL queries with a small number of conditions, supporting the retry service manager to query tasks based on conditions.

[0072] The above steps, as an inventive point of this disclosure, solve the second technical problem mentioned in the background art: "It is difficult to maintain the normal operation of the system when the application server is unable to issue a retry request or the retry fails." The reasons for the difficulty in maintaining the normal operation of the system are often as follows: the application server may be unable to issue a retry request or the retry may fail. If these factors are resolved, the effect of maintaining the normal operation of the system can be achieved. To achieve this effect, the retry service manager is further configured to update the received service data request information to obtain updated service data request information; the retry service manager is configured to determine whether to initiate a retry for the updated service data request information; in response to initiating a retry for the updated service data request information, the retry service manager sends a retry request command for the updated service data request information to the retry service container. Therefore, when the application server is unable to issue a retry request or the retry fails, the retry service manager can update the service data request information and initiate a retry for the updated service data request information that needs to be retried, thereby maintaining the normal operation of the system and effectively avoiding the occurrence of the system being unable to issue a retry request or the retry failing.

[0073] and Figure 2 Unlike the request data retry system in this embodiment, the retry service manager in this embodiment includes a pre-trained detector. The retry service manager has functions for scheduled retries, retry result verification, and retry effect analysis. The pre-trained detector is configured to detect abnormal retry states.

[0074] and Figure 2 Unlike the data retry system in this embodiment, the pre-trained detector in this embodiment is obtained through the following methods: acquiring a retry frequency set and data features, wherein the data features include: date, time, network status, database load, and data processing scheme; clustering the retry frequency set to obtain classification results; determining the retry frequencies corresponding to the classifications that meet preset conditions in the classification results as the negative sample set. A classification that meets the preset conditions can be a category in which the sample proportion of one of the categories is within 3% of all categories; labeling each negative sample in the negative sample set to obtain a labeled negative sample set; determining the labeled negative samples in the labeled negative sample set that meet the preset conditions as the target negative sample set; performing data augmentation on the target negative sample set to obtain an augmented sample set; and training the initial detector based on the augmented sample set and the data features to obtain a trained detector.

[0075] The above steps, as an inventive point of this disclosure, solve the third technical problem mentioned in the background art: "Because the retry result is determined manually, retries cannot be initiated in a timely manner. Furthermore, manual determination has a high probability of false alarms, leading to inaccurate determination of the retry result." The reasons for the inability to initiate retries in a timely manner and to accurately determine the retry result are often as follows: determining the retry result manually is time-consuming, and manual determination has a high probability of false alarms. If these factors are resolved, the effect of timely initiation of retries and obtaining accurate retry results can be achieved. To achieve this effect, a pre-trained detector is configured to detect retry anomalies. The pre-trained detector is obtained through the following steps: acquiring a set of retry frequencies and data features, including date, time, network status, database load, and data processing scheme; clustering the retry frequency set to obtain classification results; determining the retry frequencies corresponding to categories that meet preset conditions as the negative sample set; labeling each negative sample in the negative sample set to obtain a labeled negative sample set; determining the labeled negative samples that meet preset conditions in the labeled negative sample set as the target negative sample set; performing data augmentation on the target negative sample set to obtain an augmented sample set; and training the initial detector based on the augmented sample set and the aforementioned data features to obtain a trained detector. Since system retry anomalies can be considered a low-probability event, K-means clustering can be used to cluster the retry frequencies into two categories: a positive sample set and a negative sample set. K-means clustering is an iterative clustering analysis algorithm. After clustering, the samples in the negative sample library can be labeled to determine the target negative sample set, i.e., the set of true anomalous state samples. Simultaneously, data augmentation is performed on the target negative sample set. After data augmentation, a random forest can be used to fit the samples and data features to form a trained monitor. This monitor can automatically determine retry results, thus initiating retry promptly, and accurately determining retry results with a low false alarm rate.

[0076] and Figure 2 Unlike the data retry system in this embodiment, the pre-trained detector in this embodiment includes a model training module. This model training module is configured to train the initial detector based on the enhanced sample set and the data features.

[0077] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A request data retry system, comprising: The application server, retry service container, retry service manager, and target system include: The application server is configured to collect service data request information and request execution status information corresponding to the service data request information, wherein the request execution status information includes execution success and execution failure; The application server is communicatively connected to the retry service container, wherein the application server is configured to asynchronously send the service data request information to the retry service container in response to the request execution status information being an execution failure. The retry service container is communicatively connected to the retry service manager, wherein the retry service container is configured to store and display the received service data request information, and send the received service data request information to the retry service manager. The retry service manager is configured to determine whether to initiate a retry for the service data request information, and in response to initiating a retry for the service data request information, to send a retry request command for the service data request information to the retry service container. The target system is communicatively connected to the application server, wherein the retry service container is configured to, in response to receiving the retry request command, control the application server to initiate a retry to the target system for the service data request information; The target system is configured to send the generated retry results to the application server; The application server is configured to send the received retry results to the retry service container for storage; The application server includes a communication interface and a listener installed at the target network layer of the communication interface, wherein: The communication interface is configured to collect service data request information and request execution status information corresponding to the service data request information; The listener is communicatively connected to the retry service container, wherein the listener is configured to asynchronously send the service data request information to the retry service container in response to the request execution status information being an execution failure.

2. The system according to claim 1, wherein, The system also includes a program hosting server, wherein: The program hosting server is communicatively connected to the retry service container, wherein the retry service container is configured to send retry results to the program hosting server, and the retry results include retry success and retry failure. The retry service container is configured to send the service data request information to the program hosting server in response to determining that the retry result is a retry failure. The program hosting server is configured to retry the received service data request information.

3. The system according to claim 2, wherein, The listener includes a filter, wherein the filter is configured to intercept the service data request information in response to the request execution status information being an execution failure, and asynchronously send the service data request information to the retry service container.

4. The system according to claim 3, wherein, The retry service container includes a first predetermined number of databases storing data in key-value pairs and a second predetermined number of database statement parsers, wherein: The first predetermined number of databases storing data in key-value pairs are configured to support information insertion, querying, and updating based on the primary key; The second predetermined number of database statement parsers are configured to perform queries based on pre-acquired query information.

5. The system according to claim 4, wherein, The retry service manager includes a pre-trained detector, wherein the retry service manager has functions for scheduled retries, retry result verification, and retry effect analysis, and the pre-trained detector is configured to detect abnormal retry states.

6. The system according to claim 5, wherein, The pre-trained detector includes a model training module, wherein the model training module is configured to train an initial detector based on an enhanced sample set and data features.

7. The system according to claim 6, wherein, The model training module is further configured to: Random sampling is performed from the enhanced sample set to generate a sub-data set; Based on the aforementioned sub-dataset, construct multiple decision trees; Each sub-data set and the data features are respectively input into the multiple decision trees for training.

Images