A method and system for realizing autonomous service degradation recovery in a fault scenario

Abstract

The present invention discloses a method and system for realizing autonomous service degradation recovery in a fault scenario, belonging to the field of software design; the specific steps of the method are as follows: S1 performs fault monitoring according to an error code returned by a service; S2 writes an original request of a strong dependency relationship into a directory by writing a file; S3 performs availability detection by scanning the status of a dependent third-party service through a scheduled task; S4 parses the file and autonomously recovers the service after the fault is recovered; the application of the scheme of the present invention can improve the reliability of its own service, record the original request of the user after the fault occurs, and autonomously complete the blocked service after the fault is recovered.

Description

Technical Field

[0001] The present invention discloses a method and system for realizing autonomous service degradation recovery in a fault scenario, and relates to the technical field of software design. Background Art

[0002] Service degradation means that when the server pressure increases dramatically, some services and pages are strategically not processed or processed in a simpler way according to the actual business situation and traffic, thereby freeing up server resources to ensure the normal or efficient operation of core transactions.

[0003] There are many service degradation schemes, such as page degradation, read degradation, write degradation, current limit degradation, timeout degradation, fault degradation, etc. This patent targets fault degradation.

[0004] As software architecture becomes increasingly complex, the number of external services that a single service relies on gradually increases. When a dependent external service fails, the speed of manual response directly affects the reliability of the service. In the event of a failure in a third-party service, the present invention saves the original service request and can continue to execute the interrupted service after the subsequent failure is restored. This effectively reduces the manual operation workload of operation and maintenance personnel, ensuring that services can proceed without manual intervention when a failure occurs, and also avoids the loss of service requests due to slow manual response speed in the event of a failure.

[0005] Therefore, the present invention invents a method and system for realizing self-service degradation recovery in a fault scenario to solve the above problems. Summary of the Invention

[0006] In view of the problems of the prior art, the present invention provides a method and system for implementing autonomous service degradation recovery in a fault scenario. The technical solution adopted is: a method for implementing autonomous service degradation recovery in a fault scenario, wherein the specific steps of the method are as follows:

[0007] S1 performs fault monitoring based on the error code returned by the service;

[0008] S2 writes the original request of the strong dependency into the directory by writing a file;

[0009] S3 uses scheduled tasks to scan the status of third-party services and perform availability checks;

[0010] After S4 recovers from a fault, it parses the file and automatically resumes business.

[0011] The specific steps of S1 performing fault monitoring based on the error code returned by the service are as follows:

[0012] S101 sets the default configuration error code for the Mysql database service, the Oracle database service, and the HTTP service.

[0013] S102 triggers service degradation when capturing the error code configured in S101.

[0014] The specific steps for S2 to write the original request of the strong dependency into the directory by writing a file are as follows:

[0015] S201 stores the file name and file record number generated when the thread is initialized into the thread private stack;

[0016] S202 distributes the received request to the processing thread according to certain routing rules;

[0017] S203: When the processing thread receives the processing task, it queries the total number of records saved in the private stack, and generates a new file name when the total number exceeds the threshold;

[0018] S204 writes the desensitized user original request into the file according to the file name saved in the private stack and updates the total number of records.

[0019] S3 uses a scheduled task to scan the status of third-party services. The specific steps for availability detection are as follows:

[0020] The method for detecting the status of the S301 database service Mysql is to execute the select 1 from dual statement;

[0021] S302 checks the status of the interface using the HTTP service status detection method to return a status code.

[0022] A system for implementing autonomous service degradation and recovery in fault scenarios, the system specifically includes a fault monitoring module, a service degradation module, a fault recovery module, and a business recovery module:

[0023] Fault monitoring module: performs fault monitoring based on the error code returned by the service;

[0024] Service degradation module: writes the original request of the strong dependency to the directory by writing a file;

[0025] Fault recovery module: Scans the status of third-party services through scheduled tasks to perform availability checks;

[0026] Business recovery module: parses files and automatically restores business after fault recovery.

[0027] The fault monitoring module specifically includes a configuration setting module and a configuration capture module:

[0028] Configuration setting module: set the default configuration error code of Mysql database service, the default configuration error code of Oracle database service and the default configuration error code of HTTP service;

[0029] Configuration capture module: Triggers service degradation when the error code configured in the configuration setting module is captured.

[0030] The service degradation module specifically includes a thread processing module, a request allocation module, a record query module and a record update module:

[0031] Thread processing module: stores the file name and file record number generated when the thread is initialized into the thread private stack;

[0032] Request allocation module: allocates received requests to processing threads according to certain routing rules;

[0033] Record query module: When the processing thread receives a processing task, it queries the total number of records saved in the private stack. If the number exceeds the threshold, a new file name is generated.

[0034] Record update module: writes the anonymized user original request to the file according to the file name saved in the private stack and updates the total number of records.

[0035] The fault recovery module specifically includes a status detection module and an interface detection module:

[0036] Status detection module: The database service Mysql status detection method is to execute the select 1 from dual statement;

[0037] Interface detection module: Use HTTP service status detection method to check the status code returned by the interface.

[0038] The beneficial effects of the present invention are as follows: the present invention discloses a method and system for realizing autonomous service degradation and recovery under fault scenarios, including four modules: fault monitoring, service degradation, fault recovery, and business recovery; the fault monitoring module regularly monitors other services that its own service depends on for faults, such as monitoring interface status codes, monitoring database exception codes, etc., and promptly triggers service degradation after a fault occurs. The service degradation module records original business requests by writing files through multiple threads, and threads are bound to files one by one. The number of threads is fixed, and when the number of records in a file exceeds a specific threshold, subsequent requests are written to another new file; on the one hand, the processing speed of requests is guaranteed, and on the other hand, when the number of concurrent requests is large, file writing will not fail due to insufficient number of file handles; the fault and business recovery module can regularly detect the status of other services, and resume the execution of business requests that were interrupted during the fault period after the faults of other services are restored; the application of the present invention's solution can improve the reliability of its own services, record the original requests of users after a fault occurs, and autonomously complete blocked businesses after the fault is recovered. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a structural diagram of the system of the present invention; Figure 3 This is a schematic diagram of the implementation process of an embodiment of the present invention; Figure 4 This is a schematic diagram of writing files during service degradation in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0042] Example 1:

[0043] A method for implementing autonomous service degradation recovery in a fault scenario, wherein the method comprises the following steps:

[0044] S1 performs fault monitoring based on the error code returned by the service;

[0045] S2 writes the original request of the strong dependency into the directory by writing a file;

[0046] S3 uses scheduled tasks to scan the status of third-party services and perform availability checks;

[0047] After S4 failure recovery, it analyzes the files and automatically resumes the service;

[0048] First, perform fault monitoring based on the error code returned by the service according to S1, and then write the original request of the strong dependency to the directory by writing a file according to S2;

[0049] Then, according to S3, a scheduled task scans the status of dependent third-party services to perform availability checks. When the dependent service has a strong dependency on the service itself, the original request is recorded in a file and written to the / serviceName / origin directory. To ensure the availability of file handles in the operating system, a multi-threaded file writing method is used. Threads are bound to files one by one, and the number of threads is fixed. When the number of records in a file exceeds a certain threshold, subsequent requests are written to a new file. This ensures request processing speed while preventing file write failures due to insufficient file handles when there are many concurrent requests.

[0050] Finally, according to the S4 fault recovery analysis file, the business is restored automatically. After confirming the fault recovery, the scheduled task performs the following operations:

[0051] Move the files in the / serviceName / origin directory to the / serviceName / run directory.

[0052] After the move is complete, the request message recorded in the file is read and processed. If an exception occurs during processing, the exception information is written to a new file and stored in the / serviceName / error directory.

[0053] After the message processing is completed, the file is moved to the / serviceName / backup directory;

[0054] Furthermore, the specific steps of S1 performing fault monitoring according to the error code returned by the service are as follows:

[0055] S101 sets the default configuration error code for the Mysql database service, the Oracle database service, and the HTTP service.

[0056] S102 triggers service degradation when it captures the error code configured in S101;

[0057] The default configuration error code for the Mysql database service is:

[0058] 1037: Insufficient system memory. Please restart the database or the server.

[0059] 1081: Unable to establish a socket connection;

[0060] 1158: Network error, read error occurred, please check the network connection status;

[0061] 1159: Network error, read timeout, please check the network connection status;

[0062] 1160: Network error, write error occurred, please check the network connection status;

[0063] 1161: Network error, write timeout, please check the network connection status;

[0064] 1129: Database exception occurs, please restart the database;

[0065] 2003: The MySQL service is not started. Please start the service.

[0066] The default configuration error code for the Oracle database service is:

[0067] ORA-00122: Unable to initialize network configuration;

[0068] ORA-00129: Listener address verification failed;

[0069] ORA-00130: invalid listener address;

[0070] ORA-00131: network protocol does not support registration;

[0071] The default configuration error codes for HTTP services are:

[0072] 502 (Bad Gateway) The server, acting as a gateway or proxy, received an invalid response from an upstream server;

[0073] 503 (Service Unavailable) The server is currently unavailable (due to overloading or maintenance);

[0074] 504 (Gateway Timeout) The server was acting as a gateway or proxy, but did not receive the request from the upstream server in a timely manner;

[0075] When the configured error code is captured during the operation of the service, service degradation is triggered, and different degradation strategies are implemented based on the strong or weak dependency between the service and other services.

[0076] Furthermore, the specific steps for S2 to write the original request of the strong dependency into the directory by writing a file are as follows:

[0077] S201 stores the file name and file record number generated when the thread is initialized into the thread private stack;

[0078] S202 distributes the received request to the processing thread according to certain routing rules;

[0079] S203: When the processing thread receives the processing task, it queries the total number of records saved in the private stack, and generates a new file name when the total number exceeds the threshold;

[0080] S204 writes the desensitized user original request into the file according to the file name saved in the private stack and updates the total number of records;

[0081] When the thread is initialized, the file name and number of file records to be written are generated and stored in the thread private stack; when a request is received, it is assigned to the processing thread according to a certain routing rule. In the embodiment, the rewriting routing rule can be set to hash, random, polling, etc.; when the processing thread receives the processing task, it queries the total number of records saved in the private stack, and generates a new file name when it exceeds the threshold; according to the file name saved in the private stack, the desensitized user original request is written to the file and the total number of records is updated;

[0082] The above method can record the business requests during the service failure;

[0083] Furthermore, the S3 scans the status of the third-party service through a scheduled task, and the specific steps for performing availability detection are as follows:

[0084] The method for detecting the status of the S301 database service Mysql is to execute the select 1 from dual statement;

[0085] S302 checks the status of the interface using the HTTP service status detection method to return a status code.

[0086] Example 2:

[0087] A system for implementing autonomous service degradation and recovery in fault scenarios, the system specifically includes a fault monitoring module, a service degradation module, a fault recovery module, and a business recovery module:

[0088] Fault monitoring module: performs fault monitoring based on the error code returned by the service;

[0089] Service degradation module: writes the original request of the strong dependency to the directory by writing a file;

[0090] Fault recovery module: Scans the status of third-party services through scheduled tasks to perform availability checks;

[0091] Business recovery module: parses files and automatically restores business after fault recovery;

[0092] Furthermore, the fault monitoring module specifically includes a configuration setting module and a configuration capture module:

[0093] Configuration setting module: set the default configuration error code of Mysql database service, the default configuration error code of Oracle database service and the default configuration error code of HTTP service;

[0094] Configuration capture module: triggers service degradation when capturing the error code configured in the configuration setting module;

[0095] Furthermore, the service degradation module specifically includes a thread processing module, a request allocation module, a record query module and a record update module:

[0096] Thread processing module: stores the file name and file record number generated when the thread is initialized into the thread private stack;

[0097] Request allocation module: allocates received requests to processing threads according to certain routing rules;

[0098] Record query module: When the processing thread receives a processing task, it queries the total number of records saved in the private stack. If the number exceeds the threshold, a new file name is generated.

[0099] Record update module: writes the desensitized user original request to the file according to the file name saved in the private stack and updates the total number of records;

[0100] Furthermore, the fault recovery module specifically includes a status detection module and an interface detection module:

[0101] Status detection module: The database service Mysql status detection method is to execute the select 1 from dual statement;

[0102] Interface detection module: Use HTTP service status detection method to check the status code returned by the interface.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for implementing autonomous service degradation recovery in a fault scenario, characterized by: The specific steps of the method are as follows: S1 performs fault monitoring based on the error code returned by the service; S2 writes the original request of the strong dependency into the directory by writing a file; S3 scans the status of third-party services through scheduled tasks to perform availability checks; After S4 fails and recovers, it analyzes the files and automatically resumes the service; The specific steps of S1 performing fault monitoring according to the error code returned by the service are as follows: S101 sets the default configuration error code of the database service Mysql, the default configuration error code of the database service Oracle and the default configuration error code of the HTTP service; S102 triggers service degradation when capturing the error code configured in S101; The specific steps for S2 to write the original request of the strong dependency into the directory by writing a file are as follows: S201 stores the file name and file record number generated when the thread is initialized into the thread private stack; S202 distributes the received request to the processing thread according to certain routing rules; S203: When the processing thread receives the processing task, it queries the total number of records saved in the private stack, and generates a new file name when the total number of records exceeds the threshold. S204 writes the desensitized original user request into the file according to the file name saved in the private stack and updates the total number of records.

2. The method according to claim 1, characterized in that The specific steps of S3 performing availability detection by scanning the status of third-party services through scheduled tasks are as follows: The method for detecting the status of the S301 database service Mysql is to execute the select 1from dual statement; S302 checks the status of the interface through the HTTP service status detection method to return a status code.

3. A system for implementing autonomous service degradation recovery in a fault scenario, characterized by: The system specifically includes a fault monitoring module, a service degradation module, a fault recovery module and a business recovery module: Fault monitoring module: performs fault monitoring based on the error code returned by the service; Service degradation module: writes the original request of the strong dependency to the directory by writing a file; Fault recovery module: scans the status of third-party services through scheduled tasks and performs availability detection; Business recovery module: parse files and automatically restore business after failure recovery; The fault monitoring module specifically includes a configuration setting module and a configuration capture module: Configuration setting module: set the default configuration error code of Mysql database service, the default configuration error code of Oracle database service and the default configuration error code of HTTP service; Configuration capture module: triggers service degradation when the error code configured in the configuration setting module is captured; The service degradation module specifically includes a thread processing module, a request allocation module, a record query module and a record update module: Thread processing module: stores the file name and file record number generated when the thread is initialized into the thread private stack; Request allocation module: allocates received requests to processing threads according to certain routing rules; Record query module: When the processing thread receives a processing task, it queries the total number of records saved in the private stack. When the total number of records exceeds the threshold, a new file name is generated. Record update module: writes the desensitized user original request into the file according to the file name saved in the private stack and updates the total number of records.

4. The system according to claim 3, characterized in that The fault recovery module specifically includes a status detection module and an interface detection module: Status detection module: The database service Mysql status detection method is to execute the select 1from dual statement; Interface detection module: Use HTTP service status detection method to check the status code returned by the interface.

Images