Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. The data processing method comprises the following steps: acquiring attribute characteristics of a service interface request, and generating an access log according to the attribute characteristics; obtaining a historical request log, and determining a system fusing strategy according to the access log, the historical request log and a preset fusing rule; and when the system fusing strategy is fusing, determining historical response data, determining a data type of the historical response data according to a target classification model, and determining target response data according to the data type and the historical response data. According to the embodiment of the invention, the reasonability of system fusing is improved, meanwhile, the situation that the request cannot be responded in the system fusing process is avoided, and the robustness of the system and the user experience are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the widespread adoption of microservice architecture in software development, service stability and user experience have become paramount. When a service experiences a failure or response delay, a circuit breaker mechanism is typically needed to prevent the failure from spreading. However, traditional circuit breaker mechanisms often rely on fixed thresholds, lack dynamic adjustment capabilities, struggle to adapt to complex and changing traffic conditions, and usually require pre-defined circuit breaker responses or simple rules, which often fail to provide a better user experience.

[0003] Currently, the fixed threshold circuit breaker mechanism is the most traditional circuit breaker strategy. It determines whether to trigger the circuit breaker based on a preset fixed threshold. When the failure rate or response time of a service exceeds the preset threshold, the circuit breaker opens, blocking further requests to that service and preventing its failure from affecting other services. Its advantages are simple implementation, ease of understanding and configuration, and suitability for stable and predictable traffic environments. However, it lacks flexibility and struggles to handle dynamically changing traffic and complex service dependencies. Furthermore, the fixed threshold may not be suitable for all scenarios, potentially leading to false triggers or failures. The sliding window circuit breaker mechanism determines whether to trigger the circuit breaker by counting the number of requests and failures within a certain time window. It segments request data along a time dimension and calculates the failure rate to control the circuit breaker status with finer granularity. It can more accurately reflect the service's performance at different time periods, reducing false judgments. It improves flexibility and responsiveness, adapting to short-term traffic fluctuations. However, its implementation is relatively complex, requiring the maintenance and processing of time window data. In high-concurrency environments, the choice of time window may affect performance. All of the above circuit breaker mechanisms focus on the cascading failures caused by service failures, emphasizing the maintenance of overall service stability. However, the availability of service-returned data during actual degradation cannot be adapted to every system. Therefore, how to reasonably trigger circuit breakers has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, electronic device, and storage medium to solve the problem of low dynamic adjustment capability in the prior art of triggering fuses.

[0005] According to one aspect of the present invention, a data processing method is provided, wherein the method includes:

[0006] Obtain the attribute characteristics of the business interface request, and generate access logs based on the attribute characteristics;

[0007] Obtain historical request logs, and determine the system circuit breaker strategy based on the access logs, historical request logs, and preset circuit breaker rules;

[0008] When the system's circuit breaker policy is set to circuit breaker, historical response data is determined, the data type of the historical response data is determined according to the target classification model, and the target response data is determined according to the data type and the historical response data.

[0009] According to another aspect of the present invention, a data processing apparatus is provided, wherein the apparatus comprises:

[0010] The log determination module is used to obtain the attribute characteristics of business interface requests and generate access logs based on the attribute characteristics.

[0011] The strategy determination module is used to obtain historical request logs and determine the system circuit breaker strategy based on the access logs, historical request logs and preset circuit breaker rules.

[0012] The data response module is used to determine historical response data when the system's circuit breaker policy is set to circuit breaker, determine the data type of the historical response data according to the target classification model, and determine the target response data according to the data type and the historical response data.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data processing method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the data processing method of any embodiment of the present invention.

[0019] The technical solution of this invention obtains the attribute characteristics of business interface requests, generates access logs based on the attribute characteristics, obtains historical request logs, determines the system circuit breaker strategy based on the access logs, historical request logs, and preset circuit breaker rules, and determines historical response data when the system circuit breaker strategy is to circumvent the circuit breaker by obtaining historical response data, determining the data type of historical response data based on the target classification model, and determining the target response data according to the data type and historical response data. This improves the rationality of system circuit breaking and avoids the inability to respond to requests during system circuit breaking, thereby improving the robustness of the system and the user experience.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a data processing method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart of a data processing method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a data processing system according to Embodiment 3 of the present invention;

[0025] Figure 4 This is a flowchart illustrating a data processing method according to Embodiment 3 of the present invention;

[0026] Figure 5 This is a schematic diagram of the structure of a data processing device according to Embodiment 4 of the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the data processing method of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a data processing method according to Embodiment 1 of the present invention. This embodiment is applicable to automatic circuit breaking in microservice architectures and the determination of response data corresponding to requests. The method can be executed by a data processing device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110. Obtain the attribute characteristics of the business interface request and generate access logs based on the attribute characteristics.

[0033] In this context, a business interface request can be understood as a request initiated by a client to a business service to complete a specific business operation. For example, a business interface request may include a business query request and an interface call request. In practical applications, business interface requests can be Hypertext Transfer Protocol (HTTP) requests, Hypertext Transfer Protocol Secure (HTTPS) requests, etc. For instance, business interface requests may include requests to query traffic, query usernames, query balances, and order payments. Attribute characteristics can be understood as the feature information of a business interface request. Generally, attribute characteristics can include data features that carry business logic; for example, attribute characteristics may include information such as city, interface name, channel, and time. Access logs can be understood as logs generated based on the attribute characteristics of business interface requests, and access logs can have a custom format.

[0034] In this embodiment, business interface requests sent by users / clients can be received, and the characteristic information of the business interface requests can be determined as attribute information. Alternatively, information such as city, interface name, channel, and time in the business interface requests can be extracted according to preset dimensions as attribute information, and the attribute information can be encapsulated into logs as access logs according to preset formats. In actual applications, business interface requests can be intercepted through the Lua plugin in Nginx configuration, and the characteristics such as city, channel, interface name, and time in the request header and parameters can be parsed. The Lua plugin can then be used to supplement log fields and format log content to generate access logs.

[0035] S120. Obtain historical request logs and determine the system circuit breaker strategy based on access logs, historical request logs, and preset circuit breaker rules.

[0036] Historical request logs can be understood as log information confirming the responses to business interface requests at historical moments. In practice, historical request logs may include request timestamps, request Uniform Resource Locators (URLs), request parameters, response status codes, response times, and response content. System circuit breaker policies can be understood as strategies for determining whether to implement circuit breaking in a microservice architecture. In practical applications, system circuit breaker policies can include both circuit breaking and non-circuit breaking. Generally, system circuit breaker policies can be determined by pre-set circuit breaker rules. Pre-set circuit breaker rules refer to the rules that are pre-defined to determine the system circuit breaker policy.

[0037] In this embodiment, historical request logs can be extracted. Target historical access data for determining the system's circuit breaker strategy can be extracted from the access logs. The system's circuit breaker strategy is then determined based on the target historical access data and preset circuit breaker rules. In actual operation, attribute dimensions of the access logs can be determined, such as city, interface, channel, and time dimensions. Historical request logs matching the attribute dimensions within the current statistical period are identified as the first target request logs. Historical request logs matching the attribute dimensions within a historical statistical period (such as the previous statistical period) are identified as the second target request logs. The request success rate and request call volume corresponding to the first and second target request logs are determined respectively. The difference between the request success rate and request call volume corresponding to the second target request log and the request success rate and request call volume corresponding to the first target request log is then determined. At this point, the preset fixed circuit breaker threshold, preset window circuit breaker threshold, and preset difference circuit breaker threshold in the system's circuit breaker strategy can be extracted. It can then be determined whether the request success rate and request volume corresponding to the second target request log meet the preset fixed circuit breaker threshold and preset window circuit breaker threshold, respectively. Furthermore, it can be determined whether the difference between the request success rate and request volume corresponding to the second target request log and the request success rate and request volume corresponding to the first target request log meets the preset difference circuit breaker threshold. The system circuit breaker strategy is determined based on whether these conditions are met. In practical applications, the determination of whether the preset fixed circuit breaker threshold and preset window circuit breaker threshold are met can be used as the primary judgment indicator, and the determination of whether the difference circuit breaker threshold is met can be used as an auxiliary judgment indicator. If the real-time indicator meets the preset fixed circuit breaker threshold or preset window circuit breaker threshold, and the difference indicator meets the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker. If the real-time indicator does not meet the preset fixed circuit breaker threshold or the preset window circuit breaker threshold, but the difference indicator meets the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker. In other cases, the system circuit breaker strategy is determined to be non-circuit breaker.

[0038] S130. When the system circuit breaker policy is circuit breaker, determine the historical response data, determine the data type of the historical response data according to the target classification model, and determine the target response data according to the data type and the historical response data.

[0039] Historical response data can be understood as the response data in historical request logs. In practice, historical response data refers to the response data corresponding to historical request logs that match the attribute dimensions within the current statistical period, i.e., the response data of the first target request log. The target classification model can be understood as a trained model used to determine the data type of historical response data. In practical applications, the target classification model can include Random Forest models or Gradient Boosting Decision Trees (GBDTs). In practice, the target classification model can use classification algorithms such as Random Forest and GBDT to identify whether the data type of historical response data is immutable or variable. Target response data refers to the response data corresponding to business interface requests, which is the data the client needs.

[0040] In this embodiment, historical request logs matching the attribute dimension within the current statistical period can be identified, i.e., the first target request logs. The response data stored in the first target request logs is then used as historical response data. The historical response data is input into a target classification model, which determines the data type of the historical response data. The data type can be either constant or variable. If the data type is constant, the historical response data can be used as the target response data; if the data type is variable, a preset data determination rule corresponding to the historical response data can be determined, and the target response data can be determined using this preset rule. For example, if the historical response data is a balance query, the business characteristics can be determined according to the preset data determination rule, summarizing the balance operation interface information recently involved by the target user, and the user's balance can be determined as the target response data using a predefined balance calculation formula; alternatively, if the query information shows a relationship between usage and increasing dates, the target response data can be determined based on this relationship.

[0041] In this embodiment of the invention, by acquiring the attribute characteristics of business interface requests, generating access logs based on the attribute characteristics, acquiring historical request logs, and determining the system circuit breaker strategy based on the access logs, historical request logs, and preset circuit breaker rules, when the system circuit breaker strategy is to activate circuit breaking, historical response data is determined, the data type of the historical response data is determined based on the target classification model, and the target response data is determined according to the data type and historical response data. This improves the rationality of system circuit breaking and avoids the inability to respond to requests during system circuit breaking, thereby improving system robustness and user experience.

[0042] In one embodiment, after obtaining historical request logs and determining the system circuit breaker policy based on access logs, historical request logs, and preset circuit breaker rules, the method further includes:

[0043] When the system's circuit breaker policy is set to non-circuit breaker, the business interface request is sent to the target application cluster so that the target application cluster can respond with target response data according to the business interface request.

[0044] The target application cluster can be understood as the cluster that stores the response data corresponding to historical request logs.

[0045] In one embodiment, when the system's circuit breaker policy is set to non-circuit breaker, the business interface request can be directly sent to the target application cluster, which can then respond with target response data according to the business interface request. In another embodiment, after determining the response data, the request timestamp, request URL, request parameters, response status code, and response time of the business interface request can be determined. The request timestamp, request URL, request parameters, response status code, response time, and response content of the business interface request are then used to generate a request log, which is stored as a historical request log.

[0046] In one embodiment, when the system's circuit breaker policy is set to circuit breaker, historical response data is determined, the data type of the historical response data is determined according to the target classification model, and after determining the target response data according to the data type and historical response data, the method further includes:

[0047] Obtain the preset system recovery test cases corresponding to the attribute characteristics, and execute the preset system recovery test cases according to the preset execution frequency to obtain the test case results;

[0048] If the test case results meet the preset conditions more than the preset number of times, the system is determined to recover, and the system circuit breaker policy is updated to non-circuit breaker.

[0049] The preset system recovery test cases can be understood as test cases used to test whether the system can be restored to a normal state. In actual operation, preset system recovery test cases may include automatically executing a series of tests simulating user behavior (such as logging in, retrieving data, submitting forms, etc.). The preset number of times refers to the threshold for determining the number of times the system can be restored, which can be set according to business needs.

[0050] In this embodiment, corresponding preset system recovery test cases can be extracted according to attribute features, executed at a preset execution frequency, and the test case results for each execution of the preset system recovery test cases can be determined. If the test case results meet preset conditions more than a preset number of times, the system can be considered to have recovered. At this point, the system circuit breaker policy can be updated to non-circuit breaker, realizing automatic system recovery.

[0051] Example 2

[0052] Figure 2This is a flowchart of a data processing method according to Embodiment 2 of the present invention. This embodiment is a further optimization and extension based on the above embodiments, and can be combined with various optional technical solutions in the above embodiments. Figure 2 As shown, the method includes:

[0053] S201. Extract the dimension information of the business interface request in the target dimension and use the dimension information as attribute features.

[0054] Among them, the target dimension refers to the pre-set dimension, such as the city dimension, interface dimension, channel dimension, time dimension, etc.

[0055] In this embodiment, dimensional information of the business interface request can be extracted according to the target dimension, and the dimensional information can be used as attribute features. For example, the city, interface, channel, and time in the business interface request can be extracted according to the city dimension, interface dimension, channel dimension, and time dimension, and the city, interface, channel, and time can be used as attribute features.

[0056] S202. Encapsulate attribute features into structured logs and use the structured logs as access logs.

[0057] In this embodiment, the attribute features can be encapsulated according to a preset format to obtain a structured log, and the structured log can be determined as an access log.

[0058] S203. Determine the attribute dimensions of the attribute features in the access log, and use the historical request logs matched by the attribute dimensions within the current statistical period as the first target request logs, and use the historical request logs matched by the attribute dimensions within the historical statistical period as the second target request logs.

[0059] In this embodiment, the attribute dimensions of the access logs can be determined, such as city, interface, channel, and time dimensions. Historical request logs matching the attribute dimensions are identified from the historical request logs. The historical request logs matching the attribute dimensions within the current statistical period are selected as the first target request logs, and the historical request logs matching the attribute dimensions within the historical statistical period are selected as the second target request logs. Alternatively, historical request logs within the current statistical period and the historical statistical period can be determined separately, and then historical request logs matching the attribute dimensions within the current statistical period and the historical statistical period can be selected as the first and second target request logs, respectively.

[0060] S204. Determine the request success rate and request call volume of the first target request log as real-time indicators, and determine the request success rate and request call volume of the second target request log as historical indicators.

[0061] In this context, the request success rate can be understood as the proportion of successful requests to the total number of valid requests within a certain period, while the request call volume can be understood as the total number of valid requests within a certain period. In actual operation, the number of successful requests can be determined through the request URLs in the first and second target request logs.

[0062] In this embodiment, the request URL indicating a successful request can be extracted, the number of successful requests in the first target request log and the second target request log can be determined, and the total number of valid requests in the first target request log and the second target request log can be determined. The ratio of the number of successful requests in the first target request log to the total number of valid requests is determined as the request success rate of the first target request log, and the ratio of the number of successful requests in the second target request log to the total number of valid requests is determined as the request success rate of the second target request log. The request success rate and request call volume of the first target request log are used as real-time indicators, and the request success rate and request call volume of the second target request log are used as historical indicators.

[0063] S205. Extract the preset circuit breaker rules and determine the difference between historical indicators and real-time indicators as the difference indicator.

[0064] In one embodiment, the preset circuit breaker rule includes at least:

[0065] If the real-time indicators meet the preset fixed circuit breaker threshold or the preset window circuit breaker threshold, and the difference indicators meet the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker.

[0066] If the real-time metric does not meet the preset fixed circuit breaker threshold or the real-time metric does not meet the preset window circuit breaker threshold, but the difference metric meets the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker.

[0067] Among them, the preset fixed circuit breaker threshold refers to the pre-set critical value for fixed circuit breaking. For each category of real-time indicators, there is a corresponding fixed circuit breaker threshold. The preset window circuit breaker threshold is the critical value for circuit breaking within a pre-set window. The preset difference circuit breaker threshold refers to the pre-set critical value for difference circuit breaking.

[0068] In this embodiment, the difference between historical metrics and real-time metrics can be determined as the difference metric. That is, the difference between the request success rate in the real-time metrics and the request success rate in the historical metrics, as well as the difference between the request call volume in the real-time metrics and the request call volume in the historical metrics, can be determined respectively.

[0069] S206. Determine the system circuit breaker strategy according to the preset circuit breaker rules, difference index and real-time index.

[0070] In this embodiment, the difference index and the real-time index can be compared according to a preset circuit breaker rule. If it is determined that the difference index and the real-time index meet the preset circuit breaker rule, the system circuit breaker strategy is determined to be circuit breaker; if it is determined that the difference index and the real-time index do not meet the preset circuit breaker rule, the system circuit breaker strategy is determined to be non-circuit breaker.

[0071] S207. Extract the response data stored in the first target request log in the historical request log as historical response data.

[0072] In this embodiment, the response data stored in the first target request log can be identified as historical response data.

[0073] S208. Input the historical response data into the target classification model, and determine the data type of the historical response data through the target classification model.

[0074] The data types include constant data and variable data. Constant data refers to data that remains unchanged over a long period, such as username, channel, and city; variable data refers to data that changes frequently, such as balance and remaining data allowance.

[0075] In this embodiment, historical response data can be input into the target classification model, which then determines whether the historical response data is constant or changing.

[0076] S209. When the data type is determined to be constant, the historical response data shall be used as the target response data.

[0077] In this embodiment, if the data type is constant, it can be determined that the target response data is the same as the historical response data. In this case, the historical response data can be used as the target response data.

[0078] S210. When the data type is determined to be changing data, determine the corresponding preset data determination rule for the historical response data, and determine the target response data according to the historical response data and the preset data determination rule.

[0079] In this embodiment, if the data type is variable data, the corresponding preset data determination rules can be extracted from the historical response data, and the target response data can be determined by combining the historical response data and the preset data determination rules. For example, if the historical response data is a balance query, the business characteristics can be determined according to the preset data determination rules, the balance operation interface information recently involved by the target user can be summarized, and the user balance can be determined as the target response data by using a predefined balance calculation formula.

[0080] In this embodiment of the invention, dimensional information of business interface requests in the target dimension is extracted, and this dimensional information is used as attribute features. These attribute features are then encapsulated into structured logs, which are used as access logs. The attribute dimensions of the attribute features in the access logs are determined. Historical request logs matching the attribute dimensions within the current statistical period are used as the first target request logs, and historical request logs matching the attribute dimensions within the historical statistical period are used as the second target request logs. The request success rate and request call volume of the first target request logs are determined as real-time metrics, and the request success rate and request call volume of the second target request logs are determined as historical metrics. This process achieves metric determination. Furthermore, by extracting preset circuit breaker rules, the difference between historical and real-time metrics is determined as the difference value. The system uses metrics such as preset circuit breaker rules, difference metrics, and real-time metrics to determine the system's circuit breaker strategy, improving the rationality of the strategy determination. It extracts response data stored in the first target request log from the historical request log as historical response data, inputs this data into a target classification model, and determines the data type of the historical response data. If the data type is determined to be constant, the historical response data is used as the target response data. If the data type is determined to be changing, the corresponding preset data determination rule for the historical response data is determined. The target response data is then determined according to the historical response data and the preset data determination rule, enabling the determination of the target response data in the event of a system circuit breaker, thus ensuring the normal operation of the system.

[0081] Example 3

[0082] Figure 3 This is a schematic diagram of the structure of a data processing system according to Embodiment 3 of the present invention, as shown below. Figure 3 As shown, the system includes: an Nginx cluster, a gateway cluster, an application cluster, Kafka, a log storage system (Elasticsearch, Logstash, Kibana, ELK), a circuit breaker controller, and an Artificial Intelligence (AI) analysis center. This embodiment uses the execution of a data processing method by the data processing system as an example, and takes a request as a business interface request as an example to further illustrate a data processing method. In one embodiment, Figure 4 This is a flowchart illustrating a data processing method according to Embodiment 3 of the present invention. Figure 4 As shown, the method includes:

[0083] User (client) requests first enter the Nginx cluster. The Nginx cluster is responsible for load balancing of requests and integrates a circuit breaker gateway (marked with a red cross in the diagram). The circuit breaker gateway is the core interception point, determining whether the current request should trigger a circuit breaker. The circuit breaker gateway intercepts requests received by the Nginx cluster and queries the circuit breaker controller to see if the corresponding attribute dimensions (such as interface, channel, city) of the current request have triggered a circuit breaker policy. The circuit breaker controller, based on log data consumed by Kafka, calculates metrics such as request success rate and request call volume in real time to determine whether to trigger a circuit breaker rule (fixed / window / difference rule) and returns the system circuit breaker policy (fuse or not) to the circuit breaker gateway. If the system circuit breaker policy is not to trigger a circuit breaker, the request is forwarded from Nginx to the Gateway Cluster, and then from the Gateway to the Application Cluster to process the business logic and obtain the target response data (i.e., the actual service). If the system's circuit breaker policy is set to circuit breaking, the request will be forwarded to the AI ​​analysis center. The AI ​​analysis center will read the historical request logs corresponding to that request from the ELK (Elasticsearch Log Store), determine the data type (unchanging / changing) using a classification model, generate target response data that meets the requirements, and replace the returned result from the faulty application cluster (ensuring the user still receives a legitimate response). Figure 3 In the Nginx application cluster, the Nginx log (access.log) records key information about Nginx's request processing (such as request dimensions, response status, and circuit breaker status). This information is collected by Logstash and sent to Kafka, ultimately being consumed by the circuit breaker controller to calculate real-time metrics. The gateway log (request.log) records requests forwarded by the gateway to the application cluster (such as business parameters and response content). This information is collected by Logstash and stored in ELK, allowing the AI ​​analytics center to read historical data to generate degraded responses. It also supports manual querying and troubleshooting.

[0084] In one embodiment, all requests are distributed to upstream applications via Nginx load balancing, which is the first-level entry point for requests. Using Nginx and Lua plugins, custom log formats are output, and logs are collected in real-time to Kafka. This allows for analysis of service success rates over a period of time, with dimensions precisely defined as: city, interface, channel, time, etc. All requests passing through the gateway application are recorded by the gateway in the ELK log cluster. The AI ​​analysis center collects application request logs and uses machine learning to summarize the request characteristics of each interface in the logs to generate service responses, improving user experience.

[0085] In one embodiment, the circuit breaker controller can generate customized log formats: using a LUA plugin in nginx, the access logs of requests are customized to support the inclusion of circuit breaker-related feature values ​​in the logs, such as: city, interface name, channel ID, request time, etc. Logs are collected and aggregated in real time. By collecting access logs and inputting them into a Kafka message queue, the circuit breaker controller reads messages and defined circuit breaker rules in Kafka in real time. In addition to mainstream fixed circuit breakers and sliding window circuit breakers, the circuit breaker rules can also be based on historical success rates, call volume, and other data to determine whether a request needs to be circuit-broken. The controller distributes requests; in the circuit-breaker state, requests are forwarded to the AI ​​analysis center. To exit the circuit-breaker state and restore the system to normal operation, considering the system characteristics, automated test cases can be used to determine whether the service has recovered and decide whether to lift the circuit breaker while the system is under circuit breaker. These automated test cases can run continuously and without interruption, effectively reducing human intervention and ensuring timely detection and recovery of the service after it has recovered. By designing targeted automated test cases, a series of tests simulating user behavior (such as logging in, retrieving data, submitting forms, etc.) can be executed automatically. If these requests return the expected correct results, the system can be considered to have recovered. Therefore, automated systems can more accurately determine whether specific problems have been fixed, avoiding misjudgments.

[0086] AI Analysis Center: Extracts log data from Elk. Analyzes log data including request timestamps, request URLs, request parameters, response status codes, response times, and response content. Classifies log data into long-term unchanging (immutable) and frequently changing types. This can be achieved by statistically analyzing the frequency of changes in specific request parameters. Extracts features from response content and calculates the rate of change (e.g., using hash values ​​to compare changes in adjacent responses). Trains a target classification model using classification algorithms such as Random Forest and Gradient Boosting Tree (GBDT) to identify whether requests are long-term unchanging or frequently changing. Compared to traditional training data feeding, this model's advantage lies in using dynamic, real-time data. It can collect current production data samples near real-time without affecting system usage, allowing the model to closely reflect real-time production usage. The response pattern prediction model can use caching mechanisms or simple predictions based on historical data to directly cache response content for long-term unchanging data. It uses Long Short-Term Memory Neural Networks (LSTM) or time series prediction models (such as Autoregressive Integrated Moving Average (ARIMA)). For frequently changing data, time series models are used to predict response content. Real-time prediction and response generation are performed. During circuit breaker conditions, current request data, including request parameters and timestamps, is obtained. A trained classification model is used to determine whether the current request belongs to long-term unchanging data or frequently changing data. For long-term unchanging data, historical response data can be directly returned from the cache. For frequently changing data, more granular classification is performed. For example, in the cumulative amount query interface, the relationship between usage and date increases can be extracted during the feature extraction stage. This rule can then be used to simulate data return during the circuit breaker stage. For the balance query interface, based on business characteristics, information on recent balance operation interfaces of the target user can be aggregated, and user balance query information can be returned using a predefined balance calculation formula. Other types of interfaces are all executed with granular rules based on information extracted from business characteristics and feature engineering. All interfaces involved in circuit breaker are connected to the complaint system, and correlation operations are performed based on the number of complaints after the circuit breaker, providing secondary protection after the circuit breaker. Under the integrated circuit breaker system based on business characteristics and user experience, we strive to provide users with a better experience.

[0087] This embodiment leverages Nginx's high-performance synchronous non-blocking characteristics and high affinity support for Lua plugins to develop a dynamic traffic monitoring and granular circuit breaker controller. The circuit breaker controller allows for more flexible definition of circuit breaker rules. Besides fixed circuit breakers and sliding window circuit breakers, it can also perform reference-based circuit breakers, such as analyzing and comparing data from a specific fault point to determine if it meets the circuit breaker threshold. Alternatively, it can combine data with production hardware utilization indicators and complaint data for circuit breaking. This makes it more adaptable to actual production usage scenarios. Simultaneously, it effectively identifies the characteristics of request data and generates response data based on classification models and time series prediction models. When the system enters a circuit breaker state, it can dynamically select the most suitable response strategy based on the characteristics of the request, improving system robustness and user experience. Furthermore, by comprehensively utilizing ELK logging and machine learning technologies, it achieves comprehensive monitoring and intelligent analysis of system status, thereby improving system observability and automation. It fully utilizes the characteristics of existing production architectures, allowing for minimal or non-intrusive integration with the system and avoiding adaptation modifications to business-side code.

[0088] Example 4

[0089] Figure 5 This is a schematic diagram of the structure of a data processing device according to Embodiment 4 of the present invention. Figure 5 As shown, the device includes: a log determination module 51, a strategy determination module 52, and a data response module 53.

[0090] The log determination module 51 is used to obtain the attribute characteristics of business interface requests and generate access logs based on the attribute characteristics.

[0091] The strategy determination module 52 is used to obtain historical request logs and determine the system circuit breaker strategy based on the access logs, historical request logs and preset circuit breaker rules.

[0092] The data response module 53 is used to determine historical response data when the system circuit breaker policy is circuit breaker, determine the data type of historical response data according to the target classification model, and determine the target response data according to the data type and historical response data.

[0093] The technical solution of this invention obtains the attribute characteristics of business interface requests, generates access logs based on the attribute characteristics, obtains historical request logs, determines the system circuit breaker strategy based on the access logs, historical request logs, and preset circuit breaker rules, and determines historical response data when the system circuit breaker strategy is to circumvent the circuit breaker by obtaining historical response data, determining the data type of historical response data based on the target classification model, and determining the target response data according to the data type and historical response data. This improves the rationality of system circuit breaking and avoids the inability to respond to requests during system circuit breaking, thereby improving the robustness of the system and the user experience.

[0094] In one embodiment, the log determination module includes:

[0095] The feature extraction unit is used to extract the dimensional information of the business interface request in the target dimension and use the dimensional information as attribute features.

[0096] The log determination unit is used to encapsulate attribute features into structured logs and use the structured logs as access logs.

[0097] In one embodiment, the strategy determination module includes:

[0098] The log extraction unit is used to determine the attribute dimensions of the attribute features in the access log, and to use the historical request logs matched by the attribute dimensions within the current statistical period as the first target request logs, and the historical request logs matched by the attribute dimensions within the historical statistical period as the second target request logs.

[0099] The indicator determination unit is used to determine the request success rate and request call volume of the first target request log as real-time indicators, and to determine the request success rate and request call volume of the second target request log as historical indicators.

[0100] The difference determination unit is used to extract the preset circuit breaker rules and determine the difference between historical indicators and real-time indicators as the difference indicator.

[0101] The strategy determination unit is used to determine the system's circuit breaker strategy according to preset circuit breaker rules, difference indicators, and real-time indicators.

[0102] In one embodiment, the preset circuit breaker rule includes at least:

[0103] If the real-time indicators meet the preset fixed circuit breaker threshold or the preset window circuit breaker threshold, and the difference indicators meet the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker.

[0104] If the real-time metric does not meet the preset fixed circuit breaker threshold or the real-time metric does not meet the preset window circuit breaker threshold, but the difference metric meets the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker.

[0105] In one embodiment, the data response module includes:

[0106] The historical data extraction unit is used to extract the response data stored in the first target request log in the historical request log as historical response data;

[0107] The type determination unit is used to input historical response data into the target classification model and determine the data type of the historical response data through the target classification model; the data type includes invariant data and variable data;

[0108] The first data response unit is used to take historical response data as target response data when the data type is determined to be unchanging data.

[0109] The second data response unit is used to determine the corresponding preset data determination rules for historical response data when the data type is determined to be changing data, and to determine the target response data according to the historical response data and the preset data determination rules.

[0110] In one embodiment, the data processing apparatus further includes:

[0111] The data feedback module is used to send business interface requests to the target application cluster when the system's circuit breaker policy is set to non-circuit breaker, so that the target application cluster can provide the target response data according to the business interface request.

[0112] In one embodiment, the data processing apparatus further includes:

[0113] The test case determination module is used to obtain preset system recovery test cases corresponding to attribute characteristics, and execute the preset system recovery test cases according to the preset execution frequency to obtain test case results.

[0114] The strategy update module is used to determine system recovery and update the system circuit breaker strategy to non-circuit breaker when the test case results meet the preset conditions more than a preset number of times.

[0115] The data processing apparatus provided in the embodiments of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0116] Example 5

[0117] Figure 6 This is a schematic diagram of the structure of an electronic device implementing the data processing method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0118] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.

[0121] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0127] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0128] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the data processing method of any embodiment of the present invention.

[0129] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0131] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: Obtain the attribute characteristics of the business interface request, and generate access logs based on the attribute characteristics; Obtain historical request logs, and determine the system circuit breaker strategy based on the access logs, historical request logs, and preset circuit breaker rules; When the system's circuit breaker policy is set to circuit breaker, historical response data is determined, the data type of the historical response data is determined according to the target classification model, and the target response data is determined according to the data type and the historical response data.

2. The method according to claim 1, characterized in that, The step of obtaining the attribute characteristics of the business interface request and generating access logs based on the attribute characteristics includes: Extract the dimension information of the business interface request in the target dimension, and use the dimension information as attribute features; The attribute features are encapsulated into structured logs, and the structured logs are used as access logs.

3. The method according to claim 1, characterized in that, The step of obtaining historical request logs and determining the system circuit breaker strategy based on the access logs, historical request logs, and preset circuit breaker rules includes: Determine the attribute dimensions of the attribute features in the access log, and use the historical request logs matched by the attribute dimensions within the current statistical period as the first target request logs, and use the historical request logs matched by the attribute dimensions within the historical statistical period as the second target request logs; The success rate and call volume of the first target request log are determined as real-time indicators, and the success rate and call volume of the second target request log are determined as historical indicators. Extract the preset circuit breaker rules and determine the difference between the historical indicator and the real-time indicator as the difference indicator; The system circuit breaker strategy is determined according to the preset circuit breaker rules, the difference index, and the real-time index.

4. The method according to claim 3, characterized in that, The preset circuit breaker rules include at least the following: If the real-time indicators meet the preset fixed circuit breaker threshold or the preset window circuit breaker threshold, and the difference indicators meet the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker. If the real-time indicator does not meet the preset fixed circuit breaker threshold and the real-time indicator does not meet the preset window circuit breaker threshold, but the difference indicator meets the preset difference circuit breaker threshold, the system circuit breaker strategy is determined to be circuit breaker.

5. The method according to claim 1, characterized in that, The process of determining historical response data, determining the data type of the historical response data based on a target classification model, and determining target response data according to the data type and the historical response data includes: Extract the response data stored in the first target request log from the historical request log as historical response data; Historical response data is input into a target classification model, and the data type of the historical response data is determined by the target classification model; wherein, the data type includes invariant data and variable data; When it is determined that the data type is constant, the historical response data is used as the target response data; When the data type is determined to be variable data, a preset data determination rule corresponding to the historical response data is determined, and the target response data is determined according to the historical response data and the preset data determination rule.

6. The method according to claim 1, characterized in that, After obtaining the historical request logs and determining the system circuit breaker strategy based on the access logs, historical request logs, and preset circuit breaker rules, the process further includes: When the system's circuit breaker policy is set to non-circuit breaker, the business interface request is sent to the target application cluster so that the target application cluster can respond with target response data according to the business interface request.

7. The method according to claim 1, characterized in that, When the system's circuit breaker policy is set to circuit breaker, the following steps are taken: First, historical response data is determined. Then, the data type of the historical response data is determined according to a target classification model. Finally, the target response data is determined based on the data type and the historical response data. Obtain the preset system recovery test cases corresponding to the attribute characteristics, and execute the preset system recovery test cases according to the preset execution frequency to obtain the test case results; If the test case results meet the preset conditions more than the preset number of times, the system is determined to recover, and the system circuit breaker policy is updated to non-circuit breaker.

8. A data processing apparatus, characterized in that, include: The log determination module is used to obtain the attribute characteristics of business interface requests and generate access logs based on the attribute characteristics. The strategy determination module is used to obtain historical request logs and determine the system circuit breaker strategy based on the access logs, historical request logs and preset circuit breaker rules. The data response module is used to determine historical response data when the system's circuit breaker policy is set to circuit breaker, determine the data type of the historical response data according to the target classification model, and determine the target response data according to the data type and the historical response data.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data processing method according to any one of claims 1-7.

Images