An automatic configuration and execution method for interface monitoring of large-scale microservices

Abstract

The present invention relates to an automatic configuration and execution method for interface monitoring, and discloses an automatic configuration and execution method for interface monitoring for large-scale microservices, which includes: obtaining online traffic data of a single service for a preset time period in an online log. The extracted online traffic data of a single service for a preset time period is filtered according to the code coverage, and a final set is obtained. The traffic data in the final set is obtained, and the interface monitoring of each service in the service system is configured according to the traffic data in the final set. The priority of each service and the priority of the interface monitoring corresponding to each service are determined based on the configured interface monitoring of each service, and priority monitoring is performed based on the determined priority of each service and the priority of the interface monitoring corresponding to each service. By obtaining the online traffic data of a single service, filtering the data and configuring interface monitoring, the efficiency of system problem troubleshooting is effectively improved, and the priority of key services and interface monitoring is effectively ensured.

Description

Technical Field

[0001] The present invention relates to an automatic configuration and execution method for interface monitoring, and in particular to an automatic configuration and execution method for interface monitoring oriented to large-scale microservices. Background Art

[0002] With the rapid development of Internet technology, the shortcomings of monolithic architecture have become increasingly apparent. To address the tight coupling and high deployment costs of monolithic architecture, microservice architecture has emerged.

[0003] In a microservices architecture, releases can be targeted to specific services with minimal impact, low risk, and low cost. Frequent releases are possible, enabling rapid delivery of requirements. While these advantages are present, microservices also present challenges. Applications evolve from monolithic entities to comprise dozens to hundreds of distinct microservices, coupled with frequent releases. This creates two challenges for microservice interface monitoring: Currently, most interface monitoring in the industry is manually configured by developers. This approach can be disastrous for developers when faced with large-scale microservices. Hundreds of microservices require interface monitoring, and each release requires updates. The workload for configuring interface monitoring increases exponentially. Furthermore, most interface monitoring implementations in the industry are performed out of order. This approach can delay the detection of issues with important services and interfaces when faced with large-scale microservices. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problems existing in the traditional microservice architecture in the above background technology by providing an automatic configuration and execution method for interface monitoring for large-scale microservices.

[0005] In one aspect, an embodiment of the present invention provides a method for automatically configuring and executing interface monitoring for large-scale microservices, including:

[0006] Obtaining online logs, and extracting single-service online traffic data for a preset time period from the online logs;

[0007] Filtering the extracted single service online traffic data for a preset period of time according to code coverage, and obtaining a final set;

[0008] Acquire the flow data in the final set, and configure interface monitoring of each service in the service system according to the flow data in the final set;

[0009] The priority of each service and the priority of the interface monitoring corresponding to each service are determined according to the configured interface monitoring of each service, and priority monitoring is performed according to the determined priority of each service and the priority of the interface monitoring corresponding to each service.

[0010] Furthermore, before filtering the extracted single service online traffic data of the preset period according to the code coverage and obtaining the final set, the method further includes:

[0011] Pre-set sensitive conditions;

[0012] Perform sensitive data screening on single service online traffic data within a preset period based on pre-set sensitive conditions;

[0013] Replace the sensitive data of the filtered single service online traffic data for the preset time period.

[0014] Furthermore, the extracted single service online traffic data of the preset time period is screened according to the code coverage, and a final set is obtained, including:

[0015] Replay the traffic on the single service line during a preset period and pre-set a threshold H;

[0016] Dividing the traffic on the single service line during a preset period into a plurality of aggregated traffic flows;

[0017] Performing serial traffic revisit on the plurality of the set flows, and obtaining a plurality of code coverage lines in the plurality of the set flows;

[0018] Calculating the similarity between any two of the plurality of set flows based on the weighted Jaccard, and selecting the two set flows with the lowest similarity from the plurality of set flows to be included in the target set;

[0019] Based on the greedy algorithm, the remaining set flows with the lowest similarity to the target set are selected from the set flows, and merged into the target set until a specified condition greater than a threshold H is met, wherein,

[0020] If the similarity between the selected remaining set traffic and the target set is less than a threshold H, then the remaining set traffic is merged into the target set;

[0021] If the similarity between the remaining selected set traffic and the target set is greater than a threshold H, the target set is deduplicated to obtain a final set.

[0022] Furthermore, when calculating the similarity between any two of the plurality of set flows based on the weighted Jaccard, the method includes:

[0023] Obtaining a number of said aggregate flows Sn, Sn = S1, S2 . . . Sn;

[0024] Obtain the number of code coverage lines Lm in the set traffic Sn, where Lm=L1, L2...Lm;

[0025] Obtain the number of times W of the code coverage line number Lm in the single set flow Sn, W = 1, 2 ... W, and determine the number of times W of the code coverage line number Lm in the single set flow Sn as the weight of the single set flow Sn, set W(S n ,L m );

[0026] Based on the formula

[0027]

[0028] Obtaining similarities between Sn and Sj in the plurality of set flows;

[0029] Among them, Sn is a set of traffic, Sj is another set of traffic, i represents the total number of code coverage lines, and the min function and max function represent obtaining the smaller and larger of the two parameters respectively.

[0030] Furthermore, when dividing the traffic on the single service line in the preset time period and dividing it into a number of aggregate traffic flows, it includes: the traffic of the several aggregate traffic flows is 5% of the traffic on the single service line in the preset time period.

[0031] Furthermore, when serial traffic revisit is performed on the plurality of the collective traffics and a plurality of code coverage lines in the plurality of the collective traffics are obtained, the plurality of the obtained code coverage lines are not deduplicated.

[0032] Furthermore, when the flow data in the final set is obtained and interface monitoring of each service in the service system is configured according to the flow data in the final set, the process includes:

[0033] Obtaining the request address of the traffic in the final set, and configuring the request address of the interface monitoring according to the request address of the traffic in the final set;

[0034] Obtaining a request method for the traffic in the final set, and configuring the request method for the interface monitoring according to the request method for the traffic in the final set;

[0035] Obtaining request input parameters of the traffic in the final set, and configuring the request input parameters of the interface monitoring according to the request input parameters of the traffic in the final set;

[0036] The status code 200 is configured to check the status code of the interface monitoring.

[0037] Furthermore, when determining the priority of each service and the priority of the interface monitoring corresponding to each service according to the configured interface monitoring of each service, the method includes:

[0038] Obtain the call status of the configured interface monitoring, and generate a microservice system topology diagram based on the call status of the configured interface monitoring;

[0039] The priority of each of the services and the priority of the interface monitoring corresponding to each of the services are determined according to the generated microservice system topology diagram.

[0040] Furthermore, when determining the priority of each of the services according to the generated microservice system topology diagram, it includes:

[0041] Abstracting the generated microservice system topology diagram into a directed graph, and defining the services as nodes in the directed graph;

[0042] Obtaining the number of arrows pointing to the service in the directed graph, and setting the number of arrows pointing to the service in the directed graph as the in-degree of the service;

[0043] Obtaining the number of arrows pointing out of the service in the directed graph, and setting the number of arrows pointing out of the service in the directed graph as the out-degree of the service;

[0044] And based on the formula

[0045] D i =I i +O i

[0046] Obtaining the degree of said service;

[0047] Where D is the degree of the service, I is the in-degree of the service, O is the out-degree of the service, i=1,2…i;

[0048] The services are ranked in reverse order according to their degrees, and the priority of the services is determined according to the reverse ranking.

[0049] Furthermore, when determining the priority of interface monitoring corresponding to each of the services according to the generated microservice system topology diagram, it includes:

[0050] Obtaining the number of times the interface monitoring appears in the target set, and determining the number of times the interface monitoring appears in the target set as the importance of the interface monitoring;

[0051] Calculating the importance of each interface monitoring in the final set;

[0052] The interface monitoring items in the final set are sorted in reverse order according to their importance, and the priority of each interface monitoring item is determined according to its ranking.

[0053] Furthermore, when priority monitoring is performed according to the determined priority of each of the services and the priority of the interface monitoring corresponding to each of the services, the method includes:

[0054] Obtaining a service with a priority higher than that of each of the services, and monitoring the service first;

[0055] If the priorities of the services are the same, the interface monitoring of the service with a priority higher than the interface monitoring priority corresponding to the services is selected, and the service corresponding to the interface monitoring is monitored preferentially.

[0056] Compared to existing technologies, the method for automatically configuring and executing interface monitoring for large-scale microservices, according to an embodiment of the present invention, offers the following advantages: By acquiring online logs and extracting online traffic data for a single service over a preset period, critical real-time traffic information can be obtained. This helps system administrators better understand inter-service communication and resource consumption. Secondly, by filtering the extracted traffic data based on code coverage, noise can be effectively reduced, data quality can be improved, and only the most critical services and interfaces are monitored. This helps reduce the burden on the monitoring system and improve performance. Furthermore, interface monitoring for each service is automatically configured based on actual conditions, reducing manual intervention and improving configuration consistency. Furthermore, service priority is assessed using node degrees in a directed graph. Compared to manual prioritization, assessing service priority based on node degrees is more accurate. Furthermore, when services change, service priorities can be updated in real time. Finally, by prioritizing service and interface monitoring, critical services can be more targeted and ensured to function properly, effectively improving system reliability and stability.

[0057] Specifically, an embodiment of the present invention provides an automatic configuration and execution method for interface monitoring of large-scale microservices. It uses a weighted Jaccard coefficient to calculate the similarity of traffic sets. Compared to the traditional Jaccard coefficient, this method considers the impact of code line repetitions on set similarity, making similarity calculation more accurate. Furthermore, by employing a greedy algorithm, the set with the lowest similarity to the target set T is selected each time. Compared to randomly selecting sets, this method can select the optimal set combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of a directional Figure 1 A flowchart of an automatic configuration and execution method for interface monitoring of large-scale microservices.

[0059] Figure 2 Schematic diagram of the microservice system topology in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0061] like Figure 1 As shown, an embodiment of the present invention provides an automatic configuration and execution method for interface monitoring of large-scale microservices, including:

[0062] Step S100: Obtain online logs, and extract single-service online traffic data for a preset period from the online logs.

[0063] Step S200: Filter the extracted single-service online traffic data for a preset period of time according to the code coverage, and obtain a final set.

[0064] Step S300: Obtain the traffic data in the final set, and configure the interface monitoring of each service in the service system according to the traffic data in the final set.

[0065] Step S400: Determine the priority of each service and the priority of the interface monitoring corresponding to each service according to the configured interface monitoring of each service, and perform priority monitoring according to the determined priority of each service and the priority of the interface monitoring corresponding to each service.

[0066] As can be seen, in step S100, online logs are collected and single-service online traffic data for a specific period of time is extracted from these logs to construct basic data for interface monitoring. This step facilitates real-time monitoring of service performance and availability because it captures the actual traffic conditions of the service during a specific period, providing an accurate data source for interface monitoring. Secondly, in step S200, data is filtered using code coverage to select data with important reference value from the extracted traffic data and form a final set. This filtering process retains only data valuable for performance and availability monitoring based on the criticality and code coverage of the service, thereby reducing unnecessary data redundancy and improving monitoring accuracy and efficiency. The traffic data from the final set is then used in step S300 to configure interface monitoring for each service in the service system. This step effectively implements automated monitoring configuration, automatically adjusting monitoring parameters based on actual traffic data to adapt to the actual system operation. This adaptive monitoring configuration helps reduce the complexity of manual configuration and ensures that monitoring policies are consistent with actual needs. Finally, in step S400, the priority of each service and the corresponding interface monitoring priority are determined based on the configured interface monitoring. This helps ensure that critical services are monitored more frequently and with higher priority, and, when resources are limited, allocates monitoring resources to meet the needs of individual services in the system. This step improves the flexibility and intelligence of monitoring, ensuring that important services are always in the focus of monitoring.

[0067] As can be understood, step S100 enables real-time monitoring of service performance and availability by acquiring online logs and extracting online traffic data for a single service over a preset period. This step, by analyzing online traffic data, helps system administrators quickly capture and understand service operations, identify potential issues and bottlenecks, and proactively take necessary measures to maintain and optimize system performance. The subsequent step S200 introduces code coverage filtering to select and prioritize the most valuable traffic data, forming a final set. This filtering process reduces data redundancy and invalid information, ensuring that only data meaningful for performance monitoring is retained. This helps improve monitoring efficiency, reduces storage and processing burdens, and maintains the quality and availability of monitoring data. Next, step S300 uses the traffic data in the final set to automatically configure interface monitoring for each service in the service system. This automated monitoring configuration process adaptively adjusts monitoring parameters based on actual traffic data, ensuring that monitoring policies align with actual needs. This reduces manual configuration workload and improves configuration accuracy. Finally, in step S400, based on the configured interface monitoring, the priority of each service and the corresponding interface monitoring priority are determined. This helps ensure that critical services are monitored more frequently and with higher priority, and, when resources are limited, allocates monitoring resources to meet the needs of individual services in the system. This step improves the flexibility and intelligence of monitoring, ensuring that important services are always in the focus of monitoring.

[0068] Preferably, the preset period is 7 days of historical single service online traffic data from the detection date.

[0069] Specifically, in some embodiments of the present application, before filtering the extracted single-service online traffic data for a preset period based on code coverage and obtaining a final set, the method further includes: presetting sensitive conditions. The single-service online traffic data for the preset period is filtered for sensitive data based on the pre-set sensitive conditions. The sensitive data in the filtered single-service online traffic data for the preset period is replaced.

[0070] It can be seen that by considering the screening of sensitive data and applying pre-defined sensitive conditions, traffic data carrying sensitive information can be effectively identified and filtered. Subsequently, the filtered sensitive data will be replaced to ensure that private data is not leaked.

[0071] It's understandable that pre-setting sensitive conditions helps effectively identify data that may contain sensitive information, reducing potential privacy risks. Secondly, replacing filtered sensitive data further enhances data privacy protection and ensures the security of user privacy. This not only helps comply with privacy regulations and compliance requirements, but also builds user trust while maintaining data validity and providing usable data for subsequent monitoring and analysis.

[0072] Specifically, in some embodiments of the present application, the extracted single-service online traffic data of a preset time period is screened according to the code coverage, and when the final set is obtained, it includes: replaying the single-service online traffic of the preset time period and presetting a threshold H. The single-service online traffic of the preset time period is divided and divided into several set traffics. The several set traffics are serially revisited and the number of code coverage lines in the several set traffics is obtained. The similarity between the two set traffics in the several set traffics is calculated based on the weighted Jaccard, and the two set traffics with the lowest similarity in the several set traffics are selected and included in the target set. Based on the greedy algorithm, the remaining set traffics in the several set traffics are selected with the set traffic with the lowest similarity to the target set, and merged into the target set until the specified condition greater than the threshold H is met, wherein: if the similarity between the selected remaining set traffic and the target set is less than the threshold H, it is merged into the target set. If the similarity between the selected remaining set traffic and the target set is greater than the threshold H, the target set is deduplicated to obtain the final set.

[0073] As can be seen, by replaying traffic on a single service line during a preset time period, the data is segmented into multiple traffic sets, and the similarities between them are calculated. By calculating similarity based on weighted Jaccard, the two traffic sets with the lowest similarity are selected and merged into the target set. A greedy algorithm is then used to gradually merge the remaining traffic sets until the specified conditions are met. This approach helps filter out representative and diverse data, improving the diversity of monitored data and facilitating more comprehensive detection of interface performance and issues. Furthermore, by setting a threshold H, control can be exercised during the data selection process to ensure the quality and relevance of the final collection.

[0074] As you can see, by replaying traffic on a single service line for a preset period, performing data segmentation, and serializing traffic replay, we can obtain more comprehensive code coverage. This helps more comprehensively detect interface performance and issues, thereby improving the accuracy of interface monitoring. This means that system developers and operations teams can better understand interface behavior, more quickly identify and diagnose potential issues, and thus improve the overall quality of the system. Secondly, we use weighted Jaccard similarity calculation and a greedy algorithm to select aggregated traffic, ensuring that the final aggregate contains diverse data. Similarity calculation helps filter out the most dissimilar traffic, thereby better representing different use cases and scenarios. This diverse monitoring data helps developers better understand system performance in different situations, providing a broader reference for performance optimization and troubleshooting. Furthermore, setting a threshold H and implementing a traffic merging strategy ensures the relevance of the final aggregate. If the similarity is below the threshold H, traffic is merged, which helps reduce redundant data and noise. If the similarity is above the threshold H, the target aggregate is deduplicated, eliminating redundant data. This data processing approach helps maintain the cleanliness and accuracy of monitoring data.

[0075] Specifically, in some embodiments of the present application, when calculating the similarity between two sets of traffic based on the weight Jaccard, the method includes: obtaining a plurality of set traffic Sn, Sn = S1, S2 ... Sn. Obtaining the number of code coverage lines Lm in the plurality of set traffic Sn, Lm = L1, L2 ... Lm. Obtaining the number of times W of the code coverage line Lm in a single set traffic Sn, W = 1, 2 ... W, and determining the number of times W of the code coverage line Lm in the single set traffic Sn as the weight of the single set traffic Sn, setting W(S n ,L m ).

[0076] Based on the formula

[0077]

[0078] Obtain the similarity between Sn and Sj in several traffic sets. Where Sn is one traffic set, Sj is another traffic set, i represents the total number of code coverage lines, and the min and max functions respectively obtain the smaller and larger of the two parameters.

[0079] Specifically, in some embodiments of the present application, when dividing the traffic on a single service line in a preset period into several aggregate traffic flows, the traffic flows of the several aggregate traffic flows are 5% of the traffic on the single service line in the preset period.

[0080] It can be understood that by dividing the traffic on a single service line in a preset time period into several aggregated traffic flows, where each aggregated traffic flow accounts for only 5% of the total traffic flow, segmented management of data is achieved. This helps to reduce the complexity of data processing, reduces the size of a single data set, and improves the efficiency of data processing and analysis. The way of dividing data sets makes data easier to manage and process, which helps to reduce the demand for computing and storage resources. At the same time, dividing large data sets into small aggregated traffic flows also facilitates distributed processing, improves the parallelism of data processing, and thus reduces processing time. This effective data segmentation strategy helps to apply large-scale data more efficiently in the interface monitoring system, thereby effectively improving the response speed and shortening the diagnosis and processing time of problems.

[0081] Specifically, in some embodiments of the present application, when serial traffic revisits are performed on a plurality of set traffic flows and a plurality of code coverage lines in the plurality of set traffic flows are obtained, the obtained plurality of code coverage lines are not deduplicated.

[0082] It is understandable that by completely capturing and retaining the code coverage lines in several sets of traffic without performing deduplication operations, it is ensured that each code coverage line is included in the analysis, regardless of whether it appears in multiple sets of traffic. This helps to understand the actual coverage of each code coverage line in detail, including its frequency and context in different traffic sets. At the same time, by not performing deduplication operations, the code coverage can be analyzed more comprehensively, especially for those lines of code that appear in multiple traffic sets. This helps to detect potential problems, anomalies or repeated coverage, and provides more data support for improving the quality of interface monitoring.

[0083] Specifically, in some embodiments of the present application, when obtaining the traffic data in the final set and configuring the interface monitoring of each service in the service system according to the traffic data in the final set, it includes: obtaining the request address of the traffic in the final set, and configuring the request address of the interface monitoring according to the request address of the traffic in the final set. Obtaining the request method of the traffic in the final set, and configuring the request method of the interface monitoring according to the request method of the traffic in the final set. Obtaining the request input parameters of the traffic in the final set, and configuring the request input parameters of the interface monitoring according to the request input parameters of the traffic in the final set. Configure the status code check value of the interface monitoring with status code 200.

[0084] As can be seen, by obtaining the traffic request address, interface monitoring can be configured to ensure that the specific request address meets the expected one. Next, obtaining the traffic request method allows configuring interface monitoring to verify that the request method matches the required method. In addition, obtaining the traffic request input parameters allows configuring interface monitoring to check whether the input parameters meet the specifications. Finally, by configuring the status code 200, the status code check value can be set to verify that the interface returns a successful response status.

[0085] It is understandable that the configuration process allows the system to be precisely configured for the specific needs of different interfaces to ensure the accuracy of monitoring targets. This helps capture specific problems and anomalies without generating false positives or omissions. Secondly, by configuring the request address, request method, and request input parameters, the system can enhance the security of interface monitoring. It can check whether the interface is affected by improper requests and help prevent malicious behavior. At the same time, the configuration of status code 200 allows the system to focus on monitoring successful responses, reducing the monitoring of unnecessary failure states and improving monitoring efficiency. Finally, by configuring the checksum value, the system can quickly identify interfaces with abnormal status codes, helping engineers quickly diagnose problems and take corrective measures.

[0086] See Figure 2 As shown, in some embodiments of the present application, determining the priority of each service and the priority of the interface monitoring corresponding to each service based on the configured interface monitoring of each service includes: obtaining the call status of the configured interface monitoring, and generating a microservice system topology diagram based on the call status of the configured interface monitoring. Based on the generated microservice system topology diagram, the priority of each service and the priority of the interface monitoring corresponding to each service are determined.

[0087] As can be seen, a topology diagram of the microservice system can be generated based on the call status of interface monitoring. This diagram shows the dependencies between microservices, illustrating the call paths and data flows between services. Secondly, based on the generated microservice system topology diagram, the priority of each service can be determined. This typically involves factors such as the importance of the service, its dependencies on other services, and its performance. By comprehensively considering these factors, the system can assign a priority to each service. Furthermore, the priority of the interface monitoring corresponding to each service can also be determined. This is achieved based on the importance of the service and the criticality of each interface within the service. Critical interfaces are typically given higher monitoring priority to ensure their high availability and performance.

[0088] As you can see, by capturing the call status of configured interface monitoring, you can track the actual interactions between services, including call frequency and response time. Using this data to generate a microservice system topology diagram helps developers and operations teams more clearly understand the service dependencies and processes within the system. The resulting topology diagram is further used to prioritize each service and its corresponding interface monitoring. This allows for more granular resource allocation and focus on critical services. For example, high-priority services and interfaces can be allocated more monitoring and resources to ensure high availability and performance. Meanwhile, lower-priority services can have their resource allocation reduced, resulting in more efficient cost and resource management.

[0089] Specifically, in some embodiments of the present application, when determining the priority of each service based on the generated microservice system topology diagram, the method includes: abstracting the generated microservice system topology diagram into a directed graph, and defining the service as a node in the directed graph. , obtaining the number of arrows pointing to the service in the directed graph, and setting the number of arrows pointing to the service in the directed graph as the in-degree of the service. , obtaining the number of arrows pointing to the service in the directed graph, and setting the number of arrows pointing to the service in the directed graph as the out-degree of the service. , and based on the formula

[0090] D i =I i +O i

[0091] Get the degree of the service. Where D is the degree of the service, I is the in-degree of the service, and O is the out-degree of the service, i = 1, 2, ..., i. Sort the services in reverse order by their degrees and determine their priority based on the reverse order ranking.

[0092] Specifically, in some embodiments of the present application, when determining the priority of the interface monitoring corresponding to each service based on the generated microservice system topology diagram, the method includes: obtaining the number of times the interface monitoring appears in the target set, and determining the number of times the interface monitoring appears in the target set as the importance of the interface monitoring. The importance of each interface monitoring in the final set is calculated. The interface monitoring is sorted in reverse order according to the importance of each interface monitoring in the final set, and the priority of each interface monitoring is determined based on the ranking of each interface monitoring.

[0093] It can be seen that by abstracting the topology of the microservice system into a directed graph, in which services are defined as nodes in the graph. By calculating the in-degree (the number of arrows pointed to) and out-degree (the number of arrows pointing out) of each service, as well as the degree formula, the degree of each service can be obtained, and then the priority of each service can be determined. This is based on the degree of interdependence of services in the system. Services with higher degrees are generally more important and therefore have higher priorities. Further, for the priority of interface monitoring, the solution includes calculating the number of times each interface monitoring appears in the target set, which is regarded as the importance of the interface monitoring. By calculating the importance of each interface monitoring in the final set, the interface monitoring can be sorted, with important monitoring placed in front to determine the priority of each interface monitoring. This helps ensure that resources and attention are more focused on those interfaces that are more important to system performance and stability.

[0094] As can be understood, the microservice system's topology is abstracted into a directed graph, where each service is defined as a node in the graph. By calculating each service's in-degree (the number of arrows pointing to it) and out-degree (the number of arrows pointing out of it), and using the degree formula, the degree of each service can be determined. The degree represents the relative importance of the service in the system; services with higher degrees are generally more important. By ranking services in descending order of degree, the priority of each service can be determined. This helps system developers and operations teams better understand which services have the greatest impact on the performance and stability of the overall system. Furthermore, for interface monitoring prioritization, the solution involves calculating the number of times each interface monitoring feature appears in the target set and using this number as the interface monitoring feature's importance. By calculating the importance of each interface monitoring feature in the final set, the interface monitoring features can be sorted, with the most important features at the top, thereby determining their priority. This helps ensure that resources and attention are more focused on those interface monitoring features that are most critical to system performance and stability.

[0095] Specifically, in some embodiments of the present application, when determining the priority of the interface monitoring corresponding to each service based on the generated microservice system topology diagram, the method includes: obtaining the number of times the interface monitoring appears in the target set, and determining the number of times the interface monitoring appears in the target set as the importance of the interface monitoring. The importance of each interface monitoring in the final set is calculated. The interface monitoring is sorted in reverse order according to the importance of each interface monitoring in the final set, and the priority of each interface monitoring is determined based on the ranking of each interface monitoring.

[0096] As can be seen, the importance of each interface monitoring function is determined by counting the number of times each interface monitoring function appears in the target set. This importance reflects the frequency of interface monitoring in actual traffic, with more frequent monitoring functions generally indicating greater importance. The interface monitoring functions in the final set are then sorted in descending order of importance, with the most important functions at the top, thus determining the priority of each interface monitoring function.

[0097] As you can understand, by obtaining the number of times an interface monitoring item appears in the target set, this number is defined as the importance of the interface monitoring item. This importance reflects how frequently the interface monitoring item appears in actual traffic, with more frequent occurrences generally indicating greater importance. By incorporating this metric into monitoring decisions, we can more accurately determine which interface monitoring items are most critical. Next, by calculating the importance of each interface monitoring item in the final set, we sort them in descending order of importance, with the most important items at the top. This sorting method allows for more targeted processing of monitoring data, prioritizing the most critical interface monitoring items. This further enables more effective and rapid identification and resolution of potential issues.

[0098] Specifically, in some embodiments of the present application, when performing priority monitoring based on the determined priorities of each service and the priorities of the interface monitoring corresponding to each service, the method includes: obtaining a service with a priority higher than the priority of each service, and giving priority to monitoring the service. If the priorities of the services are the same, then selecting the interface monitoring of the service with a priority higher than the priority of the interface monitoring corresponding to each service, and giving priority to monitoring the service corresponding to the interface monitoring.

[0099] As can be seen, when priority monitoring is required, the solution obtains the priority of each service and then selects those services with higher priorities for monitoring. This ensures that services deemed more important are monitored more frequently. If the priorities of the services are the same, the solution prioritizes the interface monitoring of services whose corresponding service priorities are higher than the interface monitoring priorities of the services. This approach helps ensure that the resource and attention allocation of those interface monitoring services is better managed to address challenges in system performance and stability.

[0100] As you can understand, by obtaining services with higher priorities than all other services, we can monitor those services deemed most important first. This helps us respond quickly to issues related to critical services and reduces the scope of any outages. This also enables the operations team to allocate resources and attention more effectively, managing and maintaining the system. Secondly, when services have the same priority, the service with a higher priority than the interface monitoring corresponding to each service will be selected and prioritized for interface monitoring. This ensures that issues related to interface monitoring are also addressed quickly. If a particular interface monitoring is deemed critical, it can be ensured to receive sufficient monitoring and resource allocation to promptly detect and resolve issues related to that interface, further improving system availability and performance.

[0101] In the specific embodiment of this application, the above steps are implemented as follows:

[0102] From the online logs, obtain the online traffic data for service 1 for the past seven days, totaling 1,000 records. The online traffic data is shown in the following table.

[0103]

[0104]

[0105] Desensitize the traffic data in the above steps and replace sensitive user data, such as account names and passwords.

[0106] The online traffic data after desensitization is shown in the following table.

[0107]

[0108]

[0109] Set the threshold to 0.9.

[0110] Divide the traffic into 200 sets: S1, S2...S 200 The traffic volume of a single collection is 50.

[0111] In an offline environment, using a single set of traffic for serial traffic playback, the code coverage lines are shown in the following table:

[0112] The following table shows the code coverage line count.

[0113] gather Lines of code covered <![CDATA[S1]]> 4，1，4，2，2，1，2，4，2

[0114] Repeat the above steps to obtain the number of code coverage lines to obtain the number of code coverage lines for 200 traffic sets.

[0115] gather Lines of code covered <![CDATA[S1]]> 4，1，4，2，2，1，2，4，2 <![CDATA[S2]]> 3，2，3，3，2，2，3，4 …… …… <![CDATA[S 200 ]]> ……

[0116] Using weighted Jaccard, calculate the pairwise set S i 、S j The similarity between them.

[0117] The weight of set S1 on code line L1 is: W(S1, L1) = 2. The similarity between S1 and S2 is: Calculate the similarity between two sets:

[0118] Sim(S1,S3)……Sim(S1,S 200 ), Sim(S2,S3)……Sim(S2,S 200 )……Sim(S 199 ,S 200 ).

[0119] Select the two sets S with the lowest similarity 20 、S 140 , join the target set T.

[0120] Calculate the similarity between T and the remaining sets, and between any two sets: Sim(T,S1)…Sim(T,S 200 ) Select the set S with the lowest similarity to the target set T 55 , and merged into the target set T.

[0121] Calculate the new selected set S r Similarity with T Sim(S r ,T)When Sim(S r ,T)<0.9, repeat the previous step. When Sim(S r ,T)≥0.9, the target set T is deduplicated and the final set Q is obtained.

[0122] Finally, the flow in T is shown in the following table.

[0123]

[0124] The final traffic volume is shown in the following table.

[0125]

[0126]

[0127] Configure the service interface monitoring based on the traffic in the final set Q.

[0128] The interface monitoring of the service is shown in the following table.

[0129]

[0130] Repeat the above steps to configure interface monitoring for each service.

[0131] Generate a microservice system topology diagram based on the interface call situation.

[0132] Assess the service priority based on the degree of each service in the structure diagram.

[0133]

[0134]

[0135] Service priorities from high to low: service 7, service 1, service 2, service 3, service 5, service 6, service 8, service 4.

[0136] Assess monitoring priority based on interface importance.

[0137] The importance of interfaces in the final set Q is shown in the following table.

[0138]

[0139] Perform monitoring based on service priority and monitoring priority.

[0140] Service monitoring execution order: service 7, service 1, service 2, service 3, service 5, service 6, service 8, service 4.

[0141] The monitoring execution order of service 1 is shown in the following table.

[0142]

[0143] The above scenarios are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0144] In summary, embodiments of the present invention provide a method for automatically configuring and executing interface monitoring for large-scale microservices. By acquiring online logs and extracting single-service online traffic data for a preset period, critical real-time traffic information can be obtained. This helps system administrators better understand inter-service communication and resource consumption. Secondly, by filtering the extracted traffic data based on code coverage, noise can be effectively reduced, data quality can be improved, and only the most critical services and interfaces are monitored. This helps reduce the burden on the monitoring system and improve performance. Furthermore, interface monitoring for each service is automatically configured based on actual conditions, reducing manual intervention and improving configuration consistency. Furthermore, service priority is assessed using node degrees in a directed graph. Compared to manual prioritization, assessing service priority based on node degrees is more accurate. Furthermore, when services change, service priorities can be updated in real time. Finally, by prioritizing service and interface monitoring, critical services can be more targeted and ensured to function properly, thereby effectively improving system reliability and stability.

[0145] Furthermore, in an embodiment of the present invention, a method for automatically configuring and executing interface monitoring for large-scale microservices uses a weighted Jaccard coefficient to calculate the similarity of traffic sets. Compared to the traditional Jaccard coefficient, this method considers the impact of code line repetitions on set similarity, making similarity calculation more accurate. Furthermore, by employing a greedy algorithm, the set with the lowest similarity to the target set T is selected each time. Compared to randomly selecting sets, this method can select the optimal set combination.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for automatic configuration and execution of interface monitoring for large-scale microservices, characterized in that: include: Obtaining online logs, and extracting single-service online traffic data for a preset time period from the online logs; Filtering the extracted single service online traffic data for a preset period of time according to code coverage, and obtaining a final set; Acquire the flow data in the final set, and configure interface monitoring of each service in the service system according to the flow data in the final set; The priority of each service and the priority of the interface monitoring corresponding to each service are determined according to the configured interface monitoring of each service, and priority monitoring is performed according to the determined priority of each service and the priority of the interface monitoring corresponding to each service.

2. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 1, wherein: The extracted single service online traffic data of the preset time period is screened according to the code coverage, and before obtaining the final set, the method further includes: Pre-set sensitive conditions; Perform sensitive data screening on single service online traffic data within a preset period based on pre-set sensitive conditions; Replace the sensitive data of the filtered single service online traffic data for the preset time period.

3. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 2, wherein: The extracted single service online traffic data for a preset period is screened according to the code coverage, and a final set is obtained, including: Replay the traffic on the single service line during a preset period and pre-set a threshold H; Dividing the traffic on the single service line during a preset period into a plurality of aggregated traffic flows; Performing serial traffic revisit on the plurality of set flows, and obtaining a plurality of code coverage lines in the plurality of set flows; Calculating the similarity between any two of the plurality of set flows based on the weighted Jaccard, and selecting the two set flows with the lowest similarity from the plurality of set flows to be included in the target set; Based on the greedy algorithm, the remaining set flows with the lowest similarity to the target set are selected from the set flows, and merged into the target set until a specified condition greater than a threshold H is met, wherein, If the similarity between the selected remaining set traffic and the target set is less than a threshold H, then the remaining set traffic is merged into the target set; If the similarity between the remaining selected set traffic and the target set is greater than a threshold H, the target set is deduplicated to obtain a final set.

4. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 3, wherein: When calculating the similarity between any two of the plurality of set flows based on the weighted Jaccard, the method includes: Obtaining a number of said aggregate flows Sn, Sn = S1, S2 . . . Sn; Obtain the number of code coverage lines Lm in the set traffic Sn, where Lm=L1, L2...Lm; Obtain the number of times W of the code coverage line number Lm in the single set flow Sn, W = 1, 2 ... W, and determine the number of times W of the code coverage line number Lm in the single set flow Sn as the weight of the single set flow Sn, set W(S n ,L m ); Based on the formula Obtaining similarities between Sn and Sj in the plurality of set flows; Among them, Sn is a set of traffic, Sj is another set of traffic, i represents the total number of code coverage lines, and the min function and max function represent obtaining the smaller and larger of the two parameters respectively.

5. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 3, wherein: When dividing the single service online traffic in the preset time period into a plurality of aggregate traffic flows, the flow of the plurality of aggregate traffic flows is 5% of the single service online traffic in the preset time period.

6. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 3, wherein: When performing serial traffic revisit on the plurality of the set flows and obtaining a plurality of code coverage line numbers in the plurality of the set flows, the obtained plurality of the code coverage line numbers are not deduplicated.

7. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 3, wherein: Acquiring the flow data in the final set and configuring interface monitoring of each service in the service system according to the flow data in the final set includes: Obtaining the request address of the traffic in the final set, and configuring the request address of the interface monitoring according to the request address of the traffic in the final set; Obtaining a request method for the traffic in the final set, and configuring the request method for the interface monitoring according to the request method for the traffic in the final set; Obtaining request input parameters of the traffic in the final set, and configuring the request input parameters of the interface monitoring according to the request input parameters of the traffic in the final set; The status code 200 is configured to check the status code of the interface monitoring.

8. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 7, wherein: Determining the priority of each service and the priority of the interface monitoring corresponding to each service according to the configured interface monitoring of each service includes: Obtain the call status of the configured interface monitoring, and generate a microservice system topology diagram based on the call status of the configured interface monitoring; Determine the priority of each of the services and the priority of the interface monitoring corresponding to each of the services according to the generated microservice system topology diagram; Determining the priority of each of the services according to the generated microservice system topology diagram includes: Abstracting the generated microservice system topology diagram into a directed graph, and defining the services as nodes in the directed graph; Obtaining the number of arrows pointing to the service in the directed graph, and setting the number of arrows pointing to the service in the directed graph as the in-degree of the service; Obtaining the number of arrows pointing out of the service in the directed graph, and setting the number of arrows pointing out of the service in the directed graph as the out-degree of the service; And based on the formula D i =I i +O i Obtaining the degree of said service; Where D is the degree of the service, I is the in-degree of the service, O is the out-degree of the service, i=1,2…i; The services are ranked in reverse order according to their degrees, and the priority of the services is determined according to the reverse ranking.

9. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 8, wherein: Determining the priority of interface monitoring corresponding to each of the services according to the generated microservice system topology diagram includes: Obtaining the number of times the interface monitoring appears in the target set, and determining the number of times the interface monitoring appears in the target set as the importance of the interface monitoring; Calculating the importance of each interface monitoring in the final set; The interface monitoring items in the final set are sorted in reverse order according to their importance, and the priority of each interface monitoring item is determined according to its ranking.

10. The method for automatically configuring and executing interface monitoring for large-scale microservices according to claim 8, wherein: When priority monitoring is performed based on the determined priority of each of the services and the priority of the interface monitoring corresponding to each of the services, it includes Obtaining a service with a priority higher than that of each of the services, and monitoring the service first; If the priorities of the services are the same, the interface monitoring of the service with a priority higher than the interface monitoring priority corresponding to the services is selected, and the service corresponding to the interface monitoring is monitored preferentially.

Images