A two-stage-based microservice smell detection method

Abstract

The application discloses a two-stage-based microservice smell detection method, which comprises the following steps: S1, according to the microservice system source code file, adopting specific analysis items, detecting different types of microservice smells based on one or more analysis items; S2, the results of different types of smell detection algorithms are summarized; S3, in the microservice system runtime, real-time collection of runtime container performance logs, runtime path tracking logs, runtime database operation logs; S4, according to the preliminary determination result of the static and running state combined analysis detection method, various microservice runtime logs, the determination of the smell in the running state is carried out; S5, the results of the three different detection methods are summarized and output in the form of JSON string, and each detection result of each smell contains position information. The method adopts a two-stage combined method of static and running state, and can comprehensively detect and accurately position multiple and miscellaneous microservice smells.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to a method for detecting odors in microservice systems, specifically a method for detecting numerous and complex odors in microservices during both static and runtime phases. Background Technology

[0002] A lack of sufficient experience and knowledge in microservice architecture can lead to poor practices and patterns introduced during the design and implementation of microservices; these are known as microservice smells. Microservice smells not only manifest as poor code design but also harm the performance, reliability, and scalability of the microservice system, increasing maintenance difficulty. Microservice smells are highly insidious; if not detected and eliminated in time, they will gradually accumulate and severely damage multiple quality attributes of the system. Timely and accurate detection of microservice smells and guidance for developers to fix them are essential to ensuring the healthy and sustainable operation of the microservice system and guaranteeing good service quality.

[0003] Currently, anomaly detection in microservice systems falls into two main categories. One involves analyzing the microservice source code in the static phase and detecting anomalies using rule-based algorithms. The other involves analyzing real-time data from the microservice system during runtime to detect anomalies. The first method can quickly detect anomalies, but it detects relatively few types of microservice anomalies, and some detection methods have limitations. The second method can detect anomalies generated in real-time during runtime; while accurate, it also detects fewer types of microservice anomalies. Summary of the Invention

[0004] To address the limitations of existing microservice odor detection methods, this invention provides a two-stage microservice odor detection method. This method combines static and runtime phases, enabling comprehensive detection and precise localization of numerous and complex microservice odors. This invention helps microservice developers to promptly repair systems, preventing further damage and ensuring reliable and sustainable system operation.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A two-stage microservice odor detection method includes the following steps:

[0007] Step S1: Based on the microservice system source code files, select specific analysis items (metrics) and, based on one or more analysis items, detect different types of microservice odors. The specific steps are as follows:

[0008] Step S11: Traverse all microservice modules, read the contents of class files and configuration files, and extract five analysis items related to microservice anomalies: code structure analysis, dependency analysis, call relationship analysis, language usage analysis, and configuration file analysis. Specifically: code structure analysis includes analysis of APIs, target string matching, and the number of service module entities; dependency analysis includes the relationship between interfaces and classes, the number of inputs and outputs to classes and interfaces, and the references of each class (interface) to other classes (interfaces); call relationship analysis includes the call situation of each service to other services and the method call situation within each service; language usage analysis refers to the statistical analysis of the usage and proportion of different languages; and configuration file analysis includes a detailed analysis of POM files, YAML files, or Properties files.

[0009] Step S12: Combine one or more analytical items to detect different odors based on this.

[0010] Step S2: Summarize the results of different types of odor detection algorithms in Step S1, including the judgment results of static analysis detection method and the preliminary judgment results of static and dynamic combined analysis detection method.

[0011] Step S3: During the microservice system runtime, collect runtime container performance logs, runtime path tracing logs, and runtime database operation logs in real time. The specific steps are as follows:

[0012] Step S31: Request the Kubernetes API Server to collect performance information of the containers corresponding to different microservices, mainly including container name, container CPU usage and memory usage.

[0013] Step S32: By using custom annotations and custom filters for adding or modifying request header information, service call information is output and persisted when inter-service calls occur in the runtime state. Basic information of different inter-service calls is collected, including the service name and interface name of the callee, the entire service call chain TraceId, the current call chain Id, the predecessor call chain ParentId, the service caller IP and Port, etc. The above basic information is persisted in the log as runtime path tracing information.

[0014] Step S33: Define custom annotations and aspect classes, and use Spring AOP technology to output service access database operation information at runtime, including four types of operations: add, delete, modify, and query; collect data operation information when different service modules access relational databases, including log output time, data access framework, service name accessing the database, database URL, database name, second database URL, second database name, data table name, and data operation.

[0015] Step S4: Based on the preliminary judgment results of the combined static and runtime analysis and detection methods collected in Step S2, and the various microservice runtime logs collected in Step S3, determine the odors in the runtime state. The specific steps are as follows:

[0016] Step S41: Based on the preliminary judgment results of the static and runtime combined analysis detection method collected in step S2, and combined with the runtime path tracking logs and data operation information collected in real time, make the final judgment of odor through a rule-based algorithm.

[0017] Step S42: Based on the container performance logs and runtime path tracing logs collected in step S3, detect the odors that are analyzed using the runtime analysis method.

[0018] Step S5: Summarize the results of the three different detection methods: static analysis, runtime analysis, and combined static and runtime analysis, and output them in JSON string format. Each detection result for each type of odor includes location information.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] 1. This invention combines the static and runtime phases of microservices, enabling the extraction of static metric information and runtime attitude information for detecting numerous and complex microservice anomalies.

[0021] 2. This invention proposes three specific methods for detecting odors in microservices, each applicable to different types of odors. When defining a new type of odor in a microservice, the optimal detection method can be selected based on the characteristics of the odor.

[0022] 3. This invention proposes specific analytical items for detecting microservice odors in the static stage, covering 11 analytical items in 5 aspects, and based on this, proposes detection methods for different types of odors.

[0023] 4. This invention can collect three types of real-time log information of microservices in the running state and detect different types of odors through specific rule-based algorithms. Attached Figure Description

[0024] Figure 1 This is the overall flowchart for the two-stage odor detection.

[0025] Figure 2 A regular expression to match whether an API has a semantic version.

[0026] Figure 3 To match hard-coded regular expressions.

[0027] Figure 4This is a graph showing the call relationships between microservices that have circular dependencies.

[0028] Figure 5 This is a flowchart for detecting cyclically dependent odors.

[0029] Figure 6 This is a JSON fragment of the odor detection results. Detailed Implementation

[0030] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0031] This invention provides a two-stage microservice-based odor detection method, such as... Figure 1 As shown, the method includes the following steps:

[0032] Step S1: Based on the microservice system source code files, adopt specific analysis items (metrics) and detect different types of microservice odors based on one or more analysis items.

[0033] This step primarily involves static detection of microservice smells. It analyzes and traverses all microservice modules, reading the contents of class files and configuration files to extract specific analysis items related to microservice smells. Then, a rule-based algorithm is used to detect different types of microservice smells. The specific steps are as follows:

[0034] Step S11: Traverse all microservice modules, read the contents of class files and configuration files, and extract five specific analysis items: code structure analysis, dependency analysis, call relationship analysis, language usage analysis, and configuration file analysis. Specifically: Code structure analysis includes analysis of APIs, target string matching, and the number of service module entities. Dependency analysis includes the relationship between interfaces and classes, the number of inputs and outputs of classes and interfaces, and the references of each class (interface) to other classes (interfaces). Call relationship analysis includes the call situation of each service to other services and the method call situation within each service. Language usage analysis refers to the statistical analysis of the usage and proportion of different languages. Configuration file analysis includes specific analysis of POM files, YAML files, or Properties files. These five aspects comprise a total of 11 specific analysis items. The key points to be injected when extracting each analysis item are shown in Table 1.

[0035]

[0036] Step S12: Combine one or more analytical items to detect different odors based on this.

[0037] Detecting API smells without a semantic version requires analyzing the API to construct the complete URL, and then performing operations on the target string (the complete URL). Figure 2 The regular expression matching shown ultimately determines whether a semantic version exists. The same analysis item can also be used to detect various smells; for example, target string matching is key to identifying API smells and hard-coded smells that lack semantic versions. The regular expression used to match values ​​of the form IP:Port in hard-coded strings is as follows: Figure 3 As shown. Analyzing each service's calls to other services is fundamental to determining functional dispersion and enterprise service bus-style service relationships. Analyzing method calls within each service is fundamental to identifying underutilized abstract classes and interface smells. Analyzing POM files to extract dependency details is fundamental to identifying shared dependency library smells. Analyzing YAML or Properties files is fundamental to analyzing shared database smells and inappropriate service intimacy smells. Specifically, lack of semantic versioning APIs means the API URL does not contain microservice version information; hardcoding means directly specifying the IP address and port number in the source code; functional dispersion means multiple microservices are responsible for implementing the same high-level problem; enterprise service bus-style service relationships mean overusing a microservice as middleware for inter-service communication; underutilized abstract classes and interfaces mean that only some of their defined methods are implemented and called; shared dependency libraries mean different services share the same dependency; shared databases mean multiple services share the same database; and inappropriate service intimacy means that a microservice, in addition to connecting to its own dedicated database, also connects to other services' dedicated databases to obtain private data.

[0038] Step S2: Summarize the results of different types of odor detection algorithms in Step S1, including the judgment results of static analysis detection method and the preliminary judgment results of static and dynamic combined analysis detection method.

[0039] This step summarizes the odor detection results in the static stage. The preliminary judgment results obtained by combining static and operational analysis will be input into the operational stage for final judgment.

[0040] Step S3: During the operation of the microservice system, collect runtime container performance logs, runtime path tracing logs, and runtime database operation logs in real time.

[0041] This step primarily involves collecting various data generated by microservices in real time during runtime, providing data support for detecting microservice anomalies in the runtime environment. The specific steps are as follows:

[0042] Step S31: Request the Kubernetes API Server to collect performance information of the containers corresponding to different microservices, mainly including container name, container CPU usage and memory usage.

[0043] Step S32: By using custom annotations and custom filters for adding or modifying request header information, service call information is output and persisted during runtime when inter-service calls occur. The entire complete call chain has a unique TraceId, and the current call chain also has a unique ID. Basic information from different inter-service calls is collected, including the service name and interface name of the callee, the TraceId of the entire service call chain, the current call chain ID, the ParentId of the predecessor call chain, and the IP and Port of the service caller. This basic information is persistently stored in the log as runtime path tracing information.

[0044] Step S33: Define custom annotations and aspect classes, and use Spring AOP technology to output service access database operation information at runtime, including four types of operations: add, delete, modify, and query; collect data operation information when different service modules access relational databases. The database operation information specifically includes nine fields: log output time, data access framework, service name accessing the database, database URL, database name, second database URL, second database name, data table name, and data operation.

[0045] Step S4: Based on the preliminary judgment results of the combined analysis and detection method of static and runtime states collected in Step S2, and the various microservice runtime logs collected in Step S3, determine the odor in the runtime state.

[0046] This step primarily involves detecting architectural odors in runtime based on the collected data. This includes the final determination of odors using a combination of static and runtime analysis methods, as well as the determination of odors using runtime analysis alone. The specific steps are as follows:

[0047] Step S41: Based on the preliminary judgment results of the static and runtime combined analysis detection method collected in step S2, and combined with the runtime path tracking logs and data operation information collected in real time, make the final judgment of odor through a rule-based algorithm.

[0048] This detection method combines static and runtime analysis to target two anomalies: shared databases and inappropriate service intimacy. If the performance of the shared database service is unaffected, this situation should be tolerated. Similarly, if analysis of data operation information within the current time window reveals inappropriate intimacy between services without excessively close data operations (the intimacy party performing operations other than reading the private data of the intimacy party), this situation should also be tolerated. Therefore, the detection of both anomalies employs a preliminary static phase, with a final determination based on real-time conditions during the runtime phase.

[0049] Step S42: Based on the container performance logs and runtime path tracing logs collected in step S3, detect the odors that are analyzed using the runtime analysis method.

[0050] Anomalies identified through runtime analysis include five types: circular dependencies, low service availability, uneven service usage, uneven service interface usage, and uneven service resource usage. Among these:

[0051] For detecting circular dependencies, a complete service call chain is constructed based on the collected path tracing log information. Each chain is uniquely identified by an ID, connected to its predecessor chain by a ParentId, and belongs to a specific request chain by a TraceId. Invalid call chains with zero links or whose first link's parentId is not null are removed. Each node in the valid call chain is traversed to construct a call graph, and topological sorting is used to determine if a call cycle exists. If a cycle exists, it indicates a circular dependency issue, and relevant information is recorded. A service call chain with a circular dependency is shown below. Figure 4 As shown, the detection process for circular dependencies is as follows: Figure 5 As shown.

[0052] For detecting service availability anomalies, the system calculates and sorts the response times of each microservice in the current window based on the collected path tracing log information. If 90% of the service response times exceed the threshold, the current service is determined to have this anomaly.

[0053] For detecting uneven service usage, the number of service calls is counted based on collected path tracing logs. If the absolute value of the difference between the number of calls to a certain service and the average number of calls to all services in the microservice system (avgCall) exceeds a threshold λ times stdS, then the anomaly is identified. The standard deviation of the total number of service calls in the entire system, where n represents the total number of services, si represents the service number, and call si This represents the number of times service si calls other services.

[0054] For detecting uneven usage of service interfaces, the service interface call information is extracted based on the collected path tracing log information to determine if the absolute value of the difference between the number of calls to a certain interface i in service s and the average number of calls to the current service interfaces avgI exceeds the threshold θ times stdI (the standard deviation of the number of interface calls in the current service), then the anomaly is determined to exist.

[0055] For detecting uneven service resource usage, system resource consumption is determined based on collected container performance logs, and then assessed from two perspectives: CPU usage and memory usage. If the absolute value of the difference between the CPU usage of service instance j and the average CPU usage (avgCPU) of all instances exceeds a threshold ε times stdCPU, or the absolute value of the difference between the RAM usage of service instance j and the average RAM usage (avgRAM) of all instances exceeds a threshold ε times stdRAM, then the anomaly is identified. Here, stdCPU and stdRAM represent the standard deviations of CPU and RAM usage for the entire system's containers, respectively.

[0056] Step S5: Summarize the results of the three different detection methods: static analysis, runtime analysis, and combined static and runtime analysis, and output them in JSON string format. Each detection result for each type of odor includes location information.

[0057] This step summarizes all odor detection results. The outermost JSON string uses different odor types as keys, the inner JSON string uses the names of different service modules where the odor appears as keys, and the innermost JSON string contains detailed information such as the location and description of the odor. The final output is a JSON string containing all detection results.

[0058] This invention achieves comprehensive and unified detection of microservice odors based on two phases: static and runtime. It utilizes 11 different extracted static analysis items and various collected runtime microservice log information, employing a rule-based algorithm and three different detection methods: static analysis, runtime analysis, and a combination of static and runtime analysis. The proposed two-phase microservice odor detection method has high scalability, and the detection methods for newly defined odor types can be quickly implemented.

[0059] Taking the train-ticket-test system as an example, this two-stage detection method detected no API gateway smells and no shared dependency library smells. No API gateway means that external requests communicate directly with the service, and communication between services is achieved directly through service-to-service calls. Shared dependency libraries refer to situations where different microservices use the same dependency library. A JSON fragment of the smell detection results is shown below. Figure 6 As shown.

Claims

1. A two-stage microservice odor detection method, characterized in that... The method includes the following steps: Step S1: Based on the microservice system source code files, take specific analysis items and detect different types of microservice odors based on one or more analysis items; Step S2: Summarize the results of different types of odor detection algorithms in Step S1, including the judgment results of static analysis detection method and the preliminary judgment results of static and dynamic combined analysis detection method; Step S3: During the operation of the microservice system, collect runtime container performance logs, runtime path tracing logs, and runtime database operation logs in real time. Step S4: Based on the preliminary judgment results of the combined analysis and detection method of static and runtime states collected in Step S2, and the various microservice runtime logs collected in Step S3, determine the odor in the runtime state. For the detection of circular dependencies, a complete service call chain is constructed based on the collected path tracing log information. Each chain is uniquely identified by an ID, connected to the predecessor chain by a ParentId, and identified by a TraceId. Remove invalid call chains with a link count of 0 or whose parentId of the first link is not null; traverse each node of the valid call chain, construct a call graph, and use topological sorting to determine if a call cycle exists in the graph; if it exists, it proves that there is a circular dependency smell, and record the relevant information of the smell. For detecting service availability anomalies, the response times of each microservice under the current window are statistically analyzed and sorted based on the collected path tracing log information. If 90% of the service response times exceed the threshold, the current service is determined to have this anomaly. For detecting uneven service usage, the number of service calls is counted based on the collected path tracing log information. If the absolute value of the difference between the number of calls to a certain service and the average number of calls to all services in the microservice system (avgCall) exceeds the threshold λ times stdS, then the anomaly is determined to exist. stdS is the standard deviation of the number of service calls in the entire system. For detecting uneven use of service interfaces, the service interface call information is extracted based on the collected path tracing log information to determine if the absolute value of the difference between the number of calls to a certain interface i in service s and the average number of calls to the current service interface avgI exceeds the threshold θ times stdI, then the anomaly is determined to exist, where stdI is the standard deviation of the number of calls to the interface in the current service. For detecting uneven service resource usage, the system resource consumption is obtained based on the collected container performance log information, and then judged from the perspectives of CPU usage and memory usage. If the absolute value of the difference between the CPU usage of service instance j and the average CPU usage of all instances (avgCPU) exceeds the threshold ε times stdCPU, or the absolute value of the difference between the RAM usage of service instance j and the average RAM usage of all instances (avgRAM) exceeds the threshold ε times stdRAM, then the anomaly is determined to exist, where stdCPU and stdRAM are the standard deviations of the CPU and RAM usage of the entire system containers, respectively. Step S5: Summarize the results of the three different detection methods: static analysis, runtime analysis, and combined static and runtime analysis, and output them in JSON string format. Each detection result for each type of odor includes location information.

2. The two-stage microservice odor detection method according to claim 1, characterized in that... The specific steps of step S1 are as follows: Step S11: Traverse all microservice modules, read the contents of class files and configuration files, and extract five analysis items related to microservice anomalies: code structure analysis, dependency analysis, call relationship analysis, language usage analysis, and configuration file analysis. Specifically: code structure analysis includes analysis of APIs, target string matching, and the number of service module entities; dependency analysis includes the relationship between interfaces and classes, the number of inputs and outputs to classes and interfaces, and the references of each class / interface to other classes / interfaces; call relationship analysis includes the call situation of each service to other services and the method call situation within each service; language usage analysis refers to the statistical analysis of the usage and proportion of different languages; and configuration file analysis includes a detailed analysis of POM files, YAML files, or Properties files. Step S12: Combine one or more analytical items to detect different odors based on this.

3. The two-stage microservice odor detection method according to claim 1, characterized in that... The specific steps of step S3 are as follows: Step S31: Request the Kubernetes API Server to collect performance information of the containers corresponding to different microservices, mainly including container name, container CPU usage and memory usage; Step S32: By using custom annotations and custom filters for adding or modifying request header information, the service call information is output and persisted when inter-service calls occur in the runtime state. Basic information of different inter-service calls is collected, including the service name and interface name of the callee, the entire service call chain TraceId, the current call chain Id, the predecessor call chain ParentId, the service caller IP and Port. The above basic information is persisted in the log as runtime path tracing information. Step S33: Define custom annotations and aspect classes, and use Spring AOP technology to output service access database operation information at runtime, including four types of operations: add, delete, modify, and query; collect data operation information when different service modules access relational databases, including log output time, data access framework, service name accessing the database, database URL, database name, second database URL, second database name, data table name, and data operation.

4. The two-stage microservice odor detection method according to claim 1, characterized in that... The specific steps of step S4 are as follows: Step S41: Based on the preliminary judgment results of the static and running state combined analysis and detection methods collected in step S2, and combined with the real-time collected runtime path tracking logs and data operation information, make a final judgment on the odor. Step S42: Based on the container performance logs and runtime path tracing logs collected in step S3, detect the odors that are analyzed using the runtime analysis method.

5. The two-stage microservice odor detection method according to claim 4, characterized in that... The specific odors identified through runtime analysis include five types: circular dependencies, low service availability, uneven service usage, uneven service interface usage, and uneven service resource usage.

6. The two-stage microservice odor detection method according to claim 1, characterized in that... The , n represents the total number of services, si represents the service number, call si This represents the number of times service si calls other services.

Images