A method, device and medium for locating a root cause of a database memory overflow

Abstract

The application discloses a database memory overflow root cause positioning method, equipment and medium, relates to the computer technical field, and the method comprises the following steps: judging the memory exception state corresponding to the memory index data through abnormal trend analysis and associated feature analysis, so as to determine the memory overflow exception state of the database; time axis alignment is carried out on hierarchical log data, so as to obtain synchronous log data; through the preset snapshot trigger condition, the corresponding memory block in the data memory page is collected, and the memory block is dynamically inserted into the hole, so as to obtain synchronous snapshot data; based on the memory overflow exception state, the memory overflow mode is determined through memory overflow identification algorithm matching; according to the synchronous log data and the synchronous snapshot data, the memory overflow mode is positioned in multiple scenes, so as to determine the root cause node corresponding to the memory overflow mode. The application solves the technical problems that the database memory overflow root cause positioning mode is single and lacks the database automatic adaptation ability through the above method.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, device and medium for locating the root cause of database memory overflow. Background Technology

[0002] With the deepening of digital transformation, database application scenarios have expanded from traditional stand-alone deployments to cloud-native containerization and distributed clusters. The volume of business and data they handle is growing exponentially, making memory overflow issues more frequent and complex. Database memory overflows are mainly classified into three types: memory leaks, transient overflows, and resource leaks. Meanwhile, existing production environments have high requirements for database availability, prohibiting modifications to source code, service interruptions, or significant performance degradation during the troubleshooting process. Therefore, it is necessary to locate the root cause of memory overflows in a dynamic environment.

[0003] Existing technologies for locating memory overflows mainly fall into two categories. First, traditional database memory overflow location techniques are primarily designed for single-machine scenarios, lacking adaptability to cloud-native containerized or distributed environments, and offering only a single location method. Second, while memory overflow location techniques for general-purpose containers can automate the location process for containerized applications, they cannot be automatically customized to the specific business characteristics and kernel modules of the database, failing to directly adapt to the precise location requirements of database scenarios. Therefore, a low-intrusion, automatically adaptable method for locating the root cause of database memory overflows is urgently needed. Summary of the Invention

[0004] This application provides a method, device, and medium for locating the root cause of database memory overflow, which solves the technical problem that the existing methods for locating the root cause of database memory overflow are singular and lack automatic database adaptation capabilities.

[0005] In a first aspect, embodiments of this application provide a method for root cause localization of database memory overflow, characterized in that the method includes: periodically acquiring memory indicator data of the database, and determining the memory abnormal state corresponding to the memory indicator data through abnormal trend analysis and correlation feature analysis, so as to determine the memory overflow abnormal state of the database; acquiring hierarchical log data, and aligning the hierarchical log data with a time axis to obtain synchronized log data; collecting corresponding memory blocks in the data memory pages in components through preset snapshot trigger conditions, and dynamically instrumenting the memory blocks to obtain synchronized snapshot data; determining the memory overflow mode based on the memory overflow abnormal state through a memory overflow identification algorithm; and performing multi-scenario anomaly localization of the memory overflow mode according to the synchronized log data and synchronized snapshot data to determine the root cause node corresponding to the memory overflow mode.

[0006] In one implementation of this application, the abnormal memory state corresponding to memory indicator data is determined through abnormal trend analysis and correlation feature analysis to identify the abnormal memory overflow state of the database. Specifically, this includes: configuring an initial threshold for the memory indicator data based on the database parameters corresponding to the database, and dynamically adjusting the initial threshold to obtain the indicator threshold corresponding to the memory indicator data; wherein the database parameters include: database type, business load, and deployment scenario; determining the first abnormality judgment result by threshold judgment of the memory indicator data according to the indicator threshold; performing abnormal trend analysis on the memory indicator data to determine the second abnormality judgment result; wherein the types of abnormal trend analysis include: abnormal memory growth rate, abnormal connection growth rate, abnormal node resource contention rate, and abnormal distributed transaction; performing correlation feature analysis on the memory indicator data to determine the third abnormality judgment result; wherein the correlation indicators of the correlation feature analysis include: number of unreleased memory blocks, associated sessions, instantaneous overflow parameters, memory usage, and synchronization latency; and determining the abnormal state of the first, second, and third abnormality judgment results through a voting mechanism to identify the abnormal memory overflow state; wherein the abnormal memory overflow state includes: memory leak, instantaneous overflow, resource leak, containerization anomaly, and distributed anomaly.

[0007] In one implementation of this application, timeline alignment is performed on hierarchical log data to obtain synchronized log data. Specifically, this includes: standardizing the hierarchical log data to obtain full-source log data in JSON format; wherein the hierarchical levels of the hierarchical log data include: system layer, database layer, application layer, container layer, and distributed layer; and timeline alignment is performed on the full-source log data to obtain synchronized log data.

[0008] In one implementation of this application, the corresponding memory block in the component sampling data memory page is sampled based on a preset snapshot trigger condition. Specifically, this includes: acquiring memory status data, and performing a snapshot trigger judgment on the memory status data based on the snapshot trigger condition to determine the snapshot trigger type corresponding to the memory status; wherein, the snapshot trigger type includes: early warning trigger, fault pre-trigger, and timed trigger; classifying the memory pages according to the memory type of the database to obtain the memory page type; and obtaining the memory block through component sampling configuration based on the snapshot trigger type and the memory page type.

[0009] In one implementation of this application, memory blocks are dynamically instrumented to obtain synchronized snapshot data. Specifically, this includes: defining a unique marker corresponding to the memory block and dynamically instrumenting it in the memory allocation function of the database process to obtain a memory block with the inserted unique marker; wherein, the unique marker includes: a fixed identifier, an allocation source code, and a memory block ID; and performing snapshot synchronization on the memory block with the inserted unique marker to obtain synchronized snapshot data.

[0010] In one implementation of this application, based on the memory overflow abnormal state, a memory overflow mode is determined by matching with a memory overflow identification algorithm. Specifically, this includes: constructing an overflow feature set corresponding to the database type of the memory overflow abnormal state, and normalizing the overflow feature set to obtain an abnormal feature set; wherein the abnormal feature set includes: basic features and scenario-specific features; performing score matching on the feature conditions corresponding to the abnormal feature set to determine the matching score corresponding to each abnormal feature in the abnormal feature set; filtering candidate thresholds by rule traversal on the matching scores to obtain a first memory overflow mode; processing the first memory overflow mode using a full-scenario random forest algorithm to obtain a second memory overflow mode; and determining the memory overflow mode by outputting a probability value based on either the first or second memory overflow mode.

[0011] In one implementation of this application, multi-scenario anomaly localization of memory overflow patterns is performed based on synchronous log data and synchronous snapshot data to determine the root cause node corresponding to the memory overflow pattern. Specifically, this includes: calculating the memory contribution of each business scenario based on synchronous log data to determine the first root cause node; extracting abnormal business scenarios from the synchronous log data and filtering abnormal sessions for these scenarios to determine the second root cause node; obtaining the executed SQL and performing cost analysis on it to determine the third root cause node; determining the fourth root cause node through module anomaly analysis based on synchronous snapshot data; extracting the memory allocation call stack from the synchronous snapshot data and performing feature parsing on it to determine the fifth root cause node; wherein, feature parsing includes: symbol parsing and AST parsing; and determining the root cause node corresponding to the memory overflow pattern through container resource analysis and distributed tracing based on the first, second, third, fourth, and fifth root cause nodes.

[0012] In one implementation of this application, after performing multi-scenario anomaly localization on memory overflow patterns based on synchronization log data and synchronization snapshot data to determine the root cause node corresponding to the memory overflow pattern, the method further includes: organizing the root cause node into a corresponding root cause localization report through a preset Freemarker template engine, and storing the root cause localization report into a preset root cause library; updating the memory overflow identification algorithm based on the root cause library.

[0013] Secondly, embodiments of this application also provide a root cause localization device for database memory overflow, characterized in that the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement a root cause localization method for database memory overflow.

[0014] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for root cause localization of database memory overflow, which stores computer-executable instructions, characterized in that the computer-executable instructions, when executed, can implement a root cause localization method for database memory overflow.

[0015] This application provides a method, device, and medium for locating the root cause of database memory overflow. By analyzing multi-dimensional memory indicator data corresponding to abnormal memory states and automatically identifying memory overflow patterns under multiple business scenarios, it solves the technical problems of the existing technology that the root cause location method for database memory overflow is singular and lacks automatic database adaptation capability. It realizes dynamic database adaptation for locating the root cause of database memory overflow, and improves the accuracy of memory overflow pattern identification through self-optimization of the memory overflow identification algorithm. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A flowchart of a method for locating the root cause of database memory overflow provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram of the internal structure of a database memory overflow root cause localization device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] This application provides a method, device, and medium for locating the root cause of database memory overflow. By analyzing multi-dimensional memory indicator data corresponding to abnormal memory states and automatically identifying memory overflow patterns under multiple business scenarios, it solves the technical problems of the existing technology that the root cause location method for database memory overflow is singular and lacks automatic database adaptation capability. It realizes dynamic database adaptation for locating the root cause of database memory overflow, and improves the accuracy of memory overflow pattern identification through self-optimization of the memory overflow identification algorithm.

[0021] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a method for locating the root cause of database memory overflow, as provided in an embodiment of this application. Figure 1 As shown in the figure, the root cause localization method for database memory overflow provided in this application embodiment specifically includes the following steps:

[0023] Step 101: Periodically acquire the memory index data of the database, and determine the memory abnormal state corresponding to the memory index data through abnormal trend analysis and correlation feature analysis, so as to determine the memory overflow abnormal state of the database.

[0024] For example, this application collects multi-dimensional indicators from the database in real time across all scenarios, constructs a scenario-based dynamic threshold and a joint judgment logic based on multiple conditions, and uses this logic to detect the corresponding memory overflow abnormal state, providing a data foundation for algorithm updates and root cause localization.

[0025] Specifically, through anomaly trend analysis and correlation feature analysis, the abnormal memory states corresponding to memory indicator data are determined to identify the database's memory overflow anomaly state. This includes: configuring initial thresholds for memory indicator data based on the database parameters corresponding to the database, and dynamically adjusting the initial thresholds to obtain the corresponding indicator thresholds for the memory indicator data; where database parameters include: database type, business load, and deployment scenario; determining the first anomaly judgment result by thresholding the memory indicator data according to the indicator thresholds; performing anomaly trend analysis on the memory indicator data to determine the second anomaly judgment result; where the types of anomaly trend analysis include: abnormal memory growth rate, abnormal connection growth rate, abnormal node resource contention rate, and abnormal distributed transaction; performing correlation feature analysis on the memory indicator data to determine the third anomaly judgment result; where the correlation indicators in the correlation feature analysis include: number of unreleased memory blocks, associated sessions, instantaneous overflow parameters, memory usage, and synchronization latency; and using a voting mechanism to determine the anomaly state of the first, second, and third anomaly judgment results to identify the memory overflow anomaly state; where the memory overflow anomaly state includes: memory leak, instantaneous overflow, resource leak, containerization anomaly, and distributed anomaly.

[0026] In one embodiment, a distributed data collection system based on Prometheus and Exporter runs a dedicated Exporter for MySQL database types and containerized deployment scenarios, collecting basic metrics, database characteristic metrics, containerized metrics, and distributed metrics once per second.

[0027] A three-dimensional threshold configuration table is constructed based on database type, deployment scenario, and business load. Initial thresholds are configured for memory metric data. Based on the business load data of the past seven days, the baseline value of the metric is calculated using a sliding window algorithm. The threshold is equal to the baseline value multiplied by a dynamic coefficient. The initial threshold is dynamically adjusted to obtain the metric threshold corresponding to the memory metric data.

[0028] The memory metrics data are assessed based on threshold values ​​to determine the first anomaly. Anomaly trend analysis is then performed on the memory metrics data to determine the second anomaly. Anomaly trend analysis types include abnormal memory growth rate, abnormal connection growth rate, abnormal node resource contention rate, and abnormal distributed transactions. Finally, correlation feature analysis is performed on the memory metrics data to determine the third anomaly. Correlation features include the number of unreleased memory blocks, associated sessions, instantaneous overflow parameters, memory usage, and synchronization latency.

[0029] The first, second, and third anomaly judgment results are evaluated through a voting mechanism to determine the abnormal state of memory overflow. The abnormal state of memory overflow includes memory leak, instantaneous overflow, resource leak, containerization anomaly, and distributed anomaly.

[0030] Step 102: Obtain the hierarchical log data and align the hierarchical log data with the timeline to obtain synchronized log data.

[0031] For example, this application obtains synchronized log data by aligning hierarchical log data with a timeline, and provides complete log evidence for subsequent root cause localization by constructing a unified log analysis system.

[0032] Specifically, time-axis alignment of hierarchical log data is performed to obtain synchronized log data, including: standardizing the hierarchical log data to obtain full-source log data in JSON format; wherein the hierarchical levels of the hierarchical log data include: system layer, database layer, application layer, container layer, and distributed layer; and time-axis alignment of the full-source log data to obtain synchronized log data.

[0033] In one embodiment, the data collection system built on ELK and Filebeat deploys the lightweight Filebeat data collection client into the database container to achieve non-intrusive data collection. The collection scope includes system layer logs, database layer logs, application layer logs, container layer logs, and distributed layer logs.

[0034] Then, the hierarchical log data is standardized to obtain full-source log data in JSON format. Log records with format errors or missing key fields are removed by regular expressions. Core fields are extracted from the original logs, non-standard timestamps are converted to millisecond-level Unix timestamps, and database instance identifiers, log source identifiers, and deployment scenario tags are added to each log.

[0035] Then, the full-source log data is time-aligned to obtain synchronized log data. All collection terminals are synchronized with the NTP time server. The corresponding abnormal time window is locked with the time of the abnormality as the center, and the multi-source logs in the same database container are mapped to the same time axis to form key-value pairs of time and logs.

[0036] Finally, a composite index consisting of timestamps, container identifiers, and Structured Query Language identifiers is built in Elasticsearch to enable fast querying and association of multi-source logs by time range and association identifier.

[0037] Step 103: Based on the preset snapshot triggering conditions, the corresponding memory block in the component data memory page is collected, and the memory block is dynamically instrumented to obtain synchronous snapshot data.

[0038] For example, this application uses preset snapshot triggering conditions to collect corresponding memory blocks in the data memory pages and dynamically instrumentes the memory blocks to obtain synchronous snapshot data. This achieves the capture and storage of memory snapshots or heap dump files without affecting database performance, solving the problems of high snapshot capture overhead and lack of off-site storage in general container positioning technology.

[0039] Specifically, based on preset snapshot trigger conditions, the corresponding memory blocks in the component sampling data memory pages are sampled, including: acquiring memory status data, and performing snapshot trigger judgment on the memory status data based on the snapshot trigger conditions to determine the snapshot trigger type corresponding to the memory status; wherein, the snapshot trigger type includes: early warning trigger, fault pre-trigger, and timed trigger; classifying memory pages according to the database memory type to obtain the memory page type; and obtaining the memory block through component sampling configuration based on the snapshot trigger type and the memory page type.

[0040] Furthermore, memory blocks are dynamically instrumented to obtain synchronized snapshot data. Specifically, this includes: defining a unique identifier for each memory block and dynamically instrumenting it in the memory allocation function of the database process to obtain memory blocks with inserted unique identifiers; wherein, the unique identifier includes: a fixed identifier, an allocation source code, and a memory block ID; and performing snapshot synchronization on the memory blocks with inserted unique identifiers to obtain synchronized snapshot data.

[0041] In one embodiment, memory state data is acquired, and snapshot triggering is performed on the memory state data based on snapshot triggering conditions to determine the snapshot triggering type corresponding to the memory state. The snapshot triggering type includes warning triggering, fault pre-triggering, and timed triggering.

[0042] First, based on the database's memory type, memory pages are classified into different levels to obtain memory page types. The database process memory is then divided into multiple 4,000-byte memory pages, classified into heap memory, stack memory, shared memory, and kernel memory allocation for stack, large object data, and ordinary data.

[0043] Then, based on the snapshot trigger type and memory page type, memory blocks are obtained through component sampling configuration. Full sampling is used for core data memory pages, layered sampling is used for large object data memory pages, and random sampling is used for ordinary data memory pages. The sampling process replaces user-mode reading with kernel-mode memory reading to reduce context switching, and sets low priority to avoid preempting database business process resources.

[0044] It should be noted that when the trigger type is determined to be a pre-fault trigger, a standard-granularity memory snapshot is collected. The collection process uses a semaphore mechanism to ensure that only one snapshot is collected for the same instance at the same time.

[0045] Furthermore, a unique identifier is defined for each memory block, and dynamic instrumentation is performed in the memory allocation function of the database process to obtain the memory block with the unique identifier. The unique identifier includes a fixed identifier, an allocation source code, and a memory block ID, for example: DBOOM-MYSQL-COM-123456 (DBOOM is the fixed identifier, MYSQL-COM is the MySQL communication module, and 123456 is the unique memory block ID).

[0046] The memory blocks with unique markers are snapshotted to obtain synchronized snapshot data. A two-tier architecture of local temporary storage and remote persistent storage is adopted. The snapshots are synchronized to the remote storage for persistence through a synchronization mechanism, while the snapshot files are temporarily retained in the local storage to support offline analysis.

[0047] Finally, a 16-byte custom magic word is inserted into the memory allocation function of the database process using dynamic instrumentation technology based on the Pin tool. The magic word format includes a fixed identifier, allocation source code, and memory block identifier.

[0048] When requesting memory from the database, a magic word is automatically inserted into the header of the memory block and the allocation time, process ID, thread ID, and call stack information are recorded. During snapshot collection, the magic word in the header of the memory block is scanned to extract a large number of memory blocks whose allocation source codes point to the Istio Sidecar agent.

[0049] Step 104: Based on the memory overflow abnormal state, determine the memory overflow mode by matching with the memory overflow identification algorithm.

[0050] For example, the memory overflow identification algorithm in this application is achieved by combining the initial matching of the rule engine with the fine identification of the machine learning model. With this two-layer identification logic, a special feature set and matching algorithm are designed for each type of database memory overflow mode, realizing dynamic identification of memory overflow mode, which solves the problems of the general container location technology's coarse fault mode identification and lack of database specificity.

[0051] Specifically, based on the memory overflow anomaly state, the memory overflow pattern is determined by matching with a memory overflow identification algorithm, including: constructing an overflow feature set corresponding to the database type of the memory overflow anomaly state, and normalizing the overflow feature set to obtain an anomaly feature set; wherein, the anomaly feature set includes: basic features and scenario-specific features; performing score matching on the feature conditions corresponding to the anomaly feature set to determine the matching score corresponding to each anomaly feature in the anomaly feature set; filtering candidate thresholds by rule traversal on the matching scores to obtain a first memory overflow pattern; processing the first memory overflow pattern using a full-scenario random forest algorithm to obtain a second memory overflow pattern; and determining the memory overflow pattern by outputting probability values ​​based on the first or second memory overflow pattern.

[0052] In one embodiment, an overflow feature set corresponding to the database type of the memory overflow exception state is constructed, and the overflow feature set is normalized to obtain an exception feature set, which includes: basic features and contextual features.

[0053] First, the feature conditions corresponding to the abnormal feature set are matched by score to determine the matching score of each abnormal feature in the abnormal feature set. Then, the normalized feature values ​​are input into the lightweight rule engine built on Drools. The rule engine matches the feature conditions according to the preset rules and calculates the matching score.

[0054] Feature set settings can be represented through overflow patterns, basic features (general), and contextual features:

[0055] Overflow patterns include: memory leak, transient overflow, resource leak, containerization-specific, and distributed-specific.

[0056] Basic features (general) include: memory growth rate, number of unreleased memory blocks, session lifetime; memory burst growth rate, number of high-cost SQL queries, number of concurrent requests; connection growth rate, number of file handles, memory-connection correlation coefficient; cgroup memory utilization, container node resource contention rate; total memory across nodes, number of distributed transactions, node synchronization latency.

[0057] Contextual features include: cache hit rate change rate, number of cursors not closed; number of batch task executions, sorting or hash memory usage rate; number of idle connections in the connection pool, number of invalid connections across nodes; sidecar proxy memory usage rate, container image layer memory usage rate; percentage of uncommitted distributed transactions, and node memory usage variance.

[0058] Compared to the general feature set of general container localization technology, optimizing the database with context-specific features improves the targeting of pattern recognition.

[0059] Then, the candidate threshold is filtered by rule traversal of the matching score to obtain the first memory overflow mode. Overflow modes with matching scores exceeding the candidate threshold are taken as candidate modes. If there is a mode with matching scores exceeding the direct judgment threshold, the mode is directly output.

[0060] The first memory overflow pattern is processed by the random forest algorithm across the entire scenario to obtain the second memory overflow pattern. For candidate patterns that the rule engine does not directly determine, the normalized feature values ​​are input into the random forest model to calculate the probability value of each candidate pattern.

[0061] The rule engine framework is a lightweight rule engine built on Drools. It transforms the feature matching rules of overflow mode into DRL rule language. The rules can be set as overflow mode-feature condition-matching score, with the matching score ranging from 0 to 100.

[0062] Under memory leak conditions, if the memory growth rate is ≥0.5% / min, the number of unreleased memory blocks is ≥2 times the baseline value, the session lifespan is ≥1h, and the number of cursors not closed is ≥10, then the matching score is 85, and the overflow mode is memory leak.

[0063] The memory overflow mode is determined by outputting a probability value based on either the first or second memory overflow mode.

[0064] The initial matching logic inputs normalized feature values ​​into the rule engine, traverses all rule matches, and outputs candidate overflow patterns (up to 3) with a matching score ≥ 70. If there is a pattern with a matching score ≥ 90, it is directly identified as that pattern without needing to enter the model fine recognition stage.

[0065] Collect historical overflow data from the entire database and divide it into training, validation, and test sets in a 7:2:1 ratio. Each sample contains a feature set and an overflow pattern. Export the trained model as a PMML format label.

[0066] Finally, the normalized feature values ​​are input into the model, and the probability values ​​of each candidate mode are output. The mode with a probability value ≥ 0.85 is taken as the final result; if all probability values ​​are < 0.85, it is determined to be a new type of overflow mode, and the feature data is stored in the root cause library for subsequent model training.

[0067] By extracting the specific feature set corresponding to containerization anomalies, including cgroup memory usage, container node resource contention rate, and Sidecar proxy memory usage, the rule engine matches the rules for containerization anomalies. The matching score is 95 points, which exceeds the threshold of 90 points for direct judgment, and the current overflow mode is directly judged as a containerization anomaly.

[0068] Step 105: Based on the synchronization log data and synchronization snapshot data, perform multi-scenario anomaly localization for the memory overflow mode in order to determine the root cause node corresponding to the memory overflow mode.

[0069] For example, this application performs anomaly localization of memory overflow mode in multiple scenarios based on synchronous log data and synchronous snapshot data, and implements cross-container and cross-node localization methods corresponding to containerized and distributed scenarios, solving the technical problem that general container localization technology is not easy to fit the needs of database operation and maintenance.

[0070] Specifically, based on synchronization log data and synchronization snapshot data, multi-scenario anomaly localization is performed on memory overflow patterns to determine the root cause nodes corresponding to the memory overflow patterns. This includes: calculating the memory contribution of each business scenario based on synchronization log data to determine the first root cause node; extracting abnormal business scenarios from the synchronization log data and filtering abnormal sessions for these scenarios to determine the second root cause node; obtaining the executed SQL and performing cost analysis on it to determine the third root cause node; determining the fourth root cause node through module anomaly analysis based on synchronization snapshot data; extracting the memory allocation call stack from the synchronization snapshot data and performing feature analysis on it to determine the fifth root cause node; where feature analysis includes: symbol analysis and AST analysis; and determining the root cause node corresponding to the memory overflow pattern based on the first, second, third, fourth, and fifth root cause nodes through container resource analysis and distributed tracing.

[0071] Furthermore, after performing multi-scenario anomaly localization on memory overflow patterns based on synchronous log data and synchronous snapshot data to determine the root cause node corresponding to the memory overflow pattern, the method also includes: organizing the root cause node into a corresponding root cause localization report through a preset Freemarker template engine, and storing the root cause localization report into a preset root cause library; updating the memory overflow identification algorithm based on the root cause library.

[0072] In one embodiment, based on synchronized log data, the memory contribution of each business scenario is calculated to determine the first root cause node. Within the abnormal time window, all business requests are extracted by querying application layer logs. Correlation analysis is performed based on memory usage and execution time to calculate the memory contribution of each business scenario, i.e., the proportion of memory growth in that scenario to the total memory growth. Business scenarios with memory contribution exceeding a preset threshold are identified as the first root cause node.

[0073] Extract abnormal business scenarios from the synchronization log data, and perform abnormal session filtering on abnormal business scenarios to determine the second root cause node. Extract session identifiers from the abnormal business scenario logs, and query the database layer connection logs and session logs to analyze session creation time, transaction status, execution of structured query language, and memory usage. Filter out abnormal sessions that have not been released for a long time, continuously execute high-cost operations, or rank high in memory usage as the second root cause node.

[0074] The system retrieves and executes structured query languages, performs cost analysis on these languages, identifies the third root cause, extracts executed structured query languages ​​from abnormal sessions and obtains their identifiers, analyzes the execution cost of each structured query language by combining slow query logs and structured query language execution plans, and identifies structured query languages ​​whose execution cost exceeds a preset threshold as the third root cause.

[0075] Based on the synchronous snapshot data, the fourth root cause node is identified through module anomaly analysis. The memory blocks associated with the abnormal session are extracted by scanning the unique markers in the synchronous snapshot data, and the allocation source code is obtained. The allocation source is mapped to the specific module by mapping the allocation source code to the module mapping table. The module memory usage trend and the number of unreleased memory blocks are analyzed, and the module with memory allocation anomalies is identified as the fourth root cause node.

[0076] Extract the memory allocation call stack from the synchronization snapshot data and perform feature analysis on the memory allocation call stack to determine the fifth root cause node. Feature analysis includes symbol analysis and abstract syntax tree analysis. Extract the memory allocation call stack associated with the module from the synchronization snapshot data, load the symbol table and perform symbol analysis to obtain the function name, and use the abstract syntax tree to parse the source code to locate the specific memory allocation code line. The located code line is determined as the fifth root cause node.

[0077] Finally, through container resource analysis and distributed tracing, the root cause node corresponding to the memory overflow pattern was identified. The order payment interface, which contributed 45% of the memory usage, was identified as the core abnormal business scenario. Eighty abnormal sessions that had survived for more than two hours and had not been released for an extended period were selected. After ruling out issues with the structured query language itself, the memory management module proxied by the Istio Sidecar was located using magic word tags.

[0078] When the number of unreleased memory blocks in this module reached six times the baseline value, the specific line of code in the memory allocation function of the memory management module was located by calling the call stack. Container resource analysis revealed that the Sidecar agent had no independent memory limit and shared cgroup memory with the database. Ultimately, the root cause was determined to be that the memory management module of the Istio Sidecar agent had a problem with untimely memory release in the specific line of code of the memory allocation function. Furthermore, the Sidecar agent shared cgroup memory with the database without independent limit. Under high concurrency, the Sidecar agent's memory surge preempted the database's memory resources, triggering a containerization exception.

[0079] Furthermore, a dedicated report template based on the Freemarker template engine to match the memory overflow pattern is generated. The data mapping engine automatically fills the output root cause node information into the report template, generating a root cause localization report containing graded repair suggestions and optimization solutions. The graded repair suggestions include three levels: emergency repair, short-term optimization, and long-term planning. After the report is generated, it supports export in multiple formats and synchronizes the core information of the report to the operation and maintenance work order system to create operation and maintenance work orders.

[0080] Then, matching the dedicated report template for containerization exceptions, the emergency fix suggestion is to restart the Istio Sidecar agent to release the memory occupied by the exception, the short-term optimization suggestion is to configure an independent cgroup memory limit for the Sidecar agent to achieve memory isolation from the database, and the long-term planning suggestion is to add a timed memory release mechanism to the code line that fixes the Sidecar agent memory management.

[0081] Furthermore, the root cause database includes a case database, a rule database, and a feature database. After each overflow location is completed, the root cause location report, multi-source data, remediation measures, and remediation effects are automatically stored in the case database, and a unique case identifier is assigned to each case.

[0082] The memory overflow identification algorithm is updated based on the root cause library. New features are extracted from cases and added to the exclusive feature set of the corresponding overflow mode. New rules are extracted and stored in the root cause library pending review. After the review is approved, they are officially added to the feature library and rule library, and the model is retrained to add new features and new cases to the training set to optimize the identification model. The approved new rules are automatically synchronized to the rule engine to update the rule library.

[0083] New features with Sidecar proxy memory usage exceeding three times the baseline value are extracted and added to the dedicated feature set for containerization anomalies. New rules with a matching score of 90 points when cgroup memory usage reaches 90% and Sidecar proxy memory usage reaches three times the baseline value are extracted and stored in the pending review table. After approval, they are officially added to the feature library and rule library. During subsequent model self-training, the feature data of this case is added to the training set to optimize the model of the random forest algorithm.

[0084] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a root cause localization device for database memory overflow, the structure of which is as follows: Figure 2 As shown.

[0085] Figure 2 This is a schematic diagram of the internal structure of a root cause localization device for database memory overflow provided in an embodiment of this application. Figure 2 As shown, the device includes:

[0086] At least one processor 201;

[0087] And a memory 202 that is communicatively connected to at least one processor;

[0088] The memory 202 stores instructions that can be executed by at least one processor, which are executed by at least one processor 201 to enable at least one processor 201 to implement a root cause localization method for database memory overflow.

[0089] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for root cause localization of database memory overflow, storing computer-executable instructions, which, when executed, implement a root cause localization method for database memory overflow.

[0090] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0091] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0098] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0100] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for locating the root cause of database memory overflow, characterized in that, The method includes: The system periodically acquires memory indicator data from the database and uses anomaly trend analysis and correlation feature analysis to determine the memory anomaly status corresponding to the memory indicator data, thereby identifying the memory overflow anomaly status of the database. Acquire hierarchical log data and align the hierarchical log data with a timeline to obtain synchronized log data; wherein the hierarchical levels of the hierarchical log data include: system layer, database layer, application layer, container layer, and distributed layer; Based on preset snapshot trigger conditions, the corresponding memory block in the component acquisition data memory page is dynamically instrumented to obtain synchronous snapshot data. Based on the aforementioned memory overflow abnormal state, the memory overflow pattern is determined by matching with a memory overflow identification algorithm; Based on the synchronization log data and the synchronization snapshot data, anomaly localization is performed for the memory overflow mode in multiple scenarios to determine the root cause node corresponding to the memory overflow mode; By analyzing abnormal trends and correlation features, the abnormal memory status corresponding to the memory indicator data is determined, thereby identifying the memory overflow abnormal status of the database. Specifically, this includes: Based on the database parameters corresponding to the database, an initial threshold is configured for the memory metric data, and the initial threshold is dynamically adjusted to obtain the metric threshold corresponding to the memory metric data; wherein, the database parameters include: database type, business load, and deployment scenario; Based on the aforementioned threshold values, the memory index data is subjected to threshold judgment to determine the first anomaly judgment result; Anomaly trend analysis is performed on the memory indicator data to determine the second anomaly judgment result; wherein, the types of anomaly trend analysis include: abnormal memory growth rate, abnormal connection growth rate, abnormal node resource contention rate, and abnormal distributed transaction; The memory metrics data are subjected to correlation feature analysis to determine the third anomaly judgment result; wherein, the correlation metrics of the correlation feature analysis include: number of unreleased memory blocks, associated sessions, instantaneous overflow parameters, memory usage, and synchronization latency; An abnormal state is determined by voting on the first abnormal judgment result, the second abnormal judgment result, and the third abnormal judgment result to identify the memory overflow abnormal state; wherein, the memory overflow abnormal state includes: memory leak, instantaneous overflow, resource leak, containerization abnormality, and distributed abnormality.

2. The root cause localization method for database memory overflow according to claim 1, characterized in that, The hierarchical log data is time-axis aligned to obtain synchronized log data, specifically including: The hierarchical log data is standardized to obtain full-source log data in JSON format; The full-source log data is time-axis aligned to obtain the synchronized log data.

3. The root cause localization method for database memory overflow according to claim 1, characterized in that, Based on preset snapshot trigger conditions, the corresponding memory blocks in the component acquisition data memory pages specifically include: Memory status data is acquired, and based on the snapshot triggering conditions, a snapshot triggering judgment is performed on the memory status data to determine the snapshot triggering type corresponding to the memory status; wherein, the snapshot triggering type includes: early warning triggering, pre-fault triggering, and timed triggering; Based on the memory type of the database, the memory pages are classified into different levels to obtain the memory page types; Based on the snapshot trigger type and the memory page type, the memory block is obtained through component sampling configuration.

4. The root cause localization method for database memory overflow according to claim 1, characterized in that, Dynamic instrumentation is performed on the memory block to obtain synchronous snapshot data, specifically including: Define a unique identifier corresponding to the memory block, and dynamically instrument it in the memory allocation function of the database process to obtain the memory block with the unique identifier inserted; wherein, the unique identifier includes: a fixed identifier, an allocation source code, and a memory block ID; The memory block with the unique mark inserted is snapshot synchronized to obtain the synchronized snapshot data.

5. The root cause localization method for database memory overflow according to claim 1, characterized in that, Based on the aforementioned memory overflow abnormal state, the memory overflow pattern is determined through a memory overflow identification algorithm, specifically including: Construct an overflow feature set corresponding to the database type of the memory overflow abnormal state, and normalize the overflow feature set to obtain an abnormal feature set; wherein, the abnormal feature set includes: basic features and scenario-based features; The feature conditions corresponding to the set of abnormal features are matched by scores to determine the matching score corresponding to each abnormal feature in the set of abnormal features. The matching score is subjected to a rule-based traversal of candidate thresholds to obtain the first memory overflow pattern; The first memory overflow pattern is processed using the random forest algorithm across all scenarios to obtain the second memory overflow pattern. The memory overflow mode is determined by outputting a probability value based on either the first or the second memory overflow mode.

6. The root cause localization method for database memory overflow according to claim 1, characterized in that, Based on the synchronization log data and the synchronization snapshot data, multi-scenario anomaly localization is performed on the memory overflow pattern to determine the root cause node corresponding to the memory overflow pattern, specifically including: Based on the synchronous log data, the memory contribution of each business scenario is calculated to determine the first root cause node. Extract abnormal business scenarios from the synchronization log data, and perform abnormal session filtering on the abnormal business scenarios to determine the second root cause node; Obtain the executed SQL and perform cost analysis on the executed SQL to determine the third root cause node; Based on the synchronous snapshot data, the fourth root cause node is determined through module anomaly analysis; Extract the memory allocation call stack from the synchronization snapshot data, and perform feature parsing on the memory allocation call stack to determine the fifth root cause node; wherein, the feature parsing includes: symbol parsing and AST parsing; Based on the first root cause node, the second root cause node, the third root cause node, the fourth root cause node, and the fifth root cause node, the root cause node corresponding to the memory overflow mode is determined through container resource analysis and distributed tracing.

7. The root cause localization method for database memory overflow according to claim 1, characterized in that, After determining the root cause node corresponding to the memory overflow pattern by performing multi-scenario anomaly localization based on the synchronization log data and the synchronization snapshot data, the method further includes: Using a preset Freemarker template engine, the root cause nodes are organized into corresponding root cause location reports, and the root cause location reports are stored in a preset root cause library. The memory overflow identification algorithm is updated based on the root cause database.

8. A root cause localization device for database memory overflow, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to implement a root cause localization method for database memory overflow as described in any one of claims 1-7.

9. A non-volatile computer storage medium for root cause localization of database memory overflow, storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they can implement a root cause localization method for database memory overflow as described in any one of claims 1-7.

Images