Memory leak detection method and device, storage medium and electronic device
By combining 1D-CNN and global pooling layer, the real-time and accuracy of memory leak detection is achieved, and the problem of large resource consumption and time-delay of memory leak detection in the prior art can be solved. Position the root cause of leakage in the module.
Patent Information
- Application Number
- CN202310679611.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-08
- Publication Date
- 2024-12-17
AI Technical Summary
In the prior art, memory leak detection lacks real-time and efficientness. Especially in the embedded network element scenarios with resource-constrained resources, it is impossible to accurately detect memory leaks in processes and modules, and it is difficult to locate the leakage root.
The pre-trained deep learning model uses a pre-trained deep learning model to collect and store process memory usage index data, uses a time-series database to perform memory leakage detection, combines 1D-CNN and global pooling layer for classification, locates abnormal processes and modules, and realizes online memory leakage detection.
Improves the real-time and accuracy of memory leak detection, can detect memory leaks in processes and modules, simplifies the location of leak root causes, and is suitable for embedded environments with resource-constrained.
Smart Images
Figure CN119149271A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computers, and in particular to a memory leak detection method and device, a storage medium, and an electronic device. Background Art
[0002] In the related art, memory leak refers to the heap memory that has been dynamically allocated in the program, which is not released or cannot be released by the program for some reason, resulting in a waste of system memory, slowing down the program running speed or even causing serious consequences such as system crash. Memory leak defects are hidden and cumulative, and are more difficult to detect than other memory access errors. Because the cause of memory leaks is that the memory block is not released, it is an omission defect rather than a fault defect. In addition, memory leaks usually do not directly produce observable error symptoms, but gradually accumulate, reducing the overall performance of the system, and in extreme cases may cause the system to crash.
[0003] The related technology performs memory detection by analyzing historical memory status data. It does not involve scenarios where memory leak detection methods are used and is not real-time. The prediction method consumes a lot of resources and cannot be actually deployed in resource-constrained embedded network element scenarios. It only includes system memory leak detection and cannot detect finer-grained module memory leaks, making it difficult to locate the root cause of the leak.
[0004] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention
[0005] The present invention provides a memory leak detection method and device, a storage medium, and an electronic device to solve the above-mentioned technical problems existing in the related art.
[0006] According to one embodiment of the present invention, a method for detecting a memory leak is provided, comprising: locally collecting indicator data of memory usage by a process in a target network element according to a first preset period, wherein the indicator data is used to characterize the memory occupancy of the corresponding process; storing the indicator data in a timing database of the target network element, wherein the timing database is used to store indicator timing data of all processes in the target network element in multiple timing periods; querying the indicator timing data in the timing database according to a second preset period; and performing memory leak detection on the target network element using the indicator timing data using a pre-trained deep learning model.
[0007] Optionally, the pre-trained deep learning model is a timing classification model, and using the pre-trained deep learning model to perform memory leak detection on the target network element through the indicator timing data includes: for each first process in the target network element, using the corresponding indicator timing data to construct model input data; inputting the model input data into the timing classification model, and outputting the memory leakage result of the first process, wherein the memory leakage result is used to indicate whether there is a memory leak in the first process.
[0008] Optionally, the pre-trained deep learning model is a regression model, and using the pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data includes: for each second process in the target network element, generating a memory usage trend line based on the corresponding indicator time series data fitting; using the regression model to calculate the predicted time of the memory exhaustion threshold reached by the memory usage trend line; and detecting whether there is a memory leak in the second process in the target network element according to the predicted time.
[0009] Optionally, inputting the model input data into a timing classification model, and outputting the memory leakage result of the first process includes: inputting the model input data from an input layer into multiple serial one-dimensional convolutional neural networks 1D-CNN to obtain convolution data, wherein each 1D-CNN is configured with two convolution kernels of size 3; inputting the convolution data into a global pooling layer, classifying the convolution data from the global pooling layer to obtain the memory leakage result of the first process, wherein the timing classification model includes: the input layer, the multiple serial 1D-CNNs, and the global pooling layer.
[0010] Optionally, after using a pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data, the method also includes: if there is a memory leak in the target network element, determining a set of abnormal processes where the memory leak occurs; calling a waterline query interface to query the process waterline data of the target network element, wherein the process waterline data includes: process identifier PID, process waterline value; conditionally filtering the abnormal process set based on the process waterline value; selecting the target processes remaining after filtering, and locating the faulty module where the memory leak occurs in the target process, wherein the target process includes multiple modules, each module corresponding to a code segment of the target process.
[0011] Optionally, locating the faulty module where memory leakage occurs in the target process includes: collecting memory data of each module identifier MID in the target process, wherein the memory data includes: collection time, PID, MID, and memory usage of the module; for each MID, performing memory leak detection based on the corresponding memory data, and locating the faulty module where module memory leakage occurs in the target process based on the detection result; and writing the faulty module into the fault log of the target network element.
[0012] Optionally, collecting the memory data of each module identifier MID in the target process includes: controlling the AIOPS process of the target network element to send a request message for a data collection task to the HDC process of the target network element, wherein the request message is used to indicate the collection of the memory occupancy of the module in the target process; controlling the HDC process to respond to the request message, triggering the data collection interface, creating a data collection thread on the business job, and collecting the memory data of each MID in the target process through the data collection thread; controlling the HDC process to respond to the data collection status of the data collection thread, and storing the memory data of each MID in the target process into the time series database of the target process.
[0013] According to another embodiment of the present invention, a memory leak detection device is provided, including: a collection module, used to locally collect indicator data of memory usage by processes in a target network element according to a first preset period, wherein the indicator data is used to characterize the memory occupancy of the corresponding process; a storage module, used to store the indicator data in a timing database of the target network element, wherein the timing database is used to store indicator timing data of all processes in the target network element in multiple timing periods; a query module, used to query the indicator timing data in the timing database according to a second preset period; and a detection module, used to perform memory leak detection on the target network element through the indicator timing data using a pre-trained deep learning model.
[0014] Optionally, the pre-trained deep learning model is a time series classification model, and the detection module includes: a construction unit, used to construct model input data for each first process in the target network element using corresponding indicator time series data; a first detection unit, used to input the model input data into the time series classification model, and output a memory leakage result of the first process, wherein the memory leakage result is used to indicate whether there is a memory leak in the first process.
[0015] Optionally, the pre-trained deep learning model is a regression model, and the detection module includes: a fitting unit, used to generate a memory usage trend line for each second process in the target network element based on the corresponding indicator time series data fitting; a calculation unit, used to use the regression model to calculate the predicted time of the memory exhaustion threshold reached by the memory usage trend line; and a second detection unit, used to detect whether there is a memory leak in the second process in the target network element based on the predicted time.
[0016] Optionally, the first detection unit includes: a processing subunit, used to input the model input data from the input layer into multiple serial one-dimensional convolutional neural networks 1D-CNN to obtain convolution data, wherein each of the 1D-CNNs is configured with two convolution kernels of size 3; a detection subunit, used to input the convolution data into a global pooling layer, classify the convolution data from the global pooling layer, and obtain the memory leakage result of the first process, wherein the timing classification model includes: the input layer, the multiple serial 1D-CNNs, and the global pooling layer.
[0017] Optionally, the device also includes: a determination module, used to determine a set of abnormal processes where memory leaks occur after the detection module uses the pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data, if the target network element has a memory leak; a query module, used to call a waterline query interface to query the process waterline data of the target network element, wherein the process waterline data includes: process identifier PID, process waterline value; a filtering module, used to conditionally filter the abnormal process set based on the process waterline value; a positioning module, used to select the target processes remaining after filtering, and locate the faulty module where memory leak occurs in the target process, wherein the target process includes multiple modules, each module corresponds to a code segment of the target process.
[0018] Optionally, the positioning module includes: a collection unit, used to collect memory data of each module identifier MID in the target process, wherein the memory data includes: collection time, PID, MID, and module memory usage; a positioning unit, used to perform memory leak detection based on the corresponding memory data for each MID, and locate the faulty module where the module memory leak occurs in the target process based on the detection result; a writing unit, used to write the faulty module into the fault log of the target network element.
[0019] Optionally, the collection unit includes: a first control subunit, used to control the AIOPS process of the target network element to send a request message for a data collection task to the HDC process of the target network element, wherein the request message is used to indicate the collection of memory usage of the module within the target process; a second control subunit, used to control the HDC process to respond to the request message, trigger the data collection interface, create a data collection thread on the business job, and collect the memory data of each MID in the target process through the data collection thread; a third control subunit, used to control the HDC process to respond to the data collection status of the data collection thread, and store the memory data of each MID in the target process into the timing database of the target process.
[0020] According to yet another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned device embodiments when running.
[0021] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above device embodiments.
[0022] According to the embodiment of the present invention, the index data of memory usage of the process in the target network element is collected locally according to the first preset period, wherein the index data is used to characterize the memory occupancy of the corresponding process; the index data is stored in the time series database of the target network element, wherein the time series database is used to store the index time series data of all processes in the target network element in multiple time series periods, the index time series data in the time series database is queried according to the second preset period, and the pre-trained deep learning model is used to perform memory leakage detection on the target network element through the index time series data, the memory usage index of the process in the network element is collected periodically, the time series index data is stored in the network element time series database, the memory time series data in the database is queried regularly, and the deep learning time series classification model is used to determine whether the network element device has a possibility of leakage. When it is determined that the process has a suspected memory leak, the module memory usage index of the corresponding process can be further collected, and the specific module causing the process leak can be analyzed through model detection, and online memory leak detection is performed on the network element, thereby improving the real-time performance of the detection, solving the technical problems of large resource consumption and large delay of memory leak detection in related technologies, and in addition to detecting process memory leaks, specific module memory leaks can also be detected, so as to facilitate timely positioning of the root cause of the leak. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0024] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;
[0025] Figure 2 is a flow chart of a method for detecting a memory leak according to an embodiment of the present invention;
[0026] Figure 3 1D-CNN is a schematic diagram of the structure of an embodiment of the present invention;
[0027] Figure 4 is a software architecture diagram of a network element memory leak detection system in an embodiment of the present invention;
[0028] Figure 5 is an overall flow chart of memory leak detection in an embodiment of the present invention;
[0029] Figure 6 It is a structural block diagram of a memory leak detection device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0032] Example 1
[0033] The method embodiment provided in the first embodiment of the present application can be executed in a server, a computer, a network element device (such as a master control, a line card) or a similar computing device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0034] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a memory leak detection method in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0036] In this embodiment, a method for detecting a memory leak is provided. Figure 2is a flow chart of a method for detecting a memory leak according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0037] Step S202, locally collecting index data of memory usage of a process in a target network element according to a first preset period, wherein the index data is used to characterize the memory usage of the corresponding process;
[0038] The solution of this embodiment can be applied in a communication network environment. The communication network is composed of multiple interconnected network element devices. The network element devices are responsible for receiving and sending message data in the network. A single network element device includes a main control and a line card.
[0039] Step S204, storing the indicator data in a time series database of the target network element, wherein the time series database is used to store the indicator time series data of all processes in the target network element in multiple time series periods;
[0040] Step S206, querying the indicator time series data in the time series database according to the second preset period;
[0041] The delay difference in this embodiment is the time difference between the second IO processing delay and the first IO processing delay.
[0042] Step S208: Use a pre-trained deep learning model to perform memory leak detection on the target network element through indicator time series data.
[0043] Through the above steps, the index data of memory usage of the process in the target network element is collected locally according to the first preset period, wherein the index data is used to characterize the memory occupancy of the corresponding process; the index data is stored in the time series database of the target network element, wherein the time series database is used to store the index time series data of all processes in the target network element in multiple time series periods, the index time series data in the time series database is queried according to the second preset period, and the pre-trained deep learning model is used to perform memory leakage detection on the target network element through the index time series data, the memory usage index of the process in the network element is collected periodically, the time series index data is stored in the network element time series database, the memory time series data in the database is queried regularly, and the deep learning time series classification model is used to determine whether there is a possibility of leakage in the network element device. When it is determined that the process has a suspected memory leak, the module memory usage index of the corresponding process can be further collected, and the specific module that causes the process leak can be analyzed through model detection, and online memory leak detection is performed on the network element, thereby improving the real-time performance of the detection, solving the technical problems of large resource consumption and large delay in memory leak detection in related technologies, and in addition to detecting process memory leaks, specific module memory leaks can also be detected, so as to facilitate timely location of the root cause of the leak.
[0044] In one implementation of the present embodiment, the pre-trained deep learning model is a timing classification model, and the pre-trained deep learning model is used to perform memory leak detection on the target network element through indicator timing data, including: for each first process in the target network element, using the corresponding indicator timing data to construct model input data; inputting the model input data into the timing classification model, and outputting the memory leakage result of the first process, wherein the memory leakage result is used to indicate whether there is a memory leak in the first process.
[0045] In an example of this implementation, inputting model input data into a timing classification model, and outputting a memory leakage result of a first process includes: inputting the model input data from an input layer into a plurality of serial one-dimensional convolutional neural networks 1D-CNN to obtain convolution data, wherein each 1D-CNN is configured with two convolution kernels of size 3; inputting the convolution data into a global pooling layer, classifying the convolution data from the global pooling layer, and obtaining a memory leakage result of the first process, wherein the timing classification model includes: an input layer, a plurality of serial 1D-CNNs, and a global pooling layer.
[0046] When the scenario of network element memory leak detection is a resource-constrained embedded environment, a low-complexity algorithm can be selected while ensuring accuracy. This example uses the 1D-CNN time series classification algorithm. 1D-CNN is often used to solve time series-related tasks. Figure 3 1D-CNN is a schematic diagram of the structure of an embodiment of the present invention. Each convolution block consists of the following processes:
[0047]
[0048] s=BN(y),
[0049] h=ReLU(s).
[0050] Among them, the various parameters in the above formula are the conventional parameters of the convolution equation. In order to further reduce the complexity of the model, when deploying the 1D-CNN model on the network element, the size and number of CNN modules and kernels are reduced. Finally, 5 1D-CNN modules are used, each module uses 2 kernels of size 3, and the overall parameter quantity is controlled within 20,000.
[0051] In another implementation of the present embodiment, the pre-trained deep learning model is a regression model, and the use of the pre-trained deep learning model to perform memory leak detection on the target network element through indicator time series data includes: for each second process in the target network element, generating a memory usage trend line based on the corresponding indicator time series data fitting; using the regression model to calculate the predicted time of the memory exhaustion threshold reached by the memory usage trend line; and detecting whether there is a memory leak in the second process in the target network element according to the predicted time.
[0052] In this embodiment, the implementation methods of memory leak detection are divided into two categories: 1. Regression, 2. Classification. The regression method fits the trend line of memory usage based on the historical memory data, and then calculates the time to reach the memory exhaustion threshold based on the trend model, and judges whether the memory may have a leak by the arrival time. The classification method divides the historical memory data into normal memory data and leaking memory data, and judges whether the memory data has a leak by building a classification model. Typical memory leak timing patterns include linear growth or jagged linear growth.
[0053] In this embodiment, after using the pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data, it also includes:
[0054] S11, if the target network element has a memory leak, determine the abnormal process set where the memory leak occurs;
[0055] S12, calling the watermark query interface to query the process watermark data of the target network element, wherein the process watermark data includes: process identifier PID, process watermark value;
[0056] S13, conditionally filtering the abnormal process set based on the process waterline value;
[0057] S14, selecting the target process remaining after filtering, and locating the fault module where the memory leak occurs in the target process, wherein the target process includes multiple modules, and each module corresponds to a code segment of the target process.
[0058] In one example, locating a faulty module in a target process where a memory leak occurs includes: collecting memory data of each module identifier MID in the target process, wherein the memory data includes: collection time, PID, MID, and memory usage of the module; for each MID, performing memory leak detection based on the corresponding memory data, and locating the faulty module in the target process where a module memory leak occurs based on the detection result; and writing the faulty module into the fault log of the target network element.
[0059] Optionally, collecting memory data of each module identifier MID in the target process includes: controlling the AIOPS process of the target network element to send a request message for the data collection task to the HDC process of the target network element, wherein the request message is used to indicate the memory usage of the module in the target process; controlling the HDC process to respond to the request message, triggering the data collection interface, creating a data collection thread on the business job, and collecting the memory data of each MID in the target process through the data collection thread; controlling the HDC process to respond to the data collection status of the data collection thread, and storing the memory data of each MID in the target process into the time series database of the target process.
[0060] Figure 4This is a software architecture diagram of the network element memory leak detection system in an embodiment of the present invention. The entire system includes four processes: AI-DB, HDC, AIOPS and RSE. The HDC (short for data center, a data center management system that can implement resource management, virtualization, automation and other functions of the data center, and improve the efficiency and reliability of the data center) process is responsible for process memory data collection, module memory data collection and cycle scheduling. The AI-DB (artificial intelligence database) process is responsible for storing the collected memory time series data in the database. RSE (resource state engine) is a deep learning model inference engine, responsible for loading deep learning models and performing online inference. The AIOPS (artificial intelligence operation and maintenance) process is responsible for the overall memory leak detection data extraction, data processing, online inference, and result processing process connection.
[0061] Figure 5 It is the overall flow chart of memory leak detection in the embodiment of the present invention. AI-DB includes process db and mid db. board_kpi refers to the key performance indicators of the network element board, including the CPU utilization, memory utilization, temperature, voltage, fan speed and other indicators of the board. This embodiment is only related to memory occupancy and memory leakage. These indicators can be used to evaluate the performance and stability of the board and predict possible failures. In network operation and maintenance, monitoring and analyzing board_kpi is very important, which can help operation and maintenance personnel to discover and solve problems in a timely manner and ensure the normal operation of the network, including:
[0062] HDC periodically collects process memory data and stores the data in AI-DB;
[0063] The HDC periodically triggers the memory analysis task. After receiving the analysis request, the AIOPS module queries the AI-DB, processes the queried data, and then sends the inference data to the unified inference service.
[0064] The inference service uses the model to perform inference according to the model name and responds to the AIOPS with the inference result.
[0065] The AIOPS module calls the HDC waterline query interface to query the process memory waterline, and then filters the inference results by domain conditions;
[0066] If there are still suspected leaking processes after filtering, the TOP MID data collection of the corresponding process will be triggered;
[0067] After MID data collection and storage are completed, the HDC module notifies the AIOPS module to perform MID memory leak detection;
[0068] After receiving the analysis request, the AIOPS module queries the AI-DB, processes the queried data, and then sends the inference data to the unified inference service;
[0069] The inference service uses the model to perform inference according to the model name and responds to the AIOPS with the inference result. The AIOPS writes the result to the log.
[0070] By adopting the solution of this embodiment, a memory data collection and storage framework is constructed inside the network element device, and the memory data of all processes on the network element device is periodically collected, and the data is stored in the network element timing database; by offline analyzing the timing data in the database, a timing classification model is trained using a 1D-CNN network, and the model is deployed on the network element, and periodic reasoning is performed to analyze whether there is a suspected memory leak process; when the model detects a suspected memory leak process, the memory data of the corresponding module in the process is further collected and stored. When the data length meets the model reasoning requirements, the model is used to further detect the module suspected of memory leak.
[0071] This embodiment provides a memory leak detection solution, which involves a network element device in the communication field. By periodically collecting the memory usage indicators of the processes in the network element, the timing indicator data is stored in the network element timing database. The device periodically queries the memory timing data in the database, and uses the deep learning timing classification model to determine whether there is a possibility of leakage in the network element device. When it is determined that the process has a suspected memory leak, the module memory usage indicators of the corresponding process are further collected, and the specific module that caused the process leak is analyzed through model detection. This method is deployed in the network element device, and there is no need to deploy other detection equipment.
[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0073] Example 2
[0074] In this embodiment, a memory leak detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0075] Figure 6 is a structural block diagram of a memory leak detection device according to an embodiment of the present invention. Figure 6 As shown, the device comprises:
[0076] A collection module 60, configured to collect index data of memory usage of a process in a target network element locally according to a first preset period, wherein the index data is used to characterize the memory usage of the corresponding process;
[0077] A storage module 62 is used to store the indicator data into a time series database of the target network element, wherein the time series database is used to store the indicator time series data of all processes in the target network element in multiple time series periods;
[0078] A query module 64, configured to query the indicator time series data in the time series database according to a second preset period;
[0079] The detection module 66 is used to use a pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data.
[0080] Optionally, the pre-trained deep learning model is a time series classification model, and the detection module includes: a construction unit, used to construct model input data for each first process in the target network element using corresponding indicator time series data; a first detection unit, used to input the model input data into the time series classification model, and output a memory leakage result of the first process, wherein the memory leakage result is used to indicate whether there is a memory leak in the first process.
[0081] Optionally, the pre-trained deep learning model is a regression model, and the detection module includes: a fitting unit, used to generate a memory usage trend line for each second process in the target network element based on the corresponding indicator time series data fitting; a calculation unit, used to use the regression model to calculate the predicted time of the memory exhaustion threshold reached by the memory usage trend line; and a second detection unit, used to detect whether there is a memory leak in the second process in the target network element based on the predicted time.
[0082] Optionally, the first detection unit includes: a processing subunit, used to input the model input data from the input layer into multiple serial one-dimensional convolutional neural networks 1D-CNN to obtain convolution data, wherein each of the 1D-CNNs is configured with two convolution kernels of size 3; a detection subunit, used to input the convolution data into a global pooling layer, classify the convolution data from the global pooling layer, and obtain the memory leakage result of the first process, wherein the timing classification model includes: the input layer, the multiple serial 1D-CNNs, and the global pooling layer.
[0083] Optionally, the device also includes: a determination module, used to determine a set of abnormal processes where memory leaks occur after the detection module uses the pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data, if the target network element has a memory leak; a query module, used to call a waterline query interface to query the process waterline data of the target network element, wherein the process waterline data includes: process identifier PID, process waterline value; a filtering module, used to conditionally filter the abnormal process set based on the process waterline value; a positioning module, used to select the target processes remaining after filtering, and locate the faulty module where memory leak occurs in the target process, wherein the target process includes multiple modules, each module corresponds to a code segment of the target process.
[0084] Optionally, the positioning module includes: a collection unit, used to collect memory data of each module identifier MID in the target process, wherein the memory data includes: collection time, PID, MID, and module memory usage; a positioning unit, used to perform memory leak detection based on the corresponding memory data for each MID, and locate the faulty module where the module memory leak occurs in the target process based on the detection result; a writing unit, used to write the faulty module into the fault log of the target network element.
[0085] Optionally, the collection unit includes: a first control subunit, used to control the AIOPS process of the target network element to send a request message for a data collection task to the HDC process of the target network element, wherein the request message is used to indicate the collection of memory usage of the module within the target process; a second control subunit, used to control the HDC process to respond to the request message, trigger the data collection interface, create a data collection thread on the business job, and collect the memory data of each MID in the target process through the data collection thread; a third control subunit, used to control the HDC process to respond to the data collection status of the data collection thread, and store the memory data of each MID in the target process into the timing database of the target process.
[0086] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0087] Example 3
[0088] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0089] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:
[0090] S1, locally collecting index data of memory usage of a process in a target network element according to a first preset period, wherein the index data is used to characterize the memory usage of the corresponding process;
[0091] S2, storing the indicator data into a time series database of the target network element, wherein the time series database is used to store the indicator time series data of all processes in the target network element in multiple time series periods;
[0092] S3, querying the indicator time series data in the time series database according to a second preset period;
[0093] S4, using a pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data.
[0094] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.
[0095] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0096] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0097] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0098] S1, locally collecting index data of memory usage of a process in a target network element according to a first preset period, wherein the index data is used to characterize the memory usage of the corresponding process;
[0099] S2, storing the indicator data into a time series database of the target network element, wherein the time series database is used to store the indicator time series data of all processes in the target network element in multiple time series periods;
[0100] S3, querying the indicator time series data in the time series database according to a second preset period;
[0101] S4, using a pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data.
[0102] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0103] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0104] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0109] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for detecting memory leaks, characterized in that: include: Collecting locally the index data of memory usage of the process in the target network element according to the first preset period, wherein the index data is used to characterize the memory usage of the corresponding process; Storing the indicator data in a time series database of the target network element, wherein the time series database is used to store the indicator time series data of all processes in the target network element in multiple time series periods; Query the indicator time series data in the time series database according to a second preset period; A pre-trained deep learning model is used to perform memory leak detection on the target network element through the indicator time series data.
2. The method according to claim 1, characterized in that The pre-trained deep learning model is a time series classification model, and using the pre-trained deep learning model to perform memory leakage detection on the target network element through the indicator time series data includes: For each first process in the target network element, construct model input data using corresponding indicator time series data; The model input data is input into a time series classification model, and a memory leak result of the first process is output, wherein the memory leak result is used to indicate whether the first process has a memory leak.
3. The method according to claim 1, characterized in that The pre-trained deep learning model is a regression model, and using the pre-trained deep learning model to perform memory leakage detection on the target network element through the indicator time series data includes: For each second process in the target network element, generate a memory usage trend line based on the corresponding indicator time series data fitting; Calculating the predicted time of the memory exhaustion threshold reached by the memory usage trend line using the regression model; The second process in the target network element is detected according to the predicted time whether there is a memory leak.
4. The method according to claim 2, characterized in that: Inputting the model input data into a time series classification model, and outputting a memory leakage result of the first process includes: Inputting the model input data from the input layer into a plurality of serial one-dimensional convolutional neural networks 1D-CNN to obtain convolution data, wherein each of the 1D-CNN is configured with two convolution kernels of size 3; The convolution data is input into a global pooling layer, and the convolution data is classified from the global pooling layer to obtain a memory leakage result of the first process, wherein the timing classification model includes: the input layer, the multiple serial 1D-CNNs, and the global pooling layer.
5. The method according to claim 1, characterized in that After performing memory leakage detection on the target network element using the pre-trained deep learning model through the indicator time series data, the method further includes: If the target network element has a memory leak, determine a set of abnormal processes where the memory leak occurs; Calling a watermark query interface to query the process watermark data of the target network element, wherein the process watermark data includes: a process identifier PID and a process watermark value; Performing conditional filtering on the abnormal process set based on the process waterline value; The target process remaining after filtering is selected, and a fault module where a memory leak occurs in the target process is located, wherein the target process includes multiple modules, and each module corresponds to a code segment of the target process.
6. The method according to claim 5, characterized in that The fault module for locating the memory leak in the target process includes: Collecting memory data of each module identifier MID in the target process, wherein the memory data includes: collection time, PID, MID, and memory usage of the module; For each MID, memory leak detection is performed based on the corresponding memory data, and the faulty module where the module memory leak occurs in the target process is located based on the detection result; The fault module is written into the fault log of the target network element.
7. The method according to claim 6, characterized in that Collecting memory data of each module identifier MID in the target process includes: Controlling the artificial intelligence operation and maintenance AIOPS process of the target network element to send a request message for a data collection task to the data center HDC process of the target network element, wherein the request message is used to indicate the collection of the memory usage of the module in the target process; Control the HDC process to respond to the request message, trigger the data collection interface, create a data collection thread on the business job, and collect the memory data of each MID in the target process through the data collection thread; The HDC process is controlled to respond to the data collection state of the data collection thread, and the memory data of each MID in the target process is stored in the time series database of the target process.
8. A memory leak detection device, characterized in that: include: A collection module, used to collect index data of memory usage of a process in a target network element locally according to a first preset period, wherein the index data is used to characterize the memory usage of the corresponding process; A storage module, used to store the indicator data into a time series database of the target network element, wherein the time series database is used to store the indicator time series data of all processes in the target network element in multiple time series periods; A query module, used for querying the indicator time series data in the time series database according to a second preset period; A detection module is used to use a pre-trained deep learning model to perform memory leak detection on the target network element through the indicator time series data.
9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.
10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.