A method and device for solving data loss caused by abnormal collection script running
By automatically identifying and handling abnormal processes in Telegraf, the problem of data loss caused by script hangs was solved, enabling data recovery and improving system stability, while reducing operation and maintenance costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INSPUR SUZHOU INTELLIGENT TECH CO LTD
- Filing Date
- 2022-07-29
- Publication Date
- 2026-05-29
AI Technical Summary
In the Processor Plugins stage, issues such as custom script memory leaks, thread deadlocks, and network latency can cause scripts to hang, leading to soaring resource consumption and data loss. Existing solutions, such as restarting the operating system, are costly and cannot recover the data.
It automatically identifies abnormal processes in the monitoring and acquisition scripts, classifies and processes them, persists the data, recovers the data collected by the abnormal processes, and adds the recovered data to the recoverable queue based on the fault information of the abnormal processes. It combines the detection component, the abnormal acquisition script process processing component, and the recovery component to achieve data recovery.
It enables automatic identification and handling of abnormal processes, reduces Telegraf resource consumption, lowers maintenance costs, improves system stability and competitiveness, and reduces intrusion into existing business processes.
Smart Images
Figure CN115080301B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of script execution monitoring technology, specifically to a method and apparatus for solving data loss caused by abnormal script execution. Background Technology
[0002] Telegraf is a Go-based tool for collecting, processing, and aggregating server monitoring data. Its design goal is low memory usage, utilizing plugins to collect data from various services and third-party components. Telegraf is driven by four independent plugins: ① Input Plugins, ② Processor Plugins (for collecting data from the system, services, and third-party components), ③ Aggregator Plugins (for transforming, processing, and filtering data), and ④ Output Plugins (for writing data to the database). Particularly problematic is the Processor Plugins, where issues such as custom script memory leaks, thread deadlocks, network latency, and physical device failure can cause scripts to hang, leading to a surge in Telegraf resource consumption and potential data loss.
[0003] Currently, when script execution errors occur, the system administrator resolves the issue by restarting the operating system, which is costly and cannot solve the problem of data loss. Summary of the Invention
[0004] Especially in the Processor Plugins stage, issues such as custom script memory leaks, thread deadlocks, network latency, and physical device failure can cause the corresponding scripts to hang, leading to a surge in Telegraf resource consumption and data loss. This invention provides a method and apparatus for resolving data loss caused by abnormal execution of data collection scripts.
[0005] On the one hand, the technical solution of the present invention provides a method for solving the problem of data loss caused by the unresponsiveness of the data acquisition script, including the following steps:
[0006] Automatically identify abnormal processes in the monitoring and data collection scripts, and classify the abnormal processes;
[0007] Handle abnormal processes and persist the dependency data used for abnormal process data recovery and abnormal process data item recovery;
[0008] The data collected by the abnormal process is recovered and persisted to the database;
[0009] Obtain fault information of abnormal processes. If it is determined that the abnormal process is recoverable based on the obtained fault information, add the collection items of the abnormal process to the recoverable collection queue.
[0010] Furthermore, the steps for automatically identifying and classifying abnormal processes in the monitoring and acquisition script include:
[0011] Intercept all data collection items, record the name and process ID of the collection script, and write them to the collection queue;
[0012] The process status is judged based on the fault information defined in the collection queue, abnormal processes are identified and classified and written to the corresponding abnormal queue.
[0013] Furthermore, the steps of judging the process status based on the fault information defined in the collection queue, identifying abnormal processes, and classifying and writing abnormal processes into the corresponding abnormal queues include:
[0014] Check if a corresponding process exists in the operating system by its process ID;
[0015] If it does not exist, the process is considered to have terminated normally; the counter is decremented by 1 according to the name of the collection item.
[0016] If it exists, compare the start time of the process with the current time, and determine whether the time difference is greater than the set threshold.
[0017] If so, determine that the process is abnormal, classify the abnormal processes and write them into the corresponding abnormal queue, and increment the counter by 1 according to the name of the collected item;
[0018] If not, no action will be taken.
[0019] Furthermore, the exception queue includes a script-related problem queue and a non-script-related problem queue; the steps for classifying and writing exception processes into the corresponding exception queues include:
[0020] The script used to determine whether an abnormal process is a custom script;
[0021] If so, classify it directly into the non-script-specific problem type and add it to the non-script problem queue;
[0022] If not, parse the stack trace of the abnormal process;
[0023] Determine whether the parsed stack information includes the specified key information;
[0024] If it exists, it is identified as a problem type of the script itself and written to the script problem queue;
[0025] If it does not exist, it is identified as a non-script-specific problem type and written to the non-script problem queue.
[0026] Furthermore, the steps for handling abnormal processes and persisting the dependent data used for abnormal process data recovery and abnormal process collection item recovery include:
[0027] Check if the abnormal process exists in the operating system by its process ID;
[0028] If it does not exist, modify the process information in the exception queue; decrement the counter by 1 according to the name of the collection item.
[0029] If it exists, persist the stack information of the abnormal process;
[0030] Retrieve data from the stack trace corresponding to the abnormal process and persist the relevant data to a dirty data file in memory.
[0031] Delete the abnormal process.
[0032] Furthermore, the steps for persisting the stack information of the abnormal process include:
[0033] Analyze the stack trace of the abnormal process;
[0034] Capture fault information defined in the data collection queue;
[0035] The captured fault information is persisted to the exception stack information file.
[0036] Furthermore, the steps to destroy the abnormal process include:
[0037] Use the kill-9 command to destroy the abnormal process, and then check whether the destruction was successful.
[0038] If so, modify the process information in the exception queue;
[0039] If not, forcibly change the parent process of the abnormal process to 1. If the modification is successful, execute the following steps: use kill -9 pid to destroy the abnormal process and determine whether the destruction was successful; otherwise, use rm -rf / proc / pid to forcibly delete the file information corresponding to the abnormal process and modify the process information in the abnormal queue.
[0040] Determine whether the counter corresponding to the name of the collected item exceeds the second threshold;
[0041] If so, remove the corresponding collection item and restart Telegraf via a warm restart; otherwise, execute the next abnormal process in a loop.
[0042] Furthermore, the steps for recovering and persisting data collected by the abnormal process to the database include:
[0043] Information is read sequentially from the dirty data file in memory, and then the data is retrieved in reverse using the Telegraf data processing decoding tool after the reading is complete.
[0044] The acquired data is formatted and written to the InfluxDB database using the InfluxDB read / write component.
[0045] Furthermore, the steps of acquiring fault information of abnormal processes and, based on the acquired fault information, determining whether an abnormal process is recoverable, and adding the collected items of the abnormal process to the recoverable collection queue include:
[0046] Read the exception stack trace file to obtain fault information;
[0047] Determine whether the fault in the abnormal process has been recovered based on the fault information;
[0048] If so, add the relevant data collection items to the recoverable data collection queue;
[0049] Edit the collection items in the recoverable collection queue to restore the custom collection script, and restart Telegraf via a hot boot;
[0050] Otherwise, no action will be taken.
[0051] Secondly, the technical solution of the present invention also provides a device for solving the problem of data loss caused by the non-response of the acquisition script, including a detection component, an abnormal acquisition script process processing component, and a recovery component;
[0052] The detection component is used to automatically identify abnormal processes in the monitoring and acquisition scripts and to classify these abnormal processes.
[0053] An abnormal data collection script process handling component is used to process abnormal processes and persist the dependency data used for abnormal process data collection recovery and abnormal process collection item recovery.
[0054] The recovery component is used to recover and persist the data collected from the abnormal process to the database; it obtains the fault information of the abnormal process, and if it determines that the abnormal process is recoverable based on the obtained fault information, it adds the collected items of the abnormal process to the recoverable collection queue.
[0055] The detection component mainly completes the listening and identification of abnormal process information and automatically classifies abnormal information. Specifically, it includes: collection queue, abnormal queue, and abnormal process listener.
[0056] The abnormal process is completely destroyed and its resources are released in the abnormal collection script process handling component. It also persists the dependent data used for data recovery and collection process recovery. The processing includes abnormal process destruction, abnormal process stack information persistence, and abnormal process memory persistence.
[0057] The recovery component performs InfluxDB persistence of dirty data from abnormal processes and automatic recovery of abnormal data collection items. It mainly includes: dirty data information decoding, a recoverable process listener, and an InfluxDB read / write component.
[0058] As can be seen from the above technical solutions, this invention has the following advantages: it automatically identifies hung data collection scripts, can handle abnormal data collection items through various abnormal process destruction strategies, and can completely solve the problem of data loss caused by abnormal processes. This invention is integrated into the product as a plugin, without intruding on existing business logic and minimizing impact on Telegraf itself. Based on this invention, not only is the stability of Telegraf improved and its resource consumption reduced, but maintenance costs and technical risks are also lowered, enhancing the competitiveness of the AI platform among similar products.
[0059] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects.
[0060] Therefore, it is evident that the present invention has outstanding substantive features and significant progress compared with the prior art, and the beneficial effects of its implementation are also obvious. Attached Figure Description
[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0062] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.
[0063] Figure 2 This is a schematic diagram of the operation process of a device according to an embodiment of the present invention.
[0064] Figure 3 This is a schematic diagram of the detection component's workflow in one embodiment of the present invention.
[0065] Figure 4 This is a schematic diagram of the workflow of the exception collection script process processing component in one embodiment of the present invention. Detailed Implementation
[0066] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.
[0067] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for solving data loss caused by a data acquisition script not responding, including the following steps:
[0068] Step 1: Automatically identify abnormal processes in the monitoring and acquisition scripts, and classify the abnormal processes;
[0069] Step 2: Handle the abnormal process and persist the dependency data used for abnormal process data recovery and abnormal process collection item recovery;
[0070] Step 3: Recover the data collected by the abnormal process and persist it to the database;
[0071] Step 4: Obtain fault information of abnormal processes. If the abnormal process is recoverable based on the obtained fault information, add the collection items of the abnormal process to the recoverable collection queue.
[0072] In some embodiments, the step of automatically identifying abnormal processes of the monitoring and acquisition script and classifying abnormal processes includes:
[0073] Step 11: Intercept all data collection items, record the name and process ID of the collection script, and write them to the collection queue;
[0074] Step 12: Determine the process status based on the fault information defined in the collection queue, identify abnormal processes, classify abnormal processes, and write them into the corresponding abnormal queue.
[0075] In some embodiments, the steps of judging the process status based on the fault information defined in the acquisition queue, identifying abnormal processes, and classifying and writing abnormal processes into the corresponding abnormal queue include:
[0076] Step 121: Check if a corresponding process exists in the operating system by process ID;
[0077] If it does not exist, proceed to step 122; if it exists, proceed to step 123.
[0078] Step 122: Assume the process has terminated normally; decrement the counter by 1 according to the name of the data collection item;
[0079] Step 123: Compare the time when the process started execution with the current time, and determine whether the time difference is greater than the set threshold;
[0080] If yes, proceed to step 124; otherwise, do nothing.
[0081] Step 124: Determine if a process is abnormal, classify the abnormal processes and write them into the corresponding abnormal queue, and increment the counter by 1 according to the name of the collected item.
[0082] In some embodiments, the steps of judging the process status based on the fault information defined in the acquisition queue, identifying abnormal processes, and classifying and writing abnormal processes into the corresponding abnormal queue include:
[0083] S121: Traverse the collection queue to obtain the process ID of the collection item;
[0084] S122: Check if a corresponding process exists in the operating system by process ID;
[0085] If it does not exist, proceed to step S123; if it exists, proceed to step S125.
[0086] S123: The process is considered to have terminated normally; the counter is modified by -1 according to the name of the data collection item; proceed to step S124;
[0087] S124: Determine whether the collection queue has been traversed completely; if not, execute step S121 to traverse the collection queue and obtain the next process number; otherwise, execute step S127.
[0088] S125: Compare the time when the process started execution with the current time, and determine whether the time difference is greater than a set threshold.
[0089] If yes, proceed to step S126; otherwise, continue to step S122.
[0090] S126: Determine if a process is abnormal, and classify the abnormal processes and write them into the corresponding exception queue;
[0091] S127: End.
[0092] In some embodiments, the exception queue includes a script problem queue and a non-script problem queue; the step of classifying and writing exception processes into the corresponding exception queue in S126 includes:
[0093] S1261: Determine whether the script for an abnormal process is a custom script;
[0094] If yes, proceed to step S1262; otherwise, proceed to step S1263.
[0095] Step S1262: Directly classify it as a non-script-based problem type and write it into the non-script problem queue;
[0096] S1263: Parse the stack information of the abnormal process;
[0097] S1264: Determine whether the parsed stack information includes the set key information;
[0098] If it exists, execute S1265; if it does not exist, execute S1266.
[0099] S1265: This is identified as a script-specific problem type and written to the script problem queue;
[0100] S1266: This is identified as a non-script-specific problem type and written to the non-script problem queue.
[0101] In some embodiments, step 2, which involves processing the abnormal process and persisting the dependent data used for abnormal process data recovery and abnormal process collection item recovery, includes:
[0102] Step 21: Check if the abnormal process exists in the operating system by its process ID;
[0103] If it does not exist, proceed to step 22; if it exists, proceed to step 23.
[0104] Step 22: Modify the process information in the exception queue; decrement the counter by 1 according to the name of the collection item; execute step 21 to query the next process ID;
[0105] Step 23: Persist the stack information of the abnormal process; it should be noted that this step specifically includes: parsing the stack information of the abnormal process; capturing the fault information defined in the collection queue; and persisting the captured fault information to the abnormal stack information file.
[0106] Step 24: Obtain the data collected from the corresponding collection item of the abnormal process from the stack information, and persist the relevant data to the dirty data file in memory;
[0107] Step 25: Delete the abnormal process. This step specifically includes:
[0108] Use the kill-9 command to destroy the abnormal process, and then check whether the destruction was successful.
[0109] If so, modify the process information in the exception queue;
[0110] If not, forcibly change the parent process of the abnormal process to 1. If the modification is successful, execute the following steps: use kill -9 pid to destroy the abnormal process and determine whether the destruction was successful; otherwise, use rm -rf / proc / pid to forcibly delete the file information corresponding to the abnormal process and modify the process information in the abnormal queue.
[0111] Determine whether the counter corresponding to the name of the collected item exceeds the second threshold;
[0112] If so, remove the corresponding collection item and restart Telegraf via a warm restart; otherwise, execute the next abnormal process in a loop.
[0113] In some embodiments, step 3, which involves restoring and persisting the data collected by the abnormal process to the database, includes:
[0114] Step 31: Read information sequentially from the dirty data file in memory. After reading is complete, use the Telegraf data processing decoding tool to retrieve the data in reverse.
[0115] Step 32: Format the acquired data and write it to the InfluxDB database using the InfluxDB read / write component.
[0116] In some embodiments, step 4, which involves obtaining fault information of the abnormal process and, based on the obtained fault information, determining that the abnormal process is recoverable, adding the collected items of the abnormal process to the recoverable collection queue, includes:
[0117] Step 41: Read the exception stack trace file to obtain fault information;
[0118] Step 42: Determine whether the fault of the abnormal process has been recovered based on the fault information;
[0119] If yes, proceed to step 43; otherwise, do nothing.
[0120] Step 43: Add the relevant data collection items to the recoverable data collection queue;
[0121] Step 44: Edit the collection item in the recoverable collection queue to restore the custom collection script, and restart Telegraf via a warm restart.
[0122] like Figure 2 As shown, this embodiment of the invention also provides a device for solving the problem of data loss caused by the unresponsiveness of the acquisition script, including a detection component, an abnormal acquisition script process handling component, and a recovery component;
[0123] The detection component is used to automatically identify abnormal processes in the monitoring and acquisition scripts and to classify these abnormal processes.
[0124] An abnormal data collection script process handling component is used to process abnormal processes and persist the dependency data used for abnormal process data collection recovery and abnormal process collection item recovery.
[0125] The recovery component is used to recover and persist the data collected from the abnormal process to the database; it obtains the fault information of the abnormal process, and if it determines that the abnormal process is recoverable based on the obtained fault information, it adds the collected items of the abnormal process to the recoverable collection queue.
[0126] The detection component mainly completes the listening and identification of abnormal process information and automatically classifies abnormal information. Specifically, it includes: collection queue, abnormal queue, and abnormal process listener.
[0127] The abnormal process is completely destroyed and its resources are released in the abnormal collection script process handling component. It also persists the dependent data used for data recovery and collection process recovery. The processing includes abnormal process destruction, abnormal process stack information persistence, and abnormal process memory persistence.
[0128] The recovery component performs InfluxDB persistence of dirty data from abnormal processes and automatic recovery of abnormal data collection items. It mainly includes: dirty data information decoding, a recoverable process listener, and an InfluxDB read / write component.
[0129] The specific implementation process is as follows:
[0130] Detection component deployment: Initialize the collection process queue, and complete the writing of collection items and maintenance of process information through the InputPluginInterceptor; build an exception listener and complete the classification definition of the fault information that can be listened to; initialize the classified exception queue and maintain its relationship with the counter.
[0131] Anomaly collection script deployment: Data persistence: Persist the stack information corresponding to the process to the anomaly stack information file, and persist the heap memory information to the dirty data file in memory; Anomaly process destruction: Destroy the anomaly process in sequence by killing -p pid, set ppid, and rm -rf / proc / pid, and completely release the resources; Anomaly collection item removal: Remove the anomaly collection item from the Telegraf collection configuration and complete the hot deployment of Telegraf.
[0132] Data recovery deployment: Data recovery: By parsing the dirty data file in memory, the data is written to InfluxDB; Collection recovery: By parsing the exception stack information file and making judgments based on the commands recorded in the fault, the data is added to the Telegraf collection configuration after the judgment is passed, and then Telegraf hot deployment is completed.
[0133] It's important to note that the data collection queue in the detection component uses a first-in, first-out (FIFO) queue structure. It only needs to record the name and process ID of the collection script, in the format name>process ID, such as (CPU>1234, Memory>2567 or bash / var / lib / *** / test.sh>3567). An interceptor (InputPluginInterceptor) is added to the Telegraf Processor Plugins to intercept all data collection items and write them to the collection queue. The interceptor defines common fault types (network problems: handled using ping; disk I / O problems: information captured using the iostat command; disk space problems: information captured using the df command), and writes the fault type to the process's stack trace. Note: Because Telegraf executes the collection scripts periodically, there may be multiple instances of the same data collection item in the collection queue.
[0134] Abnormal process listener: This function determines the process status in the collection queue and classifies abnormal processes, such as... Figure 3 As shown, the specific process is as follows:
[0135] First, check if the process exists in the operating system by process ID (this can be done via operating system commands or by querying the process file under / proc). If it does not exist, it is considered to have terminated normally, and the counter is decremented by 1 according to the name of the collection item. Otherwise, the start time of the process is compared with the current time. If it does not exceed 50 seconds, no action is taken. Otherwise, the process is considered abnormal, and the counter is incremented by 1 according to the name of the collection item.
[0136] Determine if it is a custom script. If it is, classify it as a non-script-related issue and write it to the corresponding non-script issue queue. Otherwise, parse the process stack information (common commands are pstack pid or gstackpid). If the stack information includes the set critical information: OOM, Thread Lock, lock_wait, wait(), it is identified as a script-related issue (the exception information in the stack can be dynamically modified) and written to the corresponding script issue queue. Otherwise, it is identified as a non-script-related issue and written to the corresponding non-script issue queue.
[0137] An exception queue is a data structure consisting of a hash and a linked list. The key is used to record the name of the data collection item, such as Memory or bash / var / lib / *** / test.sh. The linked list records the exception process IDs corresponding to the data collection items, and there can be multiple IDs.
[0138] An exception collection script process handling component thoroughly destroys the exception process, releases its resources, and persists dependent data used for data recovery and collection process recovery. This includes: exception process destruction, persistence of exception process stack information, and persistence of exception process memory. Figure 4 As shown, the specific approach is as follows:
[0139] Abnormal process liveness detection: Check again whether the process information exists. If it does not exist, modify the process information in the abnormal queue and change its counter by 1. If it exists, continue to execute the following steps.
[0140] Persistence of abnormal process stack information: Parse the stack information of a process, capture the fault information defined in the "Detection Component - Collection Queue", and persist the fault information to the abnormal stack information file;
[0141] Abnormal process memory persistence: Parse the heap information of a process, obtain the data collected from the corresponding collection items from the heap information, and persist the relevant data to the dirty data file in memory;
[0142] Abnormal process destruction: After completing the above-mentioned file persistence, first try to destroy the process using kill -9 pid. If the destruction is successful, modify the process information in the exception queue (the counter does not need to be operated). Otherwise, forcibly change the parent process of the process to 1 (init process). If the modification is successful, execute the kill-related steps. If no modification is performed, use rm -rf / proc / pid to forcibly delete the file information corresponding to the process and modify the process information in the exception queue (the counter does not need to be operated).
[0143] Collection item removal processing: After the above steps are completed, it is determined whether the counter corresponding to the collection item exceeds the second threshold, which is set to 10 here. If it exceeds the threshold, the corresponding collection item is removed and Telegraf is restarted via a warm restart. Otherwise, the next abnormal process is executed in a loop.
[0144] Recovery Component: This component performs InfluxDB persistence of dirty data from abnormal processes and automatic recovery of abnormal data collection items. It includes: decoding of dirty data information, a recoverable process listener, and InfluxDB read / write components. The specific approach is as follows:
[0145] Data recovery: Primarily responsible for recovering data collected by the abnormal process. First, information is read sequentially from the dirty data file in memory. After reading, the Telegraf data processing decoding tool is used to reverse the data and format it into the following format:
[0146] collect_host_docker_mount_usagefilesystem=" / dev / mapper / centos-root",blocks=985376672i,used=920451544i,available=64925128i,usage="94%",mount_on=" / "
[0147] Next, the data is written to the InfluxDB database using the InfluxDB read / write component.
[0148] Data Acquisition Recovery: This mainly determines under what circumstances abnormal data acquisition items can resume normal acquisition. First, it reads the exception stack information file to obtain fault information. Based on the fault information and its related commands, it determines whether the fault has been recovered. If recovered, the relevant data acquisition items are added to the recoverable data acquisition queue; otherwise, no action is taken. The recoverable data acquisition queue supports manual editing to restore custom data acquisition scripts and restarts Telegraf via a hot boot.
[0149] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should also be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.
Claims
1. A method for resolving data loss caused by unresponsive data collection scripts, characterized in that, Includes the following steps: Automatically identify abnormal processes in the monitoring and data collection scripts, and classify the abnormal processes; Handle abnormal processes and persist the dependency data used for abnormal process data recovery and abnormal process data item recovery; The data collected by the abnormal process is recovered and persisted to the database; Obtain fault information of abnormal processes. If it is determined that the abnormal process is recoverable based on the obtained fault information, add the collection items of the abnormal process to the recoverable collection queue. The steps for automatically identifying and classifying abnormal processes in the monitoring and data collection script include: Intercept all data collection items, record the name and process ID of the collection script, and write them to the collection queue; The process status is judged based on the fault information defined in the collection queue, abnormal processes are identified and classified and written into the corresponding abnormal queue. The exception queue includes a script-related problem queue and a non-script-related problem queue; the steps for classifying and writing exception processes into the corresponding exception queues include: The script used to determine whether an abnormal process is a custom script; If so, classify it directly into the non-script-specific problem type and add it to the non-script problem queue; If not, parse the stack trace of the abnormal process; Determine whether the parsed stack information includes the specified key information; If it exists, it is identified as a problem type of the script itself and written to the script problem queue; If it does not exist, it is identified as a non-script-specific problem type and written to the non-script problem queue; The steps for handling abnormal processes and persisting the dependent data used for abnormal process data recovery and abnormal process data item recovery include: Check if the abnormal process exists in the operating system by its process ID; If it does not exist, modify the process information in the exception queue; decrement the counter by 1 according to the name of the collection item. If it exists, persist the stack information of the abnormal process; Retrieve data from the stack trace corresponding to the abnormal process and persist the relevant data to a dirty data file in memory. Delete the abnormal process.
2. The method for resolving data loss due to unresponsive acquisition scripts according to claim 1, characterized in that, The steps of judging the process status based on the fault information defined in the collection queue, identifying abnormal processes, and classifying abnormal processes and writing them into the corresponding abnormal queue include: Check if a corresponding process exists in the operating system by its process ID; If it does not exist, the process is considered to have terminated normally; the counter is decremented by 1 according to the name of the collection item. If it exists, compare the start time of the process with the current time, and determine whether the time difference is greater than the set threshold. If so, determine that the process is abnormal, classify the abnormal processes and write them into the corresponding abnormal queue, and increment the counter by 1 according to the name of the collected item; If not, no action will be taken.
3. The method for resolving data loss due to unresponsive acquisition scripts according to claim 1, characterized in that, The steps for persisting the stack information of an abnormal process include: Analyze the stack trace of the abnormal process; Capture fault information defined in the data collection queue; The captured fault information is persisted to the exception stack information file.
4. The method for resolving data loss due to unresponsive acquisition scripts according to claim 1, characterized in that, The steps to destroy an abnormal process include: Use the `kill -9` command to destroy the abnormal process, and then check if the destruction was successful. If so, modify the process information in the exception queue; If not, forcibly change the parent process of the abnormal process to 1. If the modification is successful, execute the following steps: use kill -9 pid to destroy the abnormal process and determine whether the destruction was successful; otherwise, use rm -rf / proc / pid to forcibly delete the file information corresponding to the abnormal process and modify the process information in the exception queue. Determine whether the counter corresponding to the name of the collected item exceeds the second threshold; If so, remove the corresponding collection item and restart Telegraf via a warm restart; otherwise, execute the next abnormal process in a loop.
5. The method for resolving data loss due to unresponsive acquisition scripts according to claim 4, characterized in that, The steps for recovering and persisting data collected by abnormal processes to the database include: Information is read sequentially from the dirty data file in memory, and then the data is retrieved in reverse using the Telegraf data processing decoding tool after the reading is complete. The acquired data is formatted and written to the InfluxDB database using the InfluxDB read / write component.
6. The method for resolving data loss due to unresponsive acquisition scripts according to claim 5, characterized in that, The steps for obtaining fault information of abnormal processes and, based on the obtained fault information, determining whether the abnormal process is recoverable, and adding the collected items of the abnormal process to the recoverable collection queue include: Read the exception stack trace file to obtain fault information; Determine whether the fault in the abnormal process has been recovered based on the fault information; If so, add the relevant data collection items to the recoverable data collection queue; Edit the collection items in the recoverable collection queue to restore the custom collection script, and restart Telegraf via a hot boot; Otherwise, no action will be taken.
7. A device for resolving data loss caused by a data acquisition script becoming unresponsive, characterized in that, It includes a detection component, an anomaly collection script process handling component, and a recovery component; The detection component is used to automatically identify abnormal processes in the monitoring and acquisition scripts and to classify these abnormal processes. An abnormal data collection script process handling component is used to process abnormal processes and persist the dependency data used for abnormal process data collection recovery and abnormal process collection item recovery. The recovery component is used to recover and persist the data collected from the abnormal process to the database; it obtains the fault information of the abnormal process, and if it determines that the abnormal process is recoverable based on the obtained fault information, it adds the collected items of the abnormal process to the recoverable collection queue. The steps for automatically identifying and classifying abnormal processes in the monitoring and data collection script include: Intercept all data collection items, record the name and process ID of the collection script, and write them to the collection queue; The process status is judged based on the fault information defined in the collection queue, abnormal processes are identified and classified and written into the corresponding abnormal queue. The exception queue includes a script-related problem queue and a non-script-related problem queue; the steps for classifying and writing exception processes into the corresponding exception queues include: The script used to determine whether an abnormal process is a custom script; If so, classify it directly into the non-script-specific problem type and add it to the non-script problem queue; If not, parse the stack trace of the abnormal process; Determine whether the parsed stack information includes the specified key information; If it exists, it is identified as a problem type of the script itself and written to the script problem queue; If it does not exist, it is identified as a non-script-specific problem type and written to the non-script problem queue; The steps for handling abnormal processes and persisting the dependent data used for abnormal process data recovery and abnormal process data item recovery include: Check if the abnormal process exists in the operating system by its process ID; If it does not exist, modify the process information in the exception queue; decrement the counter by 1 according to the name of the collection item. If it exists, persist the stack information of the abnormal process; Retrieve data from the stack trace corresponding to the abnormal process and persist the relevant data to a dirty data file in memory. Delete the abnormal process.