Dynamic real-time multi-memory fault collection method based on server

Abstract

The invention discloses a dynamic real-time multi-memory fault collection method based on a server, and relates to the technical field of computer system monitoring and fault management, and the method comprises the following steps: after the server is powered on and started, collecting time required by communication; taking the shortest communication time as a first time threshold value; starting to monitor a memory error register in the ECC register; obtaining error data in the memory error register and data related to memory errors in the system log; determining a query timestamp according to the data related to the memory error in the system log, performing query from the query timestamp, gradually obtaining the data related to the memory error in the system log, and reporting the data to a system; and the system sets a second time threshold value, and after the time for the system to monitor the memory fault reaches the preset second time threshold value, monitoring is ended. According to the method, the memory state can be dynamically monitored, a response is made immediately when a problem occurs instead of depending on regular inspection, and the fault diagnosis and repair efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer system monitoring and fault management, and in particular to a server-based dynamic real-time multi-memory fault collection method. Background Art

[0002] The field of computer system monitoring and fault management technology mainly studies how to monitor the operating status of computer systems in real time, discover and manage potential faults, and ensure the stability, efficiency and reliability of the system; In today's digital age, servers, as the core hub for various data processing, storage, and network services, bear the heavy responsibility of ensuring the stable operation of massive businesses. As server performance continues to rise, the complexity of memory systems is also increasing, and the requirements for their reliability are becoming more and more stringent; Traditional methods of collecting server memory faults have many limitations, which constitute the background of the current urgent problem to be solved; on the one hand, the previous collection methods focused on obtaining memory data of limited dimensions, and only paid attention to the basic information at the memory hardware level, which could only capture extremely limited fault manifestations and miss a large amount of key abnormal information hidden in the complex memory operation mechanism; on the other hand, timeliness is also a major shortcoming of traditional methods. Most of the existing memory fault collection follows a fixed time interval sampling strategy, with a long interval period, ranging from a few minutes to several hours. There is an urgent need for a method to solve the problem of insufficient memory fault data and timeliness caused by the single memory data collected by the server. Summary of the invention

[0003] The purpose of the present invention is to provide a server-based dynamic real-time multi-memory fault collection method to solve the problems raised in the prior art.

[0004] To achieve the above object, the present invention provides the following technical solution: a server-based dynamic real-time multi-memory fault collection method, the dynamic real-time multi-memory fault collection method specifically comprising the following steps: Step S1, after the server is powered on, the servers in the server cluster communicate with each other and the time required for communication is collected respectively; Step S2: acquiring the shortest communication time from the collected communication time, and using the shortest communication time as the first time threshold; Step S3, start monitoring the memory error register in the ECC register; Step S4: when a memory error is detected, error data in the memory error register is obtained, and data related to the memory error in the system log is obtained based on text processing; Step S5, determining a query timestamp through the data related to the memory error in the system log, wherein the query timestamp refers to marking the matching position of the memory error in the last system log obtained each time in the process of obtaining the data related to the memory error in the system log according to the text processing, and the mark is the query timestamp; Step S6, starting from the query timestamp, querying, gradually obtaining data related to the memory error in the system log, and reporting it to the system. During the gradual acquisition process, if the data related to the memory error in the system log is not completely obtained within the first time threshold, first report the currently obtained new memory error to the system, and then continue to obtain data related to the memory error in the system log until the data related to the memory error in the system log is completely obtained, and mark the query timestamp at this position; Step S7, the system sets a second time threshold, and when the preset second time threshold is reached, the monitoring ends; if the preset second time threshold is not reached, steps S3 to S6 are repeated.

[0005] In step S1, after the server is powered on, the kernel initialization is completed and various system services are loaded, including hardware checking, kernel loading and system service startup, so that the server is in normal use.

[0006] The text processing includes regular expressions, template filling and searching, and keyword matching.

[0007] The error data in the memory error register includes error type, error quantity and error occurrence address information.

[0008] The memory error related data in the system log includes the type, quantity, occurrence address and occurrence time information of the error.

[0009] When the time from monitoring and obtaining memory errors to reporting to the system is less than a first time threshold, an alarm is fed back to the administrator port.

[0010] The second time threshold is the total monitoring duration set in advance, specifically, the longest period of time that a server can run.

[0011] The dynamic real-time multi-element memory fault collection method comprises the following architecture: a communication module, a register monitoring module, a system log monitoring module, a text processing module and a fault reporting module; The communication module is responsible for managing and controlling the information exchange between servers, including data transmission and time synchronization, and collects the time required for communication and takes the shortest time required for communication as the first time threshold; The register monitoring module is used to monitor the memory error register in the ECC register, and obtain error information when an error is found; The system log monitoring module is used to obtain data related to memory errors in the system log, and obtain error information when an error is found; The text processing module is responsible for processing and integrating the error data obtained from the register monitoring module and the system log monitoring module, including deduplication, sorting and formatting operations; The fault reporting module is responsible for reporting the processed error data to the system, ensuring that the system discovers and handles the problem, and decides whether to end the monitoring according to a preset second time threshold.

[0012] The core of step S6 is the innovative use of a unique identifier (ID) system based on the server identification code, memory channel number, memory error occurrence time and type, so that even similar or repeated error information from different sources can be accurately identified and merged. This process not only improves the efficiency of data processing, but also greatly reduces the number of error reports, thereby reducing the burden on system maintenance personnel; The specific steps to query from the query timestamp, gradually obtain data related to memory errors in the system log, and report to the system include: Step S6-1, establishing an ID for each memory error, wherein the ID consists of 36 digits, 3 separators "-" and 4 decimal points ".", wherein the separators sequentially separate the 12-digit IP address, the 3-digit memory channel number, the 16-digit time, the 6-digit microseconds and the 1-digit memory error type code from left to right; wherein, the IP address firstly supplements the 4 groups of digits separated by the decimal point from the left to 3 digits, and then removes the decimal point and splices them into 12 digits; the time format adds 0 to 2 digits to the left of the hours, minutes and seconds that are less than 2 digits; Step S6-2, integrating and deduplicating the memory error-related data in the obtained system log, and combining the memory errors occurring on the same memory channel on the same server within a preset second time threshold into one memory error; Step S6-3: Report the memory error-related data in the deduplicated system log to the local system and the management node system.

[0013] ECC (Error-Correcting Code): Error checking and correction. ECC registers are registers that store ECC-related information and are used in computer systems to detect and correct memory data errors.

[0014] A server cluster is a group of independent servers connected to each other, working together, and these servers are connected through a network; The system is a local system and a management node system; The “immediately” means responding within the shortest time interval required for communication within the server cluster.

[0015] The second time threshold is the total monitoring time set in advance, specifically, the longest running period of a server, wherein the second time threshold may be an empirical value, or may be obtained by a certain method, and may be adjusted according to the actual application of the server to make it more reasonable. It should be noted that the method of adjusting the second time threshold includes but is not limited to one or more of the following methods: grid search, random search, Bayesian optimization, gradient descent method, genetic algorithm, particle swarm optimization, simulated annealing, differential evolution, model-based optimization, reinforcement learning, rule-based method, expert system, meta-heuristic algorithm, automatic machine learning.

[0016] Compared with the prior art, the beneficial effects of the present invention are: the present invention proposes a server-based dynamic real-time multi-memory fault collection method, which can dynamically monitor the memory status and allow the system to respond immediately when a problem occurs, rather than relying on periodic inspections, which improves the efficiency of fault diagnosis and repair; relevant information about memory faults is collected in real time, and the system can respond to faults quickly to improve system stability; diversified data collection provides a more comprehensive fault analysis perspective, which helps to accurately identify the root cause of the fault, thereby accelerating the troubleshooting process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a workflow diagram of a server-based dynamic real-time multi-memory fault collection method. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Example: Figure 1 As shown, the present invention provides a technical solution, a dynamic real-time multi-memory fault collection method based on a server, and the dynamic real-time multi-memory fault collection method specifically includes the following steps: Step S1, after the server is powered on, the servers in the server cluster communicate with each other and the time required for communication is collected respectively; Step S2: obtain the shortest communication time from the collected communication time, and use the shortest communication time as the first time threshold; and find the longest communication time between two servers in the server cluster to control the time for monitoring and reporting memory errors.

[0020] Step S3, start monitoring the memory error register in the ECC register; When the memory DIMM detects a memory error, it will send it to the caches and MCU memory controllers at all levels. After the CPU obtains this information, it stores it in its internal Aarch64 system registers; the register monitoring module will monitor the RAS-related Aarch64 system registers in the CPU and obtain relevant detailed information after a memory error occurs.

[0021] Step S4: when a memory error is detected, error data in the memory error register is obtained, and data related to the memory error in the system log is obtained based on text processing; Step S5, determining a query timestamp through the data related to the memory error in the system log, wherein the query timestamp refers to marking the matching position of the memory error in the last system log obtained each time in the process of obtaining the data related to the memory error in the system log according to the text processing, and the mark is the query timestamp; There are multiple moments separated by the first time threshold. , , , …, ,…,in is the initial moment, arrive During the time interval, a server's system log has If the pending information is The time has not yet been processed If there is information to be processed, the processed information will be reported to the system first, and the remaining information will continue to be processed. If the subsequent processing is still not completed, the first time threshold will be used as an interval, and the system will continue to process the remaining information at each time. Report at all times.

[0022] Step S6, starting from the query timestamp, querying, gradually obtaining data related to the memory error in the system log, and reporting it to the system. During the gradual acquisition process, if the data related to the memory error in the system log is not completely obtained within the first time threshold, first report the currently obtained new memory error to the system, and then continue to obtain data related to the memory error in the system log until the data related to the memory error in the system log is completely obtained, and mark the query timestamp at this position; Step S7, the system sets a second time threshold, and when the preset second time threshold is reached, the monitoring ends; if the preset second time threshold is not reached, steps S3 to S6 are repeated.

[0023] In step S1, after the server is powered on, the kernel initialization is completed and various system services are loaded, including hardware checking, kernel loading and system service startup, so that the server is in normal use.

[0024] The text processing includes regular expressions, template filling and searching, and keyword matching.

[0025] The error data in the memory error register includes error type, error quantity and error occurrence address information.

[0026] The memory error related data in the system log includes the type, quantity, occurrence address and occurrence time information of the error.

[0027] When the time from monitoring and obtaining memory errors to reporting to the system is less than a first time threshold, an alarm is fed back to the administrator port.

[0028] The second time threshold is the total monitoring duration set in advance, specifically, the longest period of time that a server can run.

[0029] The dynamic real-time multi-element memory fault collection method comprises the following architecture: a communication module, a register monitoring module, a system log monitoring module, a text processing module and a fault reporting module; The communication module is responsible for managing and controlling the information exchange between servers, including data transmission and time synchronization, and collects the time required for communication and takes the shortest time required for communication as the first time threshold; The register monitoring module is used to monitor the memory error register in the ECC register, and obtain error information when an error is found; The system log monitoring module is used to obtain data related to memory errors in the system log, and obtain error information when an error is found; The text processing module is responsible for processing and integrating the error data obtained from the register monitoring module and the system log monitoring module, including deduplication, sorting and formatting operations; The fault reporting module is responsible for reporting the processed error data to the system, ensuring that the system discovers and handles the problem, and decides whether to end the monitoring according to a preset second time threshold.

[0030] The core of step S6 is the innovative use of a unique identifier (ID) system based on the server identification code, memory channel number, memory error occurrence time and type, so that even similar or repeated error information from different sources can be accurately identified and merged. This process not only improves the efficiency of data processing, but also greatly reduces the number of error reports, thereby reducing the burden on system maintenance personnel; The specific steps to query from the query timestamp, gradually obtain data related to memory errors in the system log, and report to the system include: Step S6-1, establishing an ID for each memory error, wherein the ID consists of 36 digits, 3 separators "-" and 4 decimal points ".", wherein the separators sequentially separate the 12-digit IP address, the 3-digit memory channel number, the 16-digit time, the 6-digit microseconds and the 1-digit memory error type code from left to right; wherein, the IP address firstly supplements the 4 groups of digits separated by the decimal point from the left to 3 digits, and then removes the decimal point and splices them into 12 digits; the time format adds 0 to 2 digits to the left of the hours, minutes and seconds that are less than 2 digits; Step S6-2, integrating and deduplicating the memory error-related data in the obtained system log, and combining the memory errors occurring on the same memory channel on the same server within a preset second time threshold into one memory error; Among them, the memory error type code comparison table is as follows:

[0031] At time 1730427497.296820, a server with an IP address of 192.168.1.15 generated a correctable memory error on memory channel 8. The ID of the memory error is 192.168.001.015-008-20241101101817.296820-1.

[0032] Step S6-3: Report the memory error-related data in the deduplicated system log to the local system and the management node system.

[0033] The second time threshold is the total monitoring time set in advance, specifically, the longest running period of a server, wherein the second time threshold may be an empirical value, or may be obtained by a certain method, and may be adjusted according to the actual application of the server to make it more reasonable. It should be noted that the method of adjusting the second time threshold includes but is not limited to one or more of the following methods: grid search, random search, Bayesian optimization, gradient descent method, genetic algorithm, particle swarm optimization, simulated annealing, differential evolution, model-based optimization, reinforcement learning, rule-based method, expert system, meta-heuristic algorithm, automatic machine learning.

[0034] A server cluster is a group of independent servers connected to each other, working together, and these servers are connected through a network; ECC (Error-Correcting Code): Error checking and correction. ECC registers are registers that store ECC-related information and are used in computer systems to detect and correct memory data errors.

[0035] The system is a local system and a management node system; The “immediately” means responding within the shortest time interval required for communication within the server cluster.

[0036] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A server-based dynamic real-time multi-memory fault collection method, characterized in that: The dynamic real-time multi-element memory fault collection method specifically comprises the following steps: Step S1, after the server is powered on, the servers in the server cluster communicate with each other and the time required for communication is collected respectively; Step S2: acquiring the shortest communication time from the collected communication time, and using the shortest communication time as the first time threshold; Step S3, start monitoring the memory error register in the ECC register; Step S4: when a memory error is detected, error data in the memory error register is obtained, and data related to the memory error in the system log is obtained based on text processing; Step S5, determining a query timestamp through the data related to the memory error in the system log, wherein the query timestamp refers to marking the matching position of the memory error in the last system log obtained each time in the process of obtaining the data related to the memory error in the system log according to the text processing, and the mark is the query timestamp; Step S6, starting from the query timestamp, querying, gradually obtaining data related to the memory error in the system log, and reporting it to the system. During the gradual acquisition process, if the data related to the memory error in the system log is not completely obtained within the first time threshold, first report the currently obtained new memory error to the system, and then continue to obtain data related to the memory error in the system log until the data related to the memory error in the system log is completely obtained, and mark the query timestamp at this position; Step S7, the system sets a second time threshold, and when the preset second time threshold is reached, the monitoring ends; if the preset second time threshold is not reached, steps S3 to S6 are repeated.

2. A server-based dynamic real-time multi-memory fault collection method according to claim 1, characterized in that: In step S1, after the server is powered on, the kernel initialization is completed and various system services are loaded, including hardware inspection, kernel loading and system service startup, so that the server is in normal use.

3. The method for collecting dynamic and real-time multi-memory faults based on a server according to claim 1, characterized in that: The text processing includes regular expressions, template filling and searching, and keyword matching.

4. The method for collecting dynamic and real-time multi-memory faults based on a server according to claim 1, characterized in that: The error data in the memory error register includes error type, error quantity and error occurrence address information.

5. The method for collecting dynamic and real-time multi-memory faults based on a server according to claim 1, characterized in that: The memory error related data in the system log includes the type, quantity, occurrence address and occurrence time information of the error.

6. The method for collecting dynamic and real-time multi-memory faults based on a server according to claim 1, characterized in that: When the time from monitoring and obtaining memory errors to reporting to the system is less than a first time threshold, an alarm is fed back to the administrator port.

7. The method for collecting dynamic and real-time multi-memory faults based on a server according to claim 1, characterized in that: The second time threshold is the total monitoring duration set in advance, specifically, the longest period of time that a server can run.

8. The method for collecting dynamic and real-time multi-memory faults based on a server according to claim 1, characterized in that: The dynamic real-time multi-element memory fault collection method comprises the following architecture: a communication module, a register monitoring module, a system log monitoring module, a text processing module and a fault reporting module; The communication module is responsible for managing and controlling the information exchange between servers, including data transmission and time synchronization, and collects the time required for communication and uses the shortest time required for communication as the first time threshold; The register monitoring module is used to monitor the memory error register in the ECC register, and obtain error information when an error is found; The system log monitoring module is used to obtain data related to memory errors in the system log, and obtain error information when an error is found; The text processing module is responsible for processing and integrating the error data obtained from the register monitoring module and the system log monitoring module, including deduplication, sorting and formatting operations; The fault reporting module is responsible for reporting the processed error data to the system, ensuring that the system discovers and handles the problem, and decides whether to end the monitoring according to a preset second time threshold.

9. The method for collecting dynamic and real-time multi-memory faults based on a server according to claim 1, characterized in that: The specific steps to query from the query timestamp, gradually obtain data related to memory errors in the system log, and report to the system include: Step S6-1, establishing an ID for each memory error, wherein the ID consists of 36 digits, 3 separators and 4 decimal points, wherein the separators sequentially separate the 12-digit IP address, the 3-digit memory channel number, the 16-digit time, the 6-digit microseconds and the 1-digit memory error type code from left to right; wherein, the IP address firstly supplements the 4 groups of digits separated by the decimal point from the left to 3 digits, and then removes the decimal point and splices them into 12 digits; the time format adds 0 to 2 digits to the left of the hours, minutes and seconds that are less than 2 digits; Step S6-2, integrating and deduplicating the memory error-related data in the obtained system log, and combining the memory errors occurring on the same memory channel on the same server within a preset second time threshold into one memory error; Step S6-3: Report the memory error-related data in the deduplicated system log to the local system and the management node system.

Images