Power supply scheduling log management method and system based on sliding window sorting
Through the sliding window sorting method, the problem of time disordered log information in the intelligent power supply scheduling system is solved, and centralized storage and efficient query of log information are realized in the standard format.
Patent Information
- Application Number
- CN202411152601.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-06
AI Technical Summary
In the intelligent power supply scheduling system, the log information formats generated by multiple subsystems, processes and threads vary greatly, resulting in a confusing time during centralized storage, uncertain order, and difficulty in conducting correlation analysis.
The sliding window sorting method is adopted to collect and sort log information in linear order and timestamp through the log collection end. The sliding window is set to upload to the log collection end regularly and sort and store it on the collection end to ensure the correct order of the log information.
It realizes centralized storage of multiple subsystems, processes, and thread log information on a single node, solves the problem of time disorder and improves the query efficiency of log analysis.
Smart Images

Figure CN119088781A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data transmission, and in particular to a power supply scheduling log management method and system based on sliding window sorting. Background Art
[0002] When the intelligent power supply dispatching system is deployed and operated, multiple subsystems and multiple processes will run simultaneously on the same node. Each process will have multiple threads running simultaneously. During the continuous and long-term operation, these processes and threads will generate a large amount of log information. The relevant log information includes the log information generated when the platform system is running, the log data generated when the application software is running, the log data generated by related operations, the log information generated by real-time events and faults, and other log information. The intelligent power supply dispatching system needs to store these log information centrally to facilitate debugging, error checking and correlation analysis when accidents occur.
[0003] There are many subsystems, processes, and threads running on the same node at the same time. The log information formats generated by each subsystem and thread are quite different. If they are not stored centrally, it will be very inconvenient to analyze and review their respective log information.
[0004] On the same node, when the log information generated by all subsystems, processes, and threads is centrally stored, the time required to transmit and receive log data information varies. At the same time, when receiving logs, the multiple threads responsible for receiving perform thread switching based on CPU time slices, which may cause the corresponding order of log data received from multiple processes to be disordered. These reasons may cause the centrally stored log information to be in the wrong order, making it inconvenient to associate related log information of subsystems and multiple processes and perform associated query analysis. Summary of the invention
[0005] The purpose of the present invention is to provide a power supply scheduling log management method and system based on sliding window sorting to solve the above problems in the prior art.
[0006] The embodiments of the present invention are implemented by the following technical solutions:
[0007] In a first aspect, the present invention provides a power supply scheduling log management method based on sliding window sorting, comprising:
[0008] Collect the log information of the power supply dispatching system through several log collection terminals, and sort the individual log information according to the linear order and timestamp of the log occurrence;
[0009] Set a sliding window, and upload the log information collected in the previous sliding window to the log collection end every T0 time;
[0010] The log collection end sorts the collected log information through sliding window sorting and stores it in the log data system file.
[0011] Preferably, the sorting by sliding window includes:
[0012] Get the log information uploaded by each different log collection end and sort it by linear order or timestamp;
[0013] Determine whether the log information to be sorted comes from the same log collection end. If so, sort them according to the order in which they are received; if not, sort them according to the timestamp.
[0014] After sorting is completed, the sorted log queue is output.
[0015] Preferably, the sorting and storage includes a log data sorting thread and a storage thread, and the log data sorting thread includes:
[0016] Get the log information to be inserted in a collection window, and get the first log information that is sorted last and comes from the same log collection end from the sorted log queue;
[0017] If the acquisition is successful, the log information to be inserted is sorted according to the time of the first log information to be inserted, and the corresponding log information to be inserted is inserted into the sorted log queue according to the time and occurrence order;
[0018] If the acquisition is unsuccessful, the first second log information is read from the log information to be inserted, and the first third log information is acquired from the sorted log queue, where the upload time of the third log information is greater than or equal to the upload time of the second log information;
[0019] Preferably, it also includes:
[0020] If the third log information is successfully obtained, the log information to be inserted is sorted according to the upload time starting from the third log information, and the corresponding log information to be inserted is inserted into the sorted queue according to the time and the order of log occurrence;
[0021] If the third log information is not successfully obtained, the log information to be inserted is inserted from the end of the sorted log queue.
[0022] Preferably, the storage thread includes:
[0023] Setting a check time interval, when the check time interval is reached, obtaining a fourth log information of a sliding window to be stored from the sorted log queue, where the difference between the upload time of the fourth log information and the current time is less than or equal to the check time interval;
[0024] The fourth log information is removed from the sorted log queue, and a log information queue to be stored is formed, and the log information in the log information queue to be stored is stored, waiting for the next round of storage;
[0025] If the inspection time interval has not been reached, wait until the inspection time interval is reached.
[0026] In a second aspect, the present invention further provides a power supply scheduling log management system based on sliding window sorting, comprising:
[0027] The acquisition module is configured to collect log information of the power supply dispatching system through a plurality of log collection terminals, and sort the individual log information according to the linear order and timestamp of the log occurrence;
[0028] The upload module is configured to set a sliding window, and upload the log information collected in the previous sliding window to the log collection terminal every T0 time;
[0029] The storage module is configured so that the log collection end sorts the collected log information through sliding window sorting and stores it in the log data system file.
[0030] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned power supply scheduling log management method based on sliding window sorting when executing the computer program.
[0031] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned power supply scheduling log management method based on sliding window sorting.
[0032] The technical solution of the embodiment of the present invention has at least the following advantages and beneficial effects:
[0033] The method provided by the present invention mainly includes collecting log information of the power supply dispatching system through several log collection terminals, and sorting the single log information according to the linear order and timestamp of the log occurrence. The log collection terminal sorts the collected log information through sliding window sorting and stores it in the log data system file. The intelligent power supply dispatching system running on a single node centrally stores the log information generated by multiple subsystems, multiple processes, and threads in a standard format.
[0034] The above method can solve the problem that when log information is stored in a centralized manner, the time of related log information may be disordered, which makes it difficult to analyze the problem. When log information is stored in a centralized manner, the linear order of log information with the same timestamp may be destroyed when it is sorted and queried by time, which is inconvenient for log analysis. When analyzing log information, it has been stored according to time and the order of occurrence. When querying, there is no need to sort the log information, which improves query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0036] Figure 1 A flow chart of sorting and storing the log collection end of the present invention;
[0037] Figure 2 The log sorting storage flow chart of the present invention;
[0038] Figure 3 Upload the flow chart for the acquisition end of the present invention;
[0039] Figure 4 The overall structure is stored in a centralized manner for a single-node log of the present invention. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0041] The terms "first, second" and the like in the specification and claims of this application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.
[0042] The division of modules in this application is a logical division. There may be other division methods when implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.
[0043] The modules or submodules described independently may be physically separated or not: they may be implemented by software or hardware, and some modules or submodules may be implemented by software, and the processor may call the software to implement the functions of these modules or submodules, and other modules or submodules may be implemented by hardware, such as by hardware circuits. In addition, some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.
[0044] Please refer to Figure 1-Figure 4 The present invention provides a power supply scheduling log management method based on sliding window sorting, comprising:
[0045] S101: Collect log information of the power supply dispatching system through several log collection terminals, and sort the individual log information according to the linear order and timestamp of log occurrence;
[0046] The intelligent power supply dispatching system running on a single node is composed of multiple subsystems, and each subsystem is composed of multiple processes.
[0047] A log collection terminal system is run in real time on the node to centrally store the log information generated by all processes running the intelligent power supply scheduling system on the node.
[0048] Each process carries and runs a log collection terminal module, which is responsible for collecting relevant log information from the process and uploading it to the log collection terminal system for centralized storage.
[0049] The log information of a single node is stored in the log data system file using a sliding window sorting method, which facilitates the correlation analysis and query of the log information.
[0050] S102: Setting a sliding window, uploading the log information collected in the previous sliding window to the log collection terminal every T0 time;
[0051] S103: The log collection terminal sorts the collected log information through sliding window sorting and stores it in the log data system file.
[0052] Specifically, each process has a log collection terminal responsible for collecting and uploading the log information of the process.
[0053] When a single process is running continuously, each thread will generate a large amount of log information, which will be collected centrally at the log collection end first;
[0054] When the log collection module collects data, the log information of a single process is sorted according to the linear order and timestamp of the log occurrence, ensuring that the log information related to a single process is in the correct order.
[0055] In order to ensure that the log information received by the log collection end from a single process is in the correct order, a sliding window method is used. At every T0 interval, the log information collected in the previous sliding window (T0) is uploaded to the log collection end.
[0056] In this way, the order of each sliding window received by the log collector from a single process is correct, and the order of the log information received from multiple processes by the log collector will not be destroyed.
[0057] The method provided by the present invention mainly includes collecting log information of the power supply dispatching system through several log collection terminals, and sorting the single log information according to the linear order and timestamp of the log occurrence. The log collection terminal sorts the collected log information through sliding window sorting and stores it in the log data system file. The intelligent power supply dispatching system running on a single node centrally stores the log information generated by multiple subsystems, multiple processes, and threads in a standard format.
[0058] The above method can solve the problem that when log information is stored in a centralized manner, the time of related log information may be disordered, which makes it difficult to analyze the problem. When log information is stored in a centralized manner, the linear order of log information with the same timestamp may be destroyed when it is sorted and queried by time, which is inconvenient for log analysis. When analyzing log information, it has been stored according to time and the order of occurrence. When querying, there is no need to sort the log information, which improves query efficiency.
[0059] In addition, the log collection end uses multi-threading to receive multiple log data packets from multiple processes in real time, and forms an original log queue for log reception in the order of reception.
[0060] The log data information in each log data packet has the correct log order and timestamp; however, the order of the log information in the original queue received from different processes may be disordered. If the log information is stored directly in the order of receipt, the related log information may be stored out of order.
[0061] In order to facilitate log information query and analysis, the received log information needs to be sorted before being stored. An exemplary implementation of the present invention includes sorting through a sliding window:
[0062] Get the log information uploaded by each different log collection end, and sort them in linear order or timestamp; determine whether the log information to be sorted comes from the same log collection end. If so, sort them in the order of receipt; if not, sort them by timestamp; after sorting, output the sorted log queue.
[0063] Specifically, from the received original log queue, obtain each sliding window log information uploaded from different processes, and sort them by timestamp + linear order. When sorting, logs from different processes are mainly sorted by time. For log data received from the same process, sort them by the order of receipt + the linear order of the received sliding window log itself. In this way, the log information received by each process is turned into a sorted log data queue.
[0064] Considering issues such as the delay in log transmission, for the sorted log data queue, at every T2 time interval, the log data of the sliding window (T1) that is T2 time away from the current time is obtained from the sorted log queue for storage.
[0065] When logs are stored in the above manner, the stored log information is the sorted log information, which is stored by obtaining the data of the fixed window (T1) at the T2 time interval each time, thereby avoiding the log information subsequently collected from related processes. It is possible that the log information may belong to the previous sliding window (T1) when sorted by time, resulting in the log information not being stored according to the requirements of correct sorting.
[0066] In an exemplary embodiment of the present invention, the sorting storage includes a log data sorting thread and a storage thread, and the log data sorting thread includes:
[0067] S201: Obtaining log information to be inserted in a collection window, and obtaining the first log information that is sorted last and comes from the same log collection end from the sorted log queue;
[0068] S202: If the acquisition is successful, sort the log information to be inserted according to the time of the first log information position acquired, and insert the corresponding log information to be inserted into the sorted log queue according to the time and occurrence sequence;
[0069] For example, if there are two log messages A and B in the collection window, the default order in the collection window is A, B. When inserting into the sorting window, log B cannot be inserted before log A.
[0070] S203: If the acquisition is unsuccessful, read the first second log information from the log information to be inserted, and obtain the first third log information from the sorted log queue, where the upload time of the third log information is greater than or equal to the upload time of the second log information;
[0071] Specifically, it also includes:
[0072] If the third log information is successfully obtained, the log information to be inserted is sorted according to the upload time starting from the third log information, and the corresponding log information to be inserted is inserted into the sorted queue according to the time and the order of log occurrence;
[0073] If the third log information is not successfully obtained, the log information to be inserted is inserted from the end of the sorted log queue.
[0074] In an exemplary embodiment of the present invention, the storage thread includes:
[0075] Setting a check time interval, when the check time interval is reached, obtaining a fourth log information of a sliding window to be stored from the sorted log queue, where the difference between the upload time of the fourth log information and the current time is less than or equal to the check time interval;
[0076] The fourth log information is removed from the sorted log queue, and a log information queue to be stored is formed, and the log information in the log information queue to be stored is stored, waiting for the next round of storage;
[0077] If the inspection time interval has not been reached, wait until the inspection time interval is reached.
[0078] Among them, the sliding window size can be set according to the actual situation. The window size (T0) of the log collection end is in the range of 20-50 milliseconds, the window size (T1) of the log collection end is in the range of 150-250 milliseconds, and the log storage interval (T2) is in the range of 150-250 milliseconds. T1 and T2 are usually configured to be the same.
[0079] In a second aspect, the present invention further provides a power supply scheduling log management system based on sliding window sorting, comprising:
[0080] The acquisition module is configured to collect log information of the power supply dispatching system through a plurality of log collection terminals, and sort the individual log information according to the linear order and timestamp of the log occurrence;
[0081] The upload module is configured to set a sliding window, and upload the log information collected in the previous sliding window to the log collection terminal every T0 time;
[0082] The storage module is configured so that the log collection end sorts the collected log information through sliding window sorting and stores it in the log data system file.
[0083] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned power supply scheduling log management method based on sliding window sorting when executing the computer program.
[0084] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned power supply scheduling log management method based on sliding window sorting.
[0085] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0087] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A power supply scheduling log management method based on sliding window sorting, characterized in that: include: Collect the log information of the power supply dispatching system through several log collection terminals, and sort the individual log information according to the linear order and timestamp of the log occurrence; Set a sliding window, and upload the log information collected in the previous sliding window to the log collection end every T0 time; The log collection end sorts the collected log information through sliding window sorting and stores it in the log data system file.
2. The power supply scheduling log management method based on sliding window sorting according to claim 1 is characterized in that: The sorting by sliding window includes: Get the log information uploaded by each different log collection end and sort it by linear order or timestamp; Determine whether the log information to be sorted comes from the same log collection end. If so, sort them according to the order in which they are received; if not, sort them according to the timestamp. After sorting is completed, the sorted log queue is output.
3. The power supply scheduling log management method based on sliding window sorting according to claim 2 is characterized in that: The sorting storage includes a log data sorting thread and a storage thread, and the log data sorting thread includes: Get the log information to be inserted in a collection window, and get the first log information that is sorted last and comes from the same log collection end from the sorted log queue; If the acquisition is successful, the log information to be inserted is sorted according to the time of the first log information to be inserted, and the corresponding log information to be inserted is inserted into the sorted log queue according to the time and occurrence order; If the acquisition is unsuccessful, the first second log information is read from the log information to be inserted, and the first third log information is acquired from the sorted log queue, where the upload time of the third log information is greater than or equal to the upload time of the second log information.
4. The power supply scheduling log management method based on sliding window sorting according to claim 3 is characterized in that: Also includes: If the third log information is successfully obtained, the log information to be inserted is sorted according to the upload time starting from the third log information, and the corresponding log information to be inserted is inserted into the sorted queue according to the time and the order of log occurrence; If the third log information is not successfully obtained, the log information to be inserted is inserted from the end of the sorted log queue.
5. The power supply scheduling log management method based on sliding window sorting according to claim 4 is characterized in that: The storage thread includes: Setting a check time interval, when the check time interval is reached, obtaining a fourth log information of a sliding window to be stored from the sorted log queue, where the difference between the upload time of the fourth log information and the current time is less than or equal to the check time interval; The fourth log information is removed from the sorted log queue, and a log information queue to be stored is formed, and the log information in the log information queue to be stored is stored, waiting for the next round of storage; If the inspection time interval has not been reached, wait until the inspection time interval is reached.
6. The power supply scheduling log management system based on sliding window sorting is characterized by: include: The acquisition module is configured to collect log information of the power supply dispatching system through a plurality of log collection terminals, and sort the individual log information according to the linear order and timestamp of the log occurrence; The upload module is configured to set a sliding window, and upload the log information collected in the previous sliding window to the log collection terminal every T0 time; The storage module is configured so that the log collection end sorts the collected log information through sliding window sorting and stores it in the log data system file.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the power supply scheduling log management method based on sliding window sorting according to any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the power supply scheduling log management method based on sliding window sorting according to any one of claims 1 to 5 is implemented.
