Data processing method, system and device

Abstract

The present invention provides a data processing method, system, and apparatus that can be applied to the fields of equipment performance monitoring technology and data storage technology. The data processing method includes: obtaining target monitoring data used to characterize performance changes of a device under test; determining, based on predetermined index rules, target index information referenced for performing read and write operations on the target monitoring data in a predetermined file; determining, based on the target index information and the data length used to write the monitoring data in the predetermined file, an offset of a target location for storing the target monitoring data in the predetermined file relative to a base write location; and writing the target monitoring data to a target location in the predetermined file based on the base write location and the offset.

Description

Technical Field

[0001] The present invention relates to the technical fields of equipment performance monitoring and data storage, and in particular to a data processing method, system and device. Background Art

[0002] With the rapid advancement of cloud computing and big data technologies, data centers, the critical infrastructure supporting these technologies, are growing in scale and complexity. With the surge in data center traffic, ensuring high reliability, stability, and availability has become increasingly critical. Therefore, performance monitoring and management of data center computing equipment is essential. After monitoring data from computing devices is collected using data acquisition devices (such as sensors), it needs to be written to files for subsequent data analysis.

[0003] During the implementation of the present invention, we discovered that low write efficiency exists in the data storage process. For example, one approach involves rewriting all data after detecting a new piece of data. While this approach has relatively simple code implementation logic, it requires rewriting the entire file each time new data is written, consuming significant time and computing resources. Summary of the Invention

[0004] In view of the above problems, the present invention provides a data processing method, system and device.

[0005] According to a first aspect of the present invention, there is provided a data processing method, comprising:

[0006] Acquire target monitoring data used to characterize performance changes of the equipment under test;

[0007] Determining target index information referenced by a predetermined file for performing read and write operations on target monitoring data based on a predetermined index rule;

[0008] Determining an offset of a target position for storing the target monitoring data in the predetermined file relative to a base writing position based on the target index information and a data length used for writing the monitoring data in the predetermined file;

[0009] Write the target monitoring data to the target location in the predetermined file based on the base write position and offset.

[0010] A second aspect of the present invention provides a management system, comprising:

[0011] Data acquisition equipment, used to collect target monitoring data that characterizes performance changes of the equipment under test;

[0012] Data processing equipment for performing the following operations:

[0013] Upon receiving target monitoring data sent by a data acquisition device, determining target index information referenced by a read and write operation on the target monitoring data in a predetermined file based on a predetermined index rule;

[0014] Determining an offset of a target position for storing the target monitoring data in the predetermined file relative to a base writing position based on the target index information and a data length used for writing the monitoring data in the predetermined file;

[0015] Write the target monitoring data to the target location in the predetermined file based on the base write position and offset.

[0016] A third aspect of the present invention provides a data processing device, comprising:

[0017] An acquisition module, used to acquire target monitoring data for characterizing performance changes of the device under test;

[0018] A determination module, configured to determine target index information referenced by a predetermined file for performing read and write operations on target monitoring data based on a predetermined index rule;

[0019] a calculation module, configured to determine an offset of a target position for storing target monitoring data relative to a base writing position in the predetermined file based on the target index information and a data length used for writing the monitoring data in the predetermined file;

[0020] The writing module is used to write the target monitoring data to the target position in the predetermined file according to the basic writing position and the offset.

[0021] A fourth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0022] The fifth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0023] The sixth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0025] Figure 1An application scenario diagram of a data processing method according to an embodiment of the present invention is shown;

[0026] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown;

[0027] Figure 3 A schematic diagram illustrating a principle of data appending during a data writing process according to an embodiment of the present invention is shown;

[0028] Figure 4 A schematic diagram illustrating a principle of performing fixed data overwriting during a data writing process according to an embodiment of the present invention is shown;

[0029] Figure 5 A schematic diagram illustrating a principle of performing data rolling overwriting during a data writing process according to an embodiment of the present invention is shown;

[0030] Figure 6 A schematic diagram showing a principle of data transmission under a system-level architecture according to an embodiment of the present invention is shown;

[0031] Figure 7 The figure shows the changes of data written into a file in a rolling overwrite scenario according to an embodiment of the present invention;

[0032] Figure 8 A schematic diagram illustrating sorting data in a file in a rolling overwrite scenario according to an embodiment of the present invention is shown;

[0033] Figure 9 A schematic diagram illustrating a structure of a management system according to an embodiment of the present invention is shown;

[0034] Figure 10 A block diagram schematically shows a structure of a data processing device according to an embodiment of the present invention;

[0035] Figure 11 The figure schematically shows a block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0037] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0038] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0039] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0040] An embodiment of the present invention provides a data processing method, including:

[0041] Obtain target monitoring data for characterizing performance changes of the device under test; determine target index information in a predetermined file for reference in performing read and write operations on the target monitoring data based on a predetermined index rule; determine an offset of a target position for storing the target monitoring data in the predetermined file relative to a base write position based on the target index information and the data length used for writing the monitoring data in the predetermined file; and write the target monitoring data to a target position in the predetermined file according to the base write position and the offset.

[0042] Figure 1 An application scenario diagram of a data processing method according to an embodiment of the present invention is shown.

[0043] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a client 101, a message bus 102, and a server 103. The client 101, the message bus 102, and the server 103 may communicate with each other via a network, which may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0044] The client 101 may be various terminal devices, such as various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The message bus 102 is based on an inter-process communication mechanism and is used to transmit messages between different processes, and is particularly used for communication between desktop applications and system services. For example, the D-message bus (Desktop Bus, DBus) may be used.

[0046] The client 101 and the message bus 102 can communicate with each other through the Intelligent Platform Management Interface (IPMI) or the Scalable Platforms Management API (Redfish).

[0047] The server 103 may be a controller for device performance management, including but not limited to a baseboard management controller (BMC) and an open baseboard management controller (OpenBMC).

[0048] In the scenario of an embodiment of the present invention, monitoring data representing changes in hardware performance of the device under test can be collected through data acquisition equipment, such as various sensors. The device under test can be, for example, a server in a data center. The monitoring data includes, but is not limited to, the temperature of the central processing unit (CPU), the temperature of the hard disk, the input current, the output current, the input voltage, the output voltage, etc.

[0049] The server 103 can be used to obtain monitoring data and perform data processing operations such as calculation and storage on the monitoring data. The client 101 can request the server 103 to obtain some or all of the monitoring data via the message bus 102. Based on the request, the server 103 can send the monitoring data to the client 101 via the message bus 102. After receiving the monitoring data, the client 101 can visualize it, for example, in the form of a chart to the user.

[0050] The following will be based on Figure 1 The scene described by Figures 2 to 11 The data processing method of the embodiment of the invention is described in detail.

[0051] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown.

[0052] like Figure 2 As shown, the data processing method of this embodiment includes operations S201 to S204.

[0053] In operation S201, target monitoring data for characterizing performance changes of a device under test is acquired;

[0054] In operation S202, based on a predetermined index rule, target index information referenced by performing a read or write operation on target monitoring data in a predetermined file is determined;

[0055] In operation S203, based on the target index information and the data length used to write the monitoring data in the predetermined file, an offset of a target position for storing the target monitoring data relative to a base writing position in the predetermined file is determined;

[0056] In operation S204 , the target monitoring data is written to a target location in a predetermined file according to the base write location and the offset.

[0057] The method of the embodiment of the present invention can be applied in the scenario of performance monitoring of data center servers. In this scenario, monitoring data representing changes in hardware performance of the device under test can be collected through data acquisition equipment, such as various sensors. The device under test can be, for example, a server in a data center. The monitoring data includes performance data of multiple components of the device under test, such as but not limited to CPU temperature, hard disk temperature, input current, output current, input voltage, output voltage, server power consumption, etc.

[0058] After sensors collect monitoring data from the device under test, controllers (such as BMCs and OpenBMCs) can process this data, performing preprocessing, calculations, storage, and analysis. BMCs and OpenBMCs are management devices in open-source embedded systems responsible for monitoring and managing server hardware. Recording monitoring data is key to understanding the real-time operating status of each server component. By collecting logs with one click or viewing historical data graphs on a webpage, users can intuitively understand whether all devices are operating normally and whether there are any potential faults.

[0059] The above data processing method can be executed by the controller to perform operations such as writing to the acquired monitoring data. This process is a real-time writing process. As the sensor collects monitoring data in real time, the collected data is written in real time.

[0060] The target monitoring data may be the monitoring data collected during the current time window that needs to be processed. Furthermore, the target monitoring data may be processed monitoring data obtained by further processing the monitoring data collected during the current time window. The target monitoring data may be used to characterize the performance changes of a target component of the device under test within a target time window; for example, the target monitoring data may be the temperature change data of a CPU during a certain time window, or the input current change data of a server during a certain time window.

[0061] A file can be created for each component (or sensor) of the device under test to store data from that component or sensor. A predetermined file is used to store monitoring data for a target component corresponding to the target monitoring data, or to store monitoring data collected by a target sensor corresponding to the target monitoring data.

[0062] After acquiring the target monitoring data, the target monitoring data can be written to a predetermined file. During the data writing process, one method may be to adopt a strategy of writing all monitoring data at once. For example, every time a sensor adds a new piece of data, the system will rewrite the historical monitoring data of all sensors to the file. This full-text overwrite method has relatively simple code implementation logic and is relatively convenient for writing data. It is more suitable when the amount of data is small, and the writing speed is not significantly affected. However, as server integration increases, the number of sensors increases, the data storage time increases, and the read and write frequency increases, this writing method will cause large amounts of data to be frequently erased and written repeatedly, and the system's write performance will decrease.

[0063] After acquiring the target monitoring data, the method of this embodiment of the present invention can write the target monitoring data to a predetermined file through operations S202 through S204, thereby resolving the issue of degraded system write performance. Specifically, by calculating an offset, the data is directly jumped to the write location for writing, eliminating the need for full overwriting or line-by-line searching, thereby improving data writing speed.

[0064] One or more pieces of monitoring data are stored in a predetermined file. These pieces of monitoring data are written to different locations in the file, with adjacent pieces of data being located consecutively. Index information can be set for each piece of monitoring data stored in the predetermined file, allowing for read and write operations on the monitoring data in the predetermined file. The index information can be a positional sequence number representing each piece of data in the predetermined file. For example, the positional sequence number of the first piece of data is 0, the positional sequence number of the second piece of data is 1, the positional sequence number of the third piece of data is 2, and so on.

[0065] In addition, the monitoring data stored in the predetermined file can be set to a fixed data length, that is, each monitoring data has a fixed length. For example, the length of each monitoring data is 12 bytes, or 28 bytes, or 36 bytes, etc. The value of the data length depends on the content of the data to be stored.

[0066] After determining the target index information of the target monitoring data through the above operation S202, the offset of the target position for storing the target monitoring data in the predetermined file relative to the basic write position can be determined through operation S203 based on the target index information of the target monitoring data and the data length used to write the monitoring data. For example, after setting the position sequence number of the target monitoring data, the product of the position sequence number of the target monitoring data and the data length is calculated to obtain the offset, and the data write position is controlled by the offset. Based on the basic write position and the offset, the target monitoring data is written to the target position in the predetermined file, for example, the target position = basic write position + offset. The basic write position can be the starting position of the file (the basic write position is marked as 0 Byte).

[0067] For example, when writing DATA_11 to File 1, assume that File 1 already contains 10 data items: DATA_1, DATA_2, DATA_3, ..., DATA_10. The index information for these 10 data items, or their position sequence numbers, are 0, 1, 2, 3, 4, ..., 9, respectively. The length of each data item is preset to 28 bytes. When DATA_11 needs to be appended to the end of the file, the position sequence number of DATA_11 in File 1 is determined to be 10 (which means there are already 10 data items in the file). The calculated offset is 10 * 28 bytes. If the base write position is the beginning of the file, the write position of this data in File 1 is calculated based on the offset to be 10 * 28 bytes. After DATA_11 is written to this position, the data is appended to the end of the file.

[0068] For example, when DATA_11 needs to be written to a certain position in the file, for example, DATA_11 needs to overwrite DATA_3, the position serial number of DATA_11 in file 1 is determined to be 2 (the position serial number of DATA_3 is directly assigned to DATA_11, which means there are 2 data before DATA_3). The offset can be calculated to be 2*28 bytes. When the basic write position is the starting position of the file, the write position of the data in file 1 is calculated according to the offset to be 2*28 bytes. After DATA_11 is written to this position, the data is overwritten by DATA_3.

[0069] According to an embodiment of the present invention, by calculating the offset, the data is directly jumped to the location where it needs to be written to write data, without the need to overwrite the entire text to perform frequent data writing operations, thereby improving the data writing speed. In addition, by setting the data in the file to a fixed data length, the offset can be directly, quickly and accurately calculated based on the index information and data length of the written data. This method is faster, and in the scenario of data overwriting, the writing of data will not destroy the structure of the original data and will not affect the reading of other data (if the data is not fixed length, when a certain data is overwritten, if the length of the written data and the overwritten data is inconsistent, the data structure of other data will be destroyed, resulting in the data being destroyed and unable to be read), ensuring the normal reading and writing of data.

[0070] According to embodiments of the present invention, monitoring data can be stored in files in a variety of formats. Because client displays are often based on the JSON format, which is a directly readable plaintext format, a common approach is to store monitoring data in the JSON format, which offers good compatibility. However, while the JSON format offers good readability and versatility, it contains a large number of redundant characters, such as quotation marks, commas, and curly braces, which significantly increase the size of stored files. As historical data accumulates, file system storage space rapidly depletes, increasing data management difficulties and hardware storage costs. Furthermore, the JSON format cannot ensure a fixed length for each data entry. When writing data, offsets cannot be quickly and accurately calculated, resulting in complex calculation logic. Overwriting data can also potentially corrupt the original data structure, making it impossible to ensure proper data reading and writing. Furthermore, when adding new data to a file, a line-by-line search is required to determine the write location before writing. This plaintext overwriting method requires line-by-line search to locate the data row to be written, resulting in a time complexity of O(n) (n represents the number of data rows). This time complexity is higher for larger files, and the efficiency decreases with larger files. In addition, frequent read and write operations may cause disk IO bottlenecks. If the row length fluctuates greatly, the space utilization may be low due to the inability to store compactly.

[0071] Based on this, the method of the embodiment of the present invention can use a binary format, such as a binary format, to store data in a file. Using a binary format to store data has the following effects:

[0072] When storing data in binary format, only the data itself is stored without useless characters. Therefore, the file size occupied by writing the same data is significantly reduced compared to the plaintext format.

[0073] When writing, binary format data is directly read and written according to the memory layout, without the need for syntax parsing. In addition, the binary format is much faster than the plaintext format when performing serialization and deserialization operations. For batch operations and concurrent operations, the number of I / O operations is reduced, making it suitable for high-throughput scenarios.

[0074] Binary formats are typically encrypted using the Advanced Encryption Standard (AES) and asymmetric algorithms such as Rivest-Shamir-Adleman (RSA). Key cracking is complex and can effectively resist attacks such as brute force cracking and cryptanalysis, providing a solid defense for data security. Because the binary format offers streamlined storage and a smaller data size, the probability of data being maliciously scanned or intercepted during storage and network transmission is significantly reduced. Smaller data volumes mean shorter transmission times and fewer windows of exposure, while also reducing the amount of information attackers can access. This significantly improves data security and confidentiality throughout its lifecycle, providing reliable protection for the storage and transmission of sensitive information in data centers.

[0075] When storing data in binary format, only the data itself is stored, and useless characters are not stored. Therefore, in the same scenario, it is relatively easy to set each data to a fixed data length. This ensures that each data has a fixed length. When writing data, the offset can be calculated quickly and accurately. Moreover, when the data is overwritten, the original data structure will not be destroyed, ensuring that the data can be read and written normally.

[0076] When storing data in binary format, there is no need to search row by row to locate the row to be written, as in plain text format. Instead, the data is located directly by offset, avoiding row-by-row scanning. The time complexity is O(1). This method optimizes the time complexity from O(n) to O(1). That is, regardless of the file size, the speed of the overwrite operation is constant, which can greatly improve the data writing speed.

[0077] According to embodiments of the present invention, there are various scenarios for writing target monitoring data to a predetermined file, including: appending the target monitoring data to the predetermined file, i.e., writing it to the end of the data; overwriting a specific piece of existing monitoring data in the predetermined file with the target monitoring data; and adding the target monitoring data to the predetermined file in a rolling overwrite manner, i.e., overwriting the first data stored with the target monitoring data. In different scenarios, the target index information (e.g., a position sequence number) referenced by the target monitoring data in the predetermined file for read and write operations can be determined based on different indexing rules, and an offset can be calculated based on this index information.

[0078] According to an embodiment of the present invention, when target monitoring data is appended to a predetermined file, i.e., written to the end of the data, determining target index information in the predetermined file as a reference for performing read and write operations on the target monitoring data based on a predetermined index rule includes: if the amount of data in the predetermined file does not reach a predetermined threshold, determining the target index information based on the number of times at least one existing monitoring data item has been written to the predetermined file. This method is applicable when the amount of existing data in the predetermined file has not reached the limit.

[0079] Figure 3 FIG. 1 shows a schematic diagram of data appending during data writing according to an embodiment of the present invention. Figure 3 As shown in the figure, suppose a file is used to store CPU temperature data. The file has a limit of 100, meaning it can only store a maximum of 100 data items. Assume that the file already contains 99 data items, DATA_1, DATA_2, DATA_3, ..., DATA_99. The index information for these 99 data items, or their position sequence numbers, are 0, 1, 2, 3, 4, ..., 98, respectively. The length of each data item is preset to 28 bytes. When the 100th data item, DATA_100, is captured by the sensor and needs to be written to the file, it can be directly appended to the end of the file because the file limit has not been reached.

[0080] When DATA_100 is written to a file, the target index information—the position sequence number—is determined based on the number of times the existing data in the file has been written to the intended file. There are 99 pieces of data in the file, each written once, for a total of 99 writes. This determines the position sequence number of DATA_100 in the file as 99, and the calculated offset is 99*28 bytes. When the base write position is the beginning of the file, the offset calculation yields the write position of this data in File 1 as 99*28 bytes. After DATA_100 is written to this position, the data is appended to the end of the file.

[0081] According to an embodiment of the present invention, when a new piece of monitoring data is added, it is written in an appending manner rather than overwriting the entire file. This appending manner significantly improves the data writing speed and writing performance.

[0082] According to an embodiment of the present invention, in a scenario where the target monitoring data is used to fixedly overwrite one of the existing monitoring data in the predetermined file, for example, a certain historically collected data is unavailable and needs to be overwritten with newly collected data, or in a scenario where the amount of data in the predetermined file has reached a predetermined threshold and the new data needs to overwrite one of the data, the target index information referenced for performing read and write operations on the target monitoring data in the predetermined file is determined based on the predetermined index rules, including: the index information of the target existing monitoring data to be overwritten, which is determined as the target index information.

[0083] Figure 4 A schematic diagram illustrating a principle of performing fixed data overwriting during a data writing process according to an embodiment of the present invention is shown.

[0084] like Figure 4 As shown, 100 pieces of data DATA_1, DATA_2, DATA_3, ... DATA_100 are stored in the current file. The index information of these 100 pieces of data, that is, the position sequence numbers are: 0, 1, 2, 3, 4 ... 99 respectively. The length of each piece of data is preset to 28 bytes.

[0085] When the 101st data DATA_101 is collected by the sensor and needs to be written into the file, one scenario is, for example: the amount of data in the file has not yet reached the data quantity limit, or has reached the quantity limit, and a piece of historically collected data is unavailable, such as DATA_51 is unavailable, and it needs to be overwritten with the newly collected data DATA_101.

[0086] Or another scenario is, for example: the file's quantity limit is 100. Because the quantity limit is reached, when storing DATA_101, the data is full, and the newly written data will overwrite one of the existing data (assuming it is set to overwrite DATA_51).

[0087] In the above two scenarios, the target index information referenced for performing read and write operations on the target monitoring data in the predetermined file is determined based on the predetermined index rules, including: the index information of the target existing monitoring data to be overwritten, and the target index information determined as the target monitoring data, that is, the index information of the target existing monitoring data to be overwritten is directly assigned to the target monitoring data currently to be written.

[0088] like Figure 4, it is necessary to write DATA_101 to a certain position in the file and overwrite DATA_51, directly assign the position serial number of DATA_51 to DATA_101, and determine that the position serial number of DATA_101 in the file is 50 (which can also be understood as there are 50 data before DATA_51). The offset can be calculated as 50*28 Byte. When the basic write position is the starting position of the file, the write position of the data in the file is calculated according to the offset = 50*28 Byte. After DATA_101 is written to this position, the data is overwritten by DATA_51.

[0089] According to an embodiment of the present invention, in the scenario of data overwriting, the index information of the target existing monitoring data to be overwritten is directly assigned to the target monitoring data currently to be written, so that the data can be accurately and quickly overwritten. In addition, under the writing mechanism of the embodiment of the present invention, because the offset is calculated based on the index information and the data length, under the premise that all data is fixed in length, even if a certain data is randomly overwritten, it can be ensured that the writing position calculated based on the offset is always the starting position of the overwritten data, and the written data only overwrites one of the data, and will not destroy the data structure of other data, thereby ensuring the normal reading and writing of data.

[0090] According to an embodiment of the present invention, in a scenario where the amount of data in a predetermined file has reached a predetermined threshold and new data needs to overwrite one of the data, in order to ensure the referenceability of the data, rolling overwriting can be performed, that is, the latest collected data overwrites the first written data.

[0091] In this scenario, based on the number of times at least one historical monitoring data item is written into a predetermined file and a predetermined threshold, the index information of the target existing monitoring data item to be overwritten is determined, and the calculated index information of the target existing monitoring data item to be overwritten is assigned to the target monitoring data item currently being written. The at least one historical monitoring data item includes at least one existing monitoring data item and the overwritten monitoring data item.

[0092] Furthermore, determining the index information of the target existing monitoring data based on the number of times at least one historical monitoring data is written into a predetermined file and a predetermined threshold includes: determining the index information of the target existing monitoring data based on the difference between the number of times at least one historical monitoring data is written into a predetermined file and a predetermined threshold.

[0093] Figure 5 FIG. 2 shows a schematic diagram of a data rolling overwrite process according to an embodiment of the present invention. Figure 5As shown in the example, the file's limit is set to 100. The file currently contains 100 data items: DATA_101, DATA_102, DATA_3, DATA_4, ..., DATA_100. DATA_101 and DATA_102 were added after the file reached its limit, overwriting the oldest items, DATA_1 and DATA_2, respectively. The index information for these 100 data items, or their position sequence numbers, are 0, 1, 2, 3, 4, ..., 99, respectively. The length of each data item is preset to 28 bytes.

[0094] When the 103rd data item DATA_103 is collected by the sensor and needs to be written to a file, because the file quantity limit is reached, it is necessary to determine which data is to be overwritten by DATA_103 and which is currently written earliest (DATA_3 in the figure). The data to be overwritten can be determined by calculating the index information, and the calculated index information of the existing data to be overwritten can be directly assigned to DATA_103.

[0095] The multiple historical monitoring data written to the file include the existing data DATA_101, DATA_102, DATA_3, DATA_4, ... DATA_100, as well as the overwritten data DATA_1 and DATA_2. Therefore, a total of 102 data items have been written, each written once, for a total of 102 writes. The difference between the number of writes of the multiple historical data items and the predetermined threshold is calculated as 102 - 100 = 2, resulting in the position sequence number of DATA_103 = 2. The offset can be calculated as 2 * 28 bytes. When the base write position is the starting position of the file, the write position of the data in the file calculated based on the offset is 2 * 28 bytes. After DATA_103 is written to this position, it overwrites the earliest written DATA_3.

[0096] According to an embodiment of the present invention, through the above method, when the data in the file is full, rolling overwriting can be achieved, and the latest data will overwrite the earliest written data, thereby ensuring the referenceability of the data.

[0097] According to an embodiment of the present invention, the target monitoring data may be monitoring data collected during the current time window and currently in need of processing. Furthermore, the target monitoring data may be used to characterize performance changes of a target component of the device under test within a target time window; for example, the target monitoring data may be temperature change data of a CPU within a certain time window, or input current change data of a server within a certain time window.

[0098] When monitoring data for multiple components of a device under test (DUT) over multiple time windows is stored in a memory used to store predetermined files, the monitoring data for the multiple components is stored in multiple files, where a target component is one of the multiple components and the predetermined file is one of the multiple files. In other words, a file may be created for each component (or each sensor) of the DUT to store the data for that component or sensor. The predetermined file is a file used to store monitoring data for a target component corresponding to the target monitoring data, or a file used to store monitoring data collected by a target sensor corresponding to the target monitoring data.

[0099] For example, a file can be created for each sensor or component, using the sensor or component name as the file name to store all its historical data. Compared to storing all sensor or component data in a single file, this approach allows independent data write operations, avoiding interference and frequent overwrites, and significantly improving write efficiency. When the number of sensors reaches hundreds, write efficiency can be increased several times. This not only ensures timely and accurate data recording, providing a reliable basis for analytical decision-making, but also reduces system response time, enhancing real-time performance and stability. Efficient write performance reduces hardware resource consumption and lowers operating costs.

[0100] When writing, if the file for the component / sensor of the current monitoring data does not exist in the memory, it is necessary to create the file first and write the data to the file in an appended manner.

[0101] Figure 6 FIG1 shows a schematic diagram of data transmission under the system-level architecture of an embodiment of the present invention. Figure 6 As shown, the system-level architecture of the embodiment of the present invention includes a client-application layer, a message bus, a server, and a sensor, wherein the client and the message bus can communicate via the IPMI protocol or the Redfish protocol.

[0102] The message bus can use DBus as an intermediary layer for inter-process communication, achieving decoupling between processes. The core of DBus is to enable cross-process communication through bus message routing and standardized interfaces. To create a DBus, you must specify the object path (e.g., / org / example / Device), the interface (e.g., org.example.Sensor), and the bus name (e.g., org.freedesktop.DBus).

[0103] All sensor data will be updated to DBus in real time. For example, taking the temperature monitoring data of CPU0 (CPU0_Temp) as an example, the data of CPU0_Temp is stored on DBus as follows:

[0104] Bus name: xyz.openbmc_project.CPUSensor;

[0105] Object Path:

[0106] / xyz / openbmc_project / sensors / temperature / CPU0_Temp;

[0107] Interface name: xyz.openbmc_project.Sensor.Value;

[0108] Attribute name: Value;

[0109] Property type: double;

[0110] Attribute value: 36.536.

[0111] The server is a controller used to manage device performance data, such as a BMC or OpenBMC. It can write sensor data to files or read data from files.

[0112] In the scenario where the server writes data, the monitoring data collected by all sensors will be updated to the message bus in real time. The server obtains the latest monitoring data of a component collected by the sensor through the underlying driver and writes the data to the file created for the sensor or component through the method of the embodiment of the present invention. The sensor name or component name can be used as the file name, and a file can be created for each sensor name or component to store all its historical data, such as Figure 6 As shown: For different components such as CPU0, CPU1, Inlet, etc., separate files are created to record their respective temperature monitoring data.

[0113] According to an embodiment of the present invention, a data storage method may be pre-configured and stored in configuration information. During data writing, the configuration information is obtained and data is stored according to the pre-configured method.

[0114] You can pre-configure information such as the frequency of data storage (for example, once every hour), the length of time for data storage (for example, one year), and so on.

[0115] Furthermore, when writing data to different types of memory, the storage duration can be dynamically adapted. Based on this, the method of an embodiment of the present invention further includes: if the memory used to store the predetermined file is an extended memory, setting the storage duration of the predetermined file to a first duration; if the memory is not an extended memory, setting the storage duration of the predetermined file to a second duration, where the first duration is greater than the second duration. Examples of extended memory include, but are not limited to, micro secure digital cards (TF cards) and secure digital cards (SD cards).

[0116] For example, for each sensor or component, you can set the storage duration when a TF card is available and when no TF card is available. Before storing data, you can first detect whether there is a TF card. If there is no TF card, the storage duration is 1 year, and if there is a TF card, the storage duration is 10 years.

[0117] Through pre-configuration, the storage period for historical monitoring data for each sensor / component can be dynamically adapted. If not configured, when the data storage period reaches the configured upper limit, new data will automatically overwrite the oldest data entry. This approach differs from fixed storage mode in that it allows users to precisely configure storage time with or without extended storage based on sensor characteristics, and both storage modes can be used in parallel. This configuration maximizes storage resources, avoids waste, and provides users with an adaptive storage solution. Users can adjust settings at any time without modifying the underlying system, improving data management autonomy and convenience.

[0118] According to an embodiment of the present invention, the client may request the server to obtain part or all of the monitoring data via the message bus. The server may send the monitoring data to the client via the message bus based on the request. After receiving the monitoring data, the client may perform a visual display, for example, by displaying it to the user in the form of a chart.

[0119] One data transmission mechanism involves the server reading all historical data from all sensors at once. After receiving this massive amount of data, the client extracts data based on sensor name and a specified time period, displaying it as a graph. When the data storage period reaches its upper limit (for example, one or ten years), the sheer volume of data causes DBus, IPMI, or Redfish to respond very slowly, sometimes taking tens of seconds. This significantly occupies system resources, leading not only to significant data display delays but also potentially causing system downtime, significantly impacting server stability.

[0120] Based on this, the method of the embodiment of the present invention is improved by passing the sensor / component name and time window parameters in the request to achieve accurate filtering and acquisition.

[0121] The method of an embodiment of the present invention includes: responding to a data request sent by a client for obtaining target monitoring data, extracting target monitoring data corresponding to a target component and a target time window from multiple files in a memory based on field information related to the target monitoring data carried in the data request, wherein the field information includes component type information and timestamp information; and sending the target monitoring data to the client via a message bus so that the client displays the target monitoring data.

[0122] For example, when a user wants to obtain data from a specific sensor over a period of time, the server reads specific historical data from a file based on the sensor name and time period (start time, end time). This eliminates the need to return all data from all sensors, significantly reducing the pressure on data transmission and eliminating the need for secondary data processing before displaying it on the front end, which also improves speed. Specifically, the Uniform Resource Locator (URL) setting in the request is optimized by adding component name / sensor name parameters and time window parameters (including start time and end time) to the URL parameters. For example, when requesting data from a specific sensor, the URL is set as follows:

[0123] URL: / redfish / v1 / … / <sensor name> ?$start=<timestamp>&$end=<timestamp;

[0124] After receiving the URL, the server parses the parameters, obtains the sensor's data for a period of time based on the passed parameters, and returns it to the application layer in JSON format for display.

[0125] According to an embodiment of the present invention, precise filtering and retrieval is achieved by passing field information related to the target monitoring data in the request. The server only extracts matching data, reducing transmission volume. The front-end can display the data without secondary processing, significantly shortening data display time, facilitating rapid information acquisition for operations and maintenance personnel, and ensuring efficient data center operation.

[0126] According to an embodiment of the present invention, the target monitoring data may be monitoring data obtained by further processing a plurality of original monitoring data collected in the current time window.

[0127] The data management process on the server side monitors data changes from all sensors. The sensors collect raw monitoring data at a preset collection cycle, for example, every 10 seconds. If the sensor data changes during the current collection cycle, the latest raw monitoring data is obtained. If there is no change, the raw monitoring data from the previous collection cycle is obtained.

[0128] A time window can correspond to multiple collection cycles. Data processing is performed once each time window passes. The raw monitoring data collected over the multiple collection cycles corresponding to the current time window is processed to generate the target monitoring data. For example, if a collection cycle is set to 10 seconds and a time window is set to 1 minute, the sensor collects data every 10 seconds. At the end of the current time window, the six raw monitoring data collected within that minute are processed to generate the target monitoring data corresponding to the current time window.

[0129] The processing of the plurality of raw monitoring data corresponding to the current time window may be to calculate the maximum value, the minimum value and the average value of the plurality of raw monitoring data as the target monitoring data. When writing the target monitoring data into the predetermined file, the timestamp of the data record may be obtained and written into the file at the same time.

[0130] For example, if the target monitoring data includes a timestamp (4 bytes), a maximum value (8 bytes), a minimum value (8 bytes), and an average value (8 bytes), then the offset of the target location for storing the target monitoring data in the predetermined file relative to the base write location, determined based on the target index information and data length, can be: offset = index * 28 (4-byte timestamp + 8-byte maximum value + 8-byte minimum value + 8-byte average value).

[0131] According to an embodiment of the present invention, writing target monitoring data to a predetermined file indicates persistent storage of the target monitoring data. After collecting multiple raw monitoring data over multiple collection cycles within a current time window, the multiple raw monitoring data can be first written to a memory queue, and then processed and calculated to obtain target monitoring data, i.e., the maximum, minimum, and average values ​​are calculated. The target monitoring data is then also written to the memory queue, and then written to the predetermined file for persistent storage.

[0132] The data in the memory queue is temporarily stored. After the target monitoring data is written to the predefined file for persistent storage, the data in the memory queue remains in place until the server terminates, at which point the data in memory is deleted. When the server restarts, the data is restored and loaded into memory from the predefined file.

[0133] Through the above method, during the operation of the server, since the target monitoring data in the memory queue is still retained, when the client requests the target monitoring data, the data in the memory queue is directly returned without having to be read from the file. In this way, there is no need to read the file frequently, but the data can be directly parsed from the memory and returned, which speeds up the data response speed.

[0134] Based on the description of the foregoing embodiments, it can be seen that there are various scenarios for writing monitoring data into a predetermined file, such as: appending the latest acquired monitoring data to a file; for example, using the latest acquired monitoring data to overwrite one of the specific existing monitoring data in the file; for example, adding the latest acquired monitoring data to the file in a rolling overwrite manner.

[0135] In the scenario where the latest acquired monitoring data is added to the file in a rolling overwrite manner, the latest written data is written at the earliest written position, that is, at the front end of the file.

[0136] Figure 7 FIG. 1 shows the changes in the data written into the file in a rolling overwrite scenario according to an embodiment of the present invention. Figure 7 As shown, assuming that the file quantity limit is 100, when the data in the file is not overwritten, the 100 data stored in the file are sorted in order of their positions: DATA_1, DATA_2, DATA_3, DATA_4, ..., DATA_100.

[0137] When two new data items, DATA_101 and DATA_102, are added, DATA_101 and DATA_102 overwrite the oldest items, DATA_1 and DATA_2, respectively. The 100 data items stored in the file are sorted in order of chronological order: DATA_101, DATA_102, DATA_3, DATA_4, ..., DATA_100. This sorting method has drawbacks for users, as conventional reading habits dictate that the data written first is typically the first data collected, and data in a file is typically sorted in the order in which it was collected. However, if some data is overwritten, this reading behavior is violated, resulting in a poor user experience.

[0138] Based on this, after the latest acquired monitoring data is added to the file in a rolling overwriting manner, the data in the file can be sorted based on the order in which the data was collected.

[0139] Figure 8 FIG. 1 shows a schematic diagram of sorting data in a file in a rolling overwrite scenario according to an embodiment of the present invention. Figure 8 As shown in the figure, before sorting, the 100 data items stored in the file are sorted in order of priority: DATA_101, DATA_102, DATA_3, DATA_4, ... DATA_100. After sorting, the 100 data items stored in the file are sorted in order of priority: DATA_3, DATA_4, ... DATA_100, DATA_101, DATA_102. This sorting method is more in line with common reading habits and provides a better user experience.

[0140] A second aspect of the present invention provides a management system, Figure 9 The structure diagram of a management system according to an embodiment of the present invention is schematically shown.

[0141] like Figure 9 As shown, the management system 900 according to this embodiment may include a data collection device 901 and a data processing device 902 .

[0142] The data acquisition device 901 is used to collect target monitoring data that characterizes performance changes of the device under test. The data acquisition device 901 can be, for example, various sensors, used to collect monitoring data that characterizes hardware performance changes of the device under test. The device under test can be, for example, a server in a data center. The monitoring data includes, but is not limited to, CPU temperature, hard disk temperature, input current, output current, input voltage, output voltage, etc.

[0143] The data processing device 902 is configured to perform the following operations:

[0144] Upon receiving target monitoring data sent by a data acquisition device, determining target index information referenced by a read and write operation on the target monitoring data in a predetermined file based on a predetermined index rule;

[0145] Determining an offset of a target position for storing the target monitoring data in the predetermined file relative to a base writing position based on the target index information and a data length used for writing the monitoring data in the predetermined file;

[0146] Write the target monitoring data to the target location in the predetermined file based on the base write position and offset.

[0147] In one embodiment, the specific implementation method of the data processing device 902 performing the above operations can refer to the operations S201 to S204 described above, which will not be repeated here.

[0148] Based on the above data processing method, the present invention also provides a data processing device. Figure 10 The device is described in detail.

[0149] Figure 10 Schematically shows a structural block diagram of a data processing device according to an embodiment of the present invention. Figure 10 As shown, the data processing device 1000 of this embodiment includes an acquisition module 1001 , a determination module 1002 , a calculation module 1003 , and a writing module 1004 .

[0150] The acquisition module 1001 is used to acquire target monitoring data for characterizing performance changes of the device under test. In one embodiment, the acquisition module 1001 can be used to perform the operation S201 described above, which will not be repeated here.

[0151] The determination module 1002 is used to determine the target index information referenced by the predetermined file for performing read and write operations on the target monitoring data based on the predetermined index rules; in one embodiment, the determination module 1002 can be used to perform the operation S202 described above, which will not be repeated here.

[0152] The calculation module 1003 is used to determine the offset of the target position for storing the target monitoring data in the predetermined file relative to the basic write position based on the target index information and the data length used to write the monitoring data in the predetermined file; in one embodiment, the calculation module 1003 can be used to perform the operation S203 described above, which will not be repeated here.

[0153] The writing module 1004 is used to write the target monitoring data to the target location in the predetermined file according to the basic writing location and the offset. In one embodiment, the writing module 1004 can be used to perform the operation S204 described above, which will not be repeated here.

[0154] According to an embodiment of the present invention, the determination module includes a first determination unit configured to determine the index information of the target existing monitoring data to be overwritten as the target index information when the data volume of the predetermined file reaches a predetermined threshold.

[0155] According to an embodiment of the present invention, the determination module also includes a second determination unit, which is used to determine the index information of the target existing monitoring data based on the number of times at least one historical monitoring data is written into a predetermined file and a predetermined threshold, wherein the at least one historical monitoring data includes at least one existing monitoring data and overwritten monitoring data.

[0156] According to an embodiment of the present invention, the second determining unit includes a calculating subunit for determining index information of target existing monitoring data based on a difference between the number of times at least one historical monitoring data is written into a predetermined file and a predetermined threshold.

[0157] According to an embodiment of the present invention, the determination module further includes a third determination unit for determining target index information based on the number of times at least one existing monitoring data is written into the predetermined file when the data volume of the predetermined file does not reach a predetermined threshold.

[0158] According to an embodiment of the present invention, the above-mentioned device further includes:

[0159] A first setting module is used to set the storage time of the predetermined file to a first time when the memory used to store the predetermined file is an extended memory;

[0160] The second setting module is used to set the storage time of the predetermined file to a second time when the memory does not belong to the extended memory, and the first time is greater than the second time.

[0161] According to an embodiment of the present invention:

[0162] The target monitoring data is used to characterize the performance changes of the target components of the equipment under test within the target time window;

[0163] When monitoring data of multiple components of the device under test in multiple time windows are stored in a memory for storing predetermined files, the monitoring data of the multiple components are respectively stored in multiple files, wherein the target component is one of the multiple components and the predetermined file is one of the multiple files.

[0164] According to an embodiment of the present invention, the above-mentioned device further includes:

[0165] an extraction module, configured to, in response to a data request for obtaining target monitoring data sent by a client, extract the target monitoring data corresponding to the target component and the target time window from a plurality of files in a memory based on field information related to the target monitoring data carried in the data request, wherein the field information includes component type information and timestamp information;

[0166] The sending module is used to send the target monitoring data to the client via the message bus, so that the client can display the target monitoring data.

[0167] According to an embodiment of the present invention, any multiple modules among the acquisition module 1001, determination module 1002, calculation module 1003, and writing module 1004 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present invention, at least one of the acquisition module 1001, determination module 1002, calculation module 1003, and writing module 1004 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the acquisition module 1001 , the determination module 1002 , the calculation module 1003 , and the writing module 1004 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0168] Figure 11 The figure schematically shows a block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention.

[0169] like Figure 11 As shown, an electronic device 1100 according to an embodiment of the present invention includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 1101 may also include onboard memory for caching purposes. Processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0170] RAM 1103 stores various programs and data required for the operation of electronic device 1100. Processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Processor 1101 executes the programs in ROM 1102 and / or RAM 1103 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0171] According to an embodiment of the present invention, electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to bus 1104. Electronic device 1100 may also include one or more of the following components connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN card or modem. Communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1110 as needed, so that computer programs read from the removable media can be installed into storage section 1108 as needed.

[0172] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0173] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include ROM 1102 and / or RAM 1103 described above, and / or one or more memories other than ROM 1102 and RAM 1103.

[0174] The embodiments of the present invention further include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to cause the computer system to implement the method provided by the embodiments of the present invention.

[0175] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when executed by the processor 1101. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0176] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1109, and / or installed from removable media 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0177] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1109 and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0178] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0179] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0180] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0181] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire target monitoring data used to characterize performance changes of the equipment under test; When the amount of data in the predetermined file reaches a predetermined threshold, the index information of the target existing monitoring data to be overwritten is determined as the target index information in the predetermined file for reference in performing read and write operations on the target monitoring data; when the amount of data in the predetermined file does not reach the predetermined threshold, the target index information is determined based on the number of times at least one existing monitoring data is written into the predetermined file, wherein the index information represents a position sequence number of each piece of monitoring data in the predetermined file; Based on the target index information and the data length used for writing the monitoring data in the predetermined file, determine the offset of the target position for storing the target monitoring data in the predetermined file relative to the base write position; wherein the monitoring data is stored in the predetermined file using a fixed data length, and the offset is obtained by calculating the product of the position sequence number of the target monitoring data and the data length; according to the base write position and the offset, write the target monitoring data to the target position in the predetermined file, the base write position being the starting position of the file; wherein the monitoring data in the predetermined file is stored in binary format, storing only the data itself; the target monitoring data includes a timestamp, a maximum value, a minimum value, and an average value; Writing the target monitoring data into a predetermined file includes: collecting multiple original monitoring data, calculating the maximum value, minimum value and average value of the multiple original monitoring data, writing the target monitoring data into a memory queue, and writing the target monitoring data into a predetermined file for persistent storage. When the client requests the target monitoring data, the target monitoring data in the memory queue is directly returned.

2. The method according to claim 1, characterized in that The method further comprises: The index information of the target existing monitoring data is determined according to the number of times at least one historical monitoring data is written into the predetermined file and the predetermined threshold, wherein the at least one historical monitoring data includes at least one existing monitoring data and overwritten monitoring data.

3. The method according to claim 2, characterized in that Determining the index information of the target existing monitoring data according to the number of times at least one historical monitoring data is written into the predetermined file and the predetermined threshold includes: Based on the difference between the number of times the at least one historical monitoring data is written into the predetermined file and the predetermined threshold, index information of the target existing monitoring data is determined.

4. The method according to claim 1, wherein The method further comprises: In a case where the memory used to store the predetermined file is an extended memory, setting the storage duration of the predetermined file to a first duration; In a case where the memory is not an extended memory, the storage duration of the predetermined file is set to a second duration, and the first duration is greater than the second duration.

5. The method according to claim 1, wherein: The target monitoring data is used to characterize the performance changes of the target component of the device under test within a target time window; In a case where monitoring data of multiple components of the device under test in multiple time windows are stored in a memory for storing the predetermined file, the monitoring data of the multiple components are respectively stored in multiple files, wherein the target component is one of the multiple components and the predetermined file is one of the multiple files.

6. The method according to claim 5, characterized in that The method further comprises: In response to a data request for obtaining target monitoring data sent by a client, extracting the target monitoring data corresponding to a target component and a target time window from the plurality of files in the memory based on field information related to the target monitoring data carried in the data request, wherein the field information includes component type information and timestamp information; The target monitoring data is sent to the client via a message bus, so that the client displays the target monitoring data.

7. A management system, characterized in that: The system comprises: Data acquisition equipment, used to collect target monitoring data that characterizes performance changes of the equipment under test; Data processing equipment for performing the following operations: Upon receiving the target monitoring data sent by the data acquisition device, if the data volume of the predetermined file has reached a predetermined threshold, determining the index information of the target existing monitoring data to be overwritten as the target index information referenced by the predetermined file for performing read and write operations on the target monitoring data; and if the data volume of the predetermined file has not reached the predetermined threshold, determining the target index information based on the number of times at least one existing monitoring data has been written into the predetermined file, wherein the index information represents a position sequence number of each piece of monitoring data in the predetermined file; Determining, based on the target index information and the data length used for writing the monitoring data in the predetermined file, an offset of a target location for storing the target monitoring data relative to a base writing location in the predetermined file; wherein the monitoring data is stored in the predetermined file using a fixed data length, and the offset is obtained by calculating the product of a location sequence number of the target monitoring data and the data length; Writing the target monitoring data to the target position in the predetermined file according to the basic writing position and the offset, where the basic writing position is the starting position of the file; The monitoring data in the scheduled file is stored in binary format, which only stores the data itself; the target monitoring data includes timestamp, maximum value, minimum value, and average value; Writing the target monitoring data into a predetermined file includes: collecting multiple original monitoring data, calculating the maximum value, minimum value and average value of the multiple original monitoring data, writing the target monitoring data into a memory queue, and writing the target monitoring data into a predetermined file for persistent storage. When the client requests the target monitoring data, the target monitoring data in the memory queue is directly returned.

8. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire target monitoring data for characterizing performance changes of the device under test; a determination module, configured to, if the amount of data in the predetermined file has reached a predetermined threshold, determine index information of target existing monitoring data to be overwritten as target index information referenced by a read or write operation on the target monitoring data in the predetermined file; and, if the amount of data in the predetermined file has not reached the predetermined threshold, determine the target index information based on the number of times at least one existing monitoring data has been written into the predetermined file, wherein the index information represents a position sequence number of each piece of monitoring data in the predetermined file; a calculation module, configured to determine, based on the target index information and the data length used to write the monitoring data in the predetermined file, an offset of a target location for storing the target monitoring data relative to a base writing location in the predetermined file; wherein the monitoring data is stored in the predetermined file using a fixed data length, and the offset is obtained by calculating the product of a location sequence number of the target monitoring data and the data length; a writing module, configured to write the target monitoring data into the target position in the predetermined file according to the basic writing position and the offset, wherein the basic writing position is the starting position of the file; The monitoring data in the scheduled file is stored in binary format, which only stores the data itself; the target monitoring data includes timestamp, maximum value, minimum value, and average value; Writing the target monitoring data into a predetermined file includes: collecting multiple original monitoring data, calculating the maximum value, minimum value and average value of the multiple original monitoring data, writing the target monitoring data into a memory queue, and writing the target monitoring data into a predetermined file for persistent storage. When the client requests the target monitoring data, the target monitoring data in the memory queue is directly returned.

Images