Data processing method, electronic equipment and storage medium

Abstract

The invention discloses a data processing method, electronic equipment and a storage medium, and the method comprises the steps: setting a duration adjustment parameter according to the configuration information of a target sensor; when the prediction data monitored by the target sensor in the current time window is determined by using the historical data collected in the historical time window, obtaining an error value between the prediction data and the actual data collected in the current time window, and comparing the error value with the duration adjustment parameter; adjusting the first duration of the current time window to the second duration of the next time window according to the comparison result; the method comprises the following steps: screening first data sets collected in a historical time window, a current time window and a next time window by using a group of network models to obtain a second data set; through the method, the technical problem that the utilization rate of the storage space is low in the processing process of the time sequence data in the related technology is solved, and the technical effect of saving the storage space in the server is achieved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, an electronic device, and a storage medium. Background Art

[0002] In the related art, a Baseboard Management Controller (BMC) system usually collects data of multiple sensors by using a sampling method with a fixed frequency. For example, it samples data such as temperature, voltage, and power consumption every 5 minutes to monitor the status information of the server hardware structure.

[0003] However, since the monitoring data of multiple sensors fluctuates little over a long period of time, the above - mentioned method of collecting and directly storing time - series data with little change at a fixed frequency will inevitably consume a large amount of storage resources of the BMC, resulting in the technical problem of low storage space utilization rate during the processing of time - series data. Summary of the Invention

[0004] This application provides a data processing method, an electronic device, and a storage medium to at least solve the problem of low storage space utilization rate during the processing of time - series data in the related art.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, including: during the operation of the server, obtaining historical data collected within a historical time window, where the historical data is data monitored by a target sensor and used to describe the status information of the server hardware; based on the historical data, determining predicted data of the target sensor within a current time window; obtaining an error value between the predicted data and the actual data, and adjusting the first duration of the current time window to the second duration of the next time window adjacent to the current time window by comparing the error value and a duration adjustment parameter, where the duration adjustment parameter is used to represent the duration change rate of the next time window; creating a first data set based on the data collected within the historical time window, the current time window, and the next time window; using a set of network models to screen the data in the first data set to obtain a second data set, where each network model in the set of network models includes a recurrent neural network structure and an attention layer.

[0006] According to another aspect of the embodiments of the present application, there is also provided a data processing device, including: a first acquisition unit, configured to acquire historical data collected within a historical time window during the operation of the server, where the historical data is data monitored by a target sensor and used to describe the hardware state information of the server; a first processing unit, configured to determine predicted data of the target sensor within the current time window based on the historical data; a second acquisition unit, configured to acquire an error value between the predicted data and the actual data, and adjust the first duration of the current time window to a second duration of the next time window adjacent to the current time window by comparing the error value and a duration adjustment parameter, where the duration adjustment parameter is used to represent the duration change rate of the next time window; a creation unit, configured to create a first data set based on the data collected within the historical time window, the current time window, and the next time window; a second processing unit, configured to screen the data in the first data set by using a set of network models to obtain a second data set, where each network model in the set of network models includes a recurrent neural network structure and an attention layer.

[0007] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute the steps of any of the above data processing methods through the computer program.

[0008] According to yet another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the steps of any of the above data processing methods when running.

[0009] According to yet another aspect of the embodiments of the present application, there is provided a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of any of the above data processing methods.

[0010] Through the above embodiments provided by the present application, based on the historical data collected within the historical time window, the predicted data monitored by the target sensor within the current time window is predicted, and based on the comparison result between the error value and duration adjustment parameter between the predicted data and the actual data, the duration of the next time window is adjusted in real time to achieve the effect of adaptively adjusting the sampling step size. At the same time, by using a set of network models to screen the data in the first data set obtained by adaptive sampling, a second data set strongly associated with the current monitoring task is obtained. In other words, by adopting a combination of adaptive sampling frequency adjustment and data screening, the amount of time-series data collected is reduced, the demand for data storage space is decreased, the problem of excessive consumption of storage space for time-series data in the related art is solved, and the technical effect of improving the storage space utilization rate of time-series data is achieved. Description of the Drawings

[0011] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0012] Figure 1 It is a schematic diagram of the application scenario of a data processing method according to an embodiment of the present application.

[0013] Figure 2 It is a schematic flowchart of an optional data processing method according to an embodiment of the present application.

[0014] Figure 3 It is a schematic diagram of the calculation process of an optional sampling step size according to an embodiment of the present application.

[0015] Figure 4 It is a schematic diagram of an optional adaptive sampling method according to an embodiment of the present application.

[0016] Figure 5 It is a schematic diagram of an optional method for sparsifying data according to an embodiment of the present application.

[0017] Figure 6 It is a schematic diagram of the structure of an optional deep learning model according to an embodiment of the present application.

[0018] Figure 7 It is the overall flowchart of an optional data processing method according to an embodiment of the present application.

[0019] Figure 8 It is a structural block diagram of an optional data processing device according to an embodiment of the present application. Detailed Embodiments

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0021] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] In order to enable those skilled in the art of the present technology to better understand the solutions of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] According to one aspect of the embodiments of the present application, a data processing method is provided. Optionally, in this embodiment, the above data processing method may be applied, but is not limited to, a hardware scenario as Figure 1 shown, where the server device may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processors 102 may include, but are not limited to, processing devices such as a microprocessor MCU or a field programmable gate array FPGA) and a memory 104 for storing data. Among them, the above server device may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above server device. For example, the server device may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0024] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the data processing method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories may be connected to the server device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the server device. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] The embodiments of the present application can be but are not limited to applicable to scenarios that require real-time monitoring and efficient storage of time-series data, especially environments with large amounts of data, limited storage resources, and high requirements for anomaly detection. Specific examples of several application scenarios are given below:

[0027] (1) Data center and server monitoring: For example, the collection and processing of monitoring data such as temperature sensors and pressure sensors to ensure that anomalies in the server running state can be captured and alarmed in a timely manner. At the same time, this technology can also be used for the monitoring of internal hardware of the server, such as CPU temperature and memory voltage, to improve the operation and maintenance efficiency and fault response speed of the data center.

[0028] (2) Industrial automation and manufacturing: On an industrial automation production line, sensors are used to monitor the operating state of equipment and the production process, such as parameters such as pressure, temperature, and speed. Therefore, the technical solution of the present application can adaptively adjust the data collection frequency, quickly respond to equipment failures or production anomalies, and at the same time optimize the storage strategy to reduce the data storage cost.

[0029] (3) Internet of Things device management: During the operation of Internet of Things devices, a large amount of environmental and device status data is generated. The technical solution of this application can be, but is not limited to, applied in fields such as smart home and health monitoring. Through adaptive data collection and deep learning data processing, the timeliness of data is improved and the transmission and storage costs are reduced.

[0030] (4) Smart grid and energy management: In a smart grid, parameters such as voltage, current, and frequency need to be monitored in real time to ensure the stable operation of the grid and the power quality. Therefore, the technical solution of this application can not only reduce the resource consumption of data collection and storage, but also prevent grid failures through anomaly detection technology, which is of great significance for the optimal scheduling and fault prediction of the power system.

[0031] The data processing method of the embodiments of this application can be executed by a server device, or can be executed by the server device in combination with at least one of terminal devices (which can also be understood as input / output device 108). Among them, the data processing method of the embodiments of this application executed by the terminal device can also be executed by a client installed thereon.

[0032] Taking the data processing method in this embodiment executed by the server as an example, Figure 2 is a schematic flowchart of an optional data processing method according to the embodiments of this application, as Figure 2 shown, the process of this method can include steps S202 to S210.

[0033] Step S202, during the operation of the server, obtain historical data collected within a historical time window, where the historical data is data monitored by a target sensor and used to describe the hardware status information of the server.

[0034] Step S204, based on the historical data, determine the predicted data of the target sensor within the current time window.

[0035] Step S206, obtain the error value between the predicted data and the actual data, and adjust the first duration of the current time window to the second duration of the next time window adjacent to the current time window by comparing the error value and the duration adjustment parameter, where the duration adjustment parameter is used to represent the duration change rate of the next time window.

[0036] Step S208, based on the data collected within the historical time window, the current time window, and the next time window, create a first data set.

[0037] Step S210, use a set of network models to screen the data in the first data set to obtain a second data set, where each network model in the set of network models includes a recurrent neural network structure and an attention layer.

[0038] In the BMC system of the current server, the most traditional fixed frequency is mostly used for collecting sensor data, that is, collecting once at intervals of a preset time. The collected data is mostly stored in files without processing or stored in a database. Limited by the storage space of the BMC, the frequency of collecting sensor data is generally low, mostly once every 5 minutes, and the storage duration of the historical data of each sensor is generally 7 to 30 days.

[0039] Using the traditional method of collecting sensor data has at least the following disadvantages: (1) At a sampling frequency of once every 5 minutes, some abnormal conditions that occur in a short period of time may not be recorded; (2) The values of most sensors tend to be stable for a long time, and these stable data are stored without change, wasting the storage space of the BMC.

[0040] To solve the above problems, the technical solution of this application first adopts an adaptive sampling frequency to reduce the possibility of sensor values not being collected in abnormal situations; secondly, it identifies the intensively sampled data collected, retains the valuable part, and compresses the part that is stable and unchanged, making the best use of the storage space.

[0041] In this embodiment, the historical time window can be, but is not limited to, the sensor data collected within a specific time period in the sampling frequency adaptive algorithm; the target sensor can be, but is not limited to, the sensor built in the BMC of the server, for example, a temperature sensor, a voltage sensor, a power consumption sensor, etc. This application does not make any limitations in this regard.

[0042] The above-mentioned hardware status information of the server includes, but is not limited to, the real-time measured values such as the temperature of the server's CPU (Central Processing Unit), memory voltage, power consumption, etc., and these data can reflect the working status of the server hardware.

[0043] Combined with the description in the above embodiments, the technical solution of this application can be widely applied to scenarios that require real-time monitoring and efficient storage of time-series data. For the sake of easy understanding, the implementation process of the above data processing method will be described below by taking the process of collecting and processing sensor data during server monitoring as an example.

[0044] S11, initialize the settings;

[0045] According to the characteristics and parameters of the target sensor, set a minimum step size S min 、maximum step size S max and slow start threshold S t, where the slow start threshold is a key parameter in the TCP congestion control process, used to control the switch between the slow start phase and the congestion avoidance phase of a TCP connection. Its function is to limit the growth rate of the congestion window to avoid network congestion.

[0046] In this embodiment, the slow start threshold can also be understood as a duration adjustment parameter, used to control the duration change rate of the next time window. In other words, the duration adjustment parameter is used to control how fast the acquisition step (the duration of each time window) increases or decreases.

[0047] It should be noted that TCP (Transmission Control Protocol) congestion control is a very important mechanism in network communication. Its core idea is to dynamically adjust the sending rate of the sender to avoid network congestion and make the most efficient use of network resources as much as possible.

[0048] The goals of TCP congestion control include (1) avoiding network congestion: preventing the sender from sending too many data packets, causing routers and links in the network to be overloaded, thereby leading to packet loss and increased delay; (2) making efficient use of network resources: improving data transmission efficiency as much as possible on the premise of not causing congestion and making full use of network bandwidth.

[0049] S12, data acquisition and adaptive sampling;

[0050] The first stage (data acquisition): Collect sensor data according to the current step S.

[0051] The second stage (step adjustment): Analyze the difference between the new data and the predicted value based on the average value of the recent n data points.

[0052] Among them, the recent n data points can be but are not limited to at least one time window closer to the current one within the historical time window. For example, if the current time window is the 5th time window and n = 2, then the data collected within the 3rd and 4th time windows are obtained, and based on the data collected within these two time windows, the sensor data that may be collected within the 5th time window (which can also be understood as the current time window) is predicted. The implementation process of determining the predicted data monitored by the target sensor within the current time window based on historical data will be described in detail below in combination with specific embodiments.

[0053] S13, sparsification processing process;

[0054] (1) Data preprocessing and model training:

[0055] In a data center environment, a deep learning model is trained using a large-scale training dataset (i.e., dense raw sensor data), adopting an Encoder-Decoder framework. The encoder can use, but is not limited to, an LSTM (Long Short-Term Memory) structure, and the decoder includes an Attention layer to identify and locate key change points in time-series data.

[0056] (2) Small model cluster deployment:

[0057] Given the BMC hardware resource limitations of data center servers, the knowledge of the trained large model is transferred to multiple small models, which automatically adjust their scales and quantities according to the resources of the BMC to adapt to different computing and storage conditions.

[0058] The training method of a teacher-student model is used, with the deep learning large model as the teacher model to guide the learning process of the small models (student models), ensuring that the small models can effectively capture the features and change trends of the data.

[0059] (3) Data sparsification analysis and output:

[0060] The data collected from sensors is input into the small model cluster, and each small model independently analyzes the data and outputs the sparsification results.

[0061] (4) The verification program compares the outputs of each small model and determines the optimal sparsified dataset according to the loss function (weighted sum of similarity and sparsity).

[0062] (5) The sparsified dataset is stored in the database of the BMC or in a specified file for subsequent data analysis and fault diagnosis.

[0063] In this embodiment, the above actual data can be, but is not limited to, the data of the target sensor actually collected within the current time window, and the duration adjustment parameter S t can be, but is not limited to, determined according to the minimum step size S min , the maximum step size S max For example,

[0064] It should be noted that for sensors with stable values or unimportant sensors, such as voltage or current sensors, etc., a relatively large minimum step size and maximum step size can be set, while for sensors with obvious numerical changes or relatively important sensors, smaller minimum and maximum step sizes are set, and the minimum step size is taken as the initial step size.

[0065] The duration adjustment parameter is used to control the speed of the duration change in the next time window. By the error value between the predicted data and the actual data, the value of the duration adjustment parameter can be changed.

[0066] As Figure 3 shown, when the difference between the predicted value and the actual data d collected within the i-th time window (the previous time window) i is less than δ, the duration adjustment parameter Then it is judged whether the first duration of the previous time window is less than According to the judgment result, the first duration is adjusted to obtain the second duration of the next time window.

[0067] That is to say, during the operation of the server, through the adaptive sampling frequency algorithm, the historical data in the historical time window is continuously obtained, the sensor data in each current time window is predicted, and the sampling step of the next time window is adaptively adjusted based on the predicted data, the actual data, and the duration adjustment parameter, so as to reduce the data acquisition of sensors with relatively stable data and save storage space; on the contrary, if the data of the sensor fluctuates greatly, the first duration can be reduced, that is, the interval duration between two data acquisitions is shortened, so as to more carefully capture the changes in the time series data.

[0068] After continuously adjusting to obtain the duration of the next time window in the above manner, the data collected in the historical time window, the previous time window, and the next time window is obtained, and the data is determined as the first data set. If the number of data in the first data set reaches the data processing condition (for example, more than 1,000 or 10,000 data), the first data set is sparsified to further reduce the data storage space.

[0069] Assume that one data monitored by one sensor is collected within one time window. Then, when the number of data collected within the time window sequence composed of the historical time window, the current time window, and the next time window reaches the target threshold, the data collected within the time window sequence is used as the first data set (which can also be understood as a dense data set), and the first data set is sparsified by using a pre-trained set of network models to obtain the second data set.

[0070] Obviously, the storage space required for the data in the second data set after sparsification is less than the storage space required for the data in the first data set.

[0071] A method combining adaptive sampling and data sparsification proposed in the embodiments of the present application aims to improve the acquisition and storage logic of server hardware status data. By dynamically adjusting the sampling frequency and intelligently screening data, it ensures the sensitivity in anomaly detection and avoids excessive occupation of storage space, providing strong technical support for server operation and maintenance management and data center resource optimization.

[0072] In the above manner, based on the historical data collected within the historical time window, the predicted data monitored by the target sensor within the current time window is predicted, and based on the comparison result of the error value and duration adjustment parameter between the predicted data and the actual data, the duration of the next time window is adjusted in real time to achieve the effect of adaptively adjusting the sampling step. At the same time, by using a set of network models to screen the data in the first data set obtained by adaptive sampling, a second data set strongly associated with the current monitoring task is obtained. In other words, by adopting the method of combining adaptive adjustment of the sampling frequency and data screening, the amount of collected time series data is reduced, the demand for data storage space is reduced, the problem of excessive consumption of storage space for time series data in the related technology is solved, and the technical effect of improving the utilization rate of the storage space of time series data is achieved.

[0073] In an exemplary embodiment, determining the predicted data of the target sensor within the current time window based on the historical data includes: determining a first hyperparameter based on the configuration information of the target sensor, where the first hyperparameter is used to represent the number of partial time windows closest to the current time window in the historical time window; determining partial data collected within partial time windows from the historical data based on the first hyperparameter; and averaging the partial data to obtain the predicted data collected within the current window.

[0074] Among them, the first hyperparameter can but is not limited to representing the number of the closest partial time windows for prediction before the current time window. This is because sensor data has a certain short-term stability and long-term trend, and recent data can better reflect the continuation of the current state. For example, if the first hyperparameter is set to 3, then when predicting the data of the current time window, only the data collected within the 3 historical time windows closest to the current time window needs to be considered to accurately capture the immediate changes in the server hardware status.

[0075] Select partial historical data: According to the number n specified by the first hyperparameter, extract the data within the corresponding time window from the historical data. This part of the data is the basis for determining the predicted data, and its selection directly affects the accuracy of the prediction. For example, if the target sensor is the CPU temperature and the first hyperparameter is set to 3, then the CPU temperature data collected within the nearest 3 time windows is used to predict the temperature change within the next time window.

[0076] Calculate predicted data: Take the average of the selected "partial data" to obtain the average value. And use this average value as the predicted data for the current time window. The method of determining the predicted data by taking the average can not only reduce the influence of noise but also provide a stable basic predicted value.

[0077] The above predicted data will be used to compare with the actually collected data, and whether to adjust the sampling frequency (the time interval between two collections) will be determined according to the magnitude of the error value. For example, if the target sensor monitors the memory voltage, then the average value of this part of the data represents the predicted memory voltage value.

[0078] The predicted data of the target sensor within the current time window obtained by the above method can be used but not limited to subsequent dynamically adjusting the sampling step size, increasing the sampling step size for data with smaller fluctuations, thereby reducing the collection of such data and improving the efficiency of server operation and maintenance and the resource utilization rate of the data center.

[0079] In an exemplary embodiment, obtaining the error value between the predicted data and the actual data, and adjusting the first duration of the current time window to the second duration of the next time window adjacent to the current time window by comparing the error value and the duration adjustment parameter includes: determining the first interval time and the second interval time during the process of collecting the timing data of the target sensor based on the configuration information of the target sensor; determining the duration adjustment parameter based on the first interval time and the second interval time; and adjusting the first duration to obtain the second duration of the next time window by comparing the first duration with the duration adjustment parameter when the error value is less than the second threshold.

[0080] As Figure 4 shown, in the data collection stage, the first stage is to collect sensor data according to the current sampling step corresponding to the current time window, and the second stage is to dynamically adjust the sampling step of the next time window (S old for the next time window (S new ).

[0081] During the second stage, first, according to the characteristics of the target sensor itself, set the first interval time (the minimum step size or the minimum sampling period S min ) and the second interval time (the maximum step size or the maximum sampling period S max ). At the same time, adopt the idea of TCP congestion control to set the slow start threshold (which can also be understood as the duration adjustment parameter).

[0082] Then, by comparing the predicted data and the actual data obtained in the above embodiment, obtain the comparison result, and determine the adjusted second duration according to the comparison result.

[0083] Specifically, as Figure 3 shown, when the comparison result indicates that the error value is less than the second threshold δ, further determine whether the sampling step S old of the current time window is less than If it is less than then increase the sampling step of the next time window, and specifically determine the sampling step of the next time window, that is, the second duration, according to the calculation method in the following embodiments.

[0084] On the contrary, as Figure 3 shown, if the comparison result indicates that the error value is greater than or equal to the second threshold δ, that is, the error between the new data and the predicted value exceeds the allowable range, then S new = S min , and at the same time update the slow start threshold S t to

[0085] Obviously, when the error value is less than the second threshold, it means that the error between the predicted value and the actual data is within the allowable range, and at this time the step size will be increased; on the contrary, the step size will be decreased.

[0086] It should be noted that in addition to using the above error value as the basis for step size change, some additional reference criteria can also be added. For example, the change in the inlet air temperature will cause changes in other temperature sensors and the fan speed. Therefore, if a strong change in the inlet air temperature is detected, the temperature sensor needs to shorten the sampling step.

[0087] In addition, when the power of the server increases significantly, the CPU utilization rate suddenly increases, the memory utilization rate increases significantly, etc., the sampling step of the corresponding sensor will also be reduced to increase the sampling of time series data and obtain refined and valuable status data.

[0088] In this embodiment, by refining the conditions and processes for sampling frequency adjustment, the flexibility and adaptability of sensor data acquisition are ensured. By combining with the above predicted data, it is possible to intelligently adjust the data acquisition frequency according to the real-time changes in the server hardware state, so as to maximize the use efficiency of storage resources while meeting the monitoring requirements. In this way, the operation and maintenance team of the data center can obtain more accurate hardware status information, while reducing the pressure on the storage infrastructure and achieving reasonable allocation and management of resources.

[0089] In an exemplary embodiment, adjusting the first duration by comparing the first duration with a duration adjustment parameter to obtain a second duration for the next time window includes: when the first duration is less than the duration adjustment parameter, determining the product of the first duration and a second hyperparameter as the second duration; when the first duration is greater than or equal to the duration adjustment parameter, determining the sum of the first duration and a third hyperparameter as the second duration.

[0090] As Figure 3 shown, comparing the first duration S old with the optimized duration adjustment parameter If the sampling step size of the current time window then the new step size S of the next time window new = kS old (accelerated growth); conversely, if then the new step size S of the next time window new = S old + m (linear growth).

[0091] It can be seen that when the current sampling period has been increased to a certain extent, the server hardware state may be in a stable state for a long time. To avoid missing abnormal states due to excessive growth of the sampling period, in this embodiment, the sampling period of the next time window is adjusted by multiplying the first duration of the current time window by a third hyperparameter, ensuring data integrity and anomaly detection ability and improving storage efficiency.

[0092] For example, if the first duration is less than the optimized duration adjustment parameter and the first duration is 5 minutes and the second hyperparameter is set to 2, then the second duration of the next time window will be adjusted to 10 minutes, reflecting the rapid growth of the sampling step size in a stable state.

[0093] Conversely, if the first duration is greater than or equal to the optimized duration adjustment parameter and the first duration is 5 minutes and the third hyperparameter is set to 3, then the second duration of the next time window will be 8 minutes, reflecting the slow growth of the sampling step size in a stable state.

[0094] That is to say, by comparing the first duration with the optimized duration adjustment parameter and combining the second hyperparameter and the third hyperparameter, the change rate of the sampling step size of the next time window, or the speed of change of the sampling step size, is determined.

[0095] In addition, when the difference between the actual data and the predicted data exceeds the allowable range, the second duration of the next time window is adjusted to S new = S min , and the slow start threshold S t is updated to

[0096] However, it should be noted that when increasing the duration of the next time window according to the above growth strategy, if the increased step size reaches S max , the growth stops.

[0097] By adopting the above method, the current state of the server can be analyzed based on the data within the historical period. When the server state is relatively stable, the duration of the next time window can be quickly increased, avoiding storing all the data in the stable state unchanged and wasting a large amount of storage space. At the same time, it can also reduce the probability of events where abnormal situations occurring in a short period may not be recorded.

[0098] In addition, when it is analyzed that the current state of the server is relatively stable, the duration of the next time window can be slowly increased, the data acquisition step size can be appropriately slowed down, and the acquisition of non-critical data can also be reduced, thereby reducing the acquisition of redundant data and achieving the effect of saving storage space.

[0099] In an exemplary embodiment, the above-mentioned use of a group of network models to screen the data in the first data set to obtain a second data set includes: when a group of network models includes N learning models, by inputting the first data set into the N learning models respectively, sparsifying the first data set to obtain N sparse data sets, where N is a positive integer greater than or equal to 2, and the N learning models are trained using a pre-trained target network model as the teacher model, and the target network model has the same structure as the N learning models but different structural parameters; based on the data included in each sparse data set, screening out the target data set from the N sparse data sets, where the target data set is the second data set.

[0100] When the number of data points collected within the historical time window, the current time window, and the next time window reaches the target threshold (for example, more than 100 or 1000) according to the above adaptive acquisition algorithm, all the data collected in the time window sequence is used as the first data set (which can also be understood as dense data), and the first data set is input into the Figure 5 model cluster shown to obtain a group of sparse data sets.

[0101] By screening a group of sparse data sets, a target sparse data set is obtained, and the target sparse data set is determined as the second data set to store the second data set in the target file or the database of the BMC.

[0102] It should be noted that the group of network models in the above model cluster can be but not limited to the Encoder-Decoder framework, and this framework includes but is not limited to Figure 6The shown recurrent neural network structure (LSTM structure) and attention layer (Attention layer), where the LSTM structure is used to extract features from the input data, and then the Attention layer is used to locate the key time points in the time series data, so as to realize the sparsification processing of the input data (the first data set).

[0103] The above set of network models can be, but are not limited to, smaller network models trained with a large deep learning model as the teacher model, and the large deep learning model has the same model structure as each network model in the set of network models, but different structural parameters. In other words, the structural complexity of each network model in the set of network models is lower than that of the large deep learning model.

[0104] Each model in the set of network models will take the first data set (i.e., the sensor data collected after adaptive sampling frequency adjustment) as input and independently perform sparsification processing. Since the N learning models are trained by the same teacher model, they have the same network structure. However, due to the independent adjustment of network parameters during training, each learning model will generate a unique "sparse data set". These sparse data sets can be, but are not limited to, generated based on the deep learning Encoder-Decoder framework, where the encoder uses LSTM for feature extraction, and the decoder contains an Attention mechanism to locate the key change points in the time series data, so as to achieve the effect of data sparsification.

[0105] After sparsifying the first data set according to each network model in the above set of network models, sparse data sets corresponding to each network model are obtained. Obviously, the number of data points in the sparse data set is less than the number of data points in the first data set.

[0106] Finally, according to the data contained in each sparse data set, the target data set is selected from the N sparse data sets.

[0107] After selecting the target data set from the above multiple sparse data sets, it is determined as the second data set, that is, determined as the object for storage and further analysis. Due to the effective sparsification processing, the occupation of storage space will be greatly reduced, while maintaining the sensitivity to the changes in the server hardware state and the integrity of the records.

[0108] In this embodiment, by using N learning models, it is ensured that the system can still complete efficient data sparsification processing under different computing resource conditions. Even if the scale of a single model is limited, through teacher-student training and multi-model screening, it is still possible to output high-quality data sets that meet the storage and analysis requirements.

[0109] Adopting the technical solution in this embodiment, through multi-model parallel processing and the teacher-student training method, a flexible and efficient implementation path is provided for data sparsification processing in server monitoring. By selecting the optimal sparse data set, not only can the consumption of storage space be significantly reduced, but also the high value and high reliability of the data can be maintained. The implementation of this solution not only improves the intelligent level of data processing, but also enhances the adaptability and robustness of the system in the face of diverse BMC hardware resources.

[0110] In an exemplary embodiment, screening out the target data set from N sparse data sets based on the data included in each sparse data set includes: obtaining the number of data included in each sparse data set to obtain N values; determining the sparse data set corresponding to the smallest value among the N values as the target data set; or based on the time series information of the data in each sparse data set, determining N similarities between the output data in each sparse data set and the input data in the first data set at the same time point; determining the target similarity with the largest value from the N similarities, and determining the sparse data set corresponding to the target similarity as the target data set; or obtaining the number of data included in each sparse data set to obtain N values; and based on the time series information of the data in each sparse data set, determining N similarities between the output data in each sparse data set and the input data in the first data set at the same time point; based on the N values and the N similarities, determining the sparse data set with the smallest number of data and the highest similarity between the output data and the input data from the N sparse data sets; determining the sparse data set with the smallest number of data and the highest similarity between the output data and the input data as the target data set.

[0111] This embodiment further refines the method of screening out the target data set (i.e., the second data set) from N sparse data sets, and provides three different strategies in total to ensure that the selected data set not only performs excellently in terms of sparsity, but also meets the highest standards in terms of the integrity of data retention and the accuracy of time series information. These three strategies are the minimum value selection based on the number of data, the maximum value selection based on similarity, and the strategy of comprehensive evaluation by combining the two.

[0112] First, obtain the number of data points included in each sparse data set to form a list of N values. Then, select the minimum value among these values, and use the corresponding data set as the target data set. This strategy gives priority to the sparsity of the data set and aims to minimize the storage requirements.

[0113] Secondly, selection based on the maximum similarity: This strategy focuses on the similarity between the output sparse dataset and the input dataset. By calculating the similarity between the output data and the input data of each sparse dataset and the first dataset at the same time point, the maximum similarity value is found and used as the basis for dataset selection. The calculation of similarity is usually based on the loss function of the deep learning model, such as the mean square error (MSE) and mean absolute error (MAE) calculated by the Bessel curve, etc., to ensure that while the target dataset is sparsified, the representative trend and key information of the original dataset are maintained.

[0114] Finally, considering the above two selection considerations, in this embodiment, a comprehensive evaluation strategy is also provided, that is, the sparse dataset with the fewest data points and the highest similarity is determined as the final target dataset. The execution of this strategy usually requires an evaluation function that can convert the two metrics of the number of data and similarity into a single evaluation criterion. For example, a weighted sum function can be defined to multiply the reciprocal of the number of data points by the similarity value, and then the sparse dataset corresponding to the maximum value is taken as the target dataset.

[0115] After determining the optimal sparse dataset through one of the above strategies, it replaces the first dataset and becomes the second dataset for storage and subsequent analysis. The generation of this dataset not only reflects the powerful ability of the deep learning model in data sparsification processing but also ensures the efficient use of resources and the maintenance of data integrity.

[0116] Through the technical solution in this embodiment, the most suitable sparse dataset screening method can be flexibly selected according to the specific application scenario and resource constraints, enabling the system to minimize the storage cost and consumption of computing resources while ensuring the data value. Whether in a resource-constrained environment or a scenario that requires high-precision data recording, the best data processing solution can be found through the implementation of the above strategies, significantly improving the intelligent level of server monitoring and data center operation and maintenance.

[0117] In an exemplary embodiment, before the above method of screening the data in the first dataset by using a set of network models to obtain the second dataset, the method further includes: creating a training dataset, where the data in the training dataset is the data collected by the target sensor according to a preset duration, and the preset duration is less than the first duration and the second duration; using the training dataset to train the deep learning model to be trained to obtain the target network model; using the target network model as the teacher model and the set of network models to be trained as the student model to train the set of network models to be trained to obtain the trained set of network models, where the target network model has the same structure as the set of network models.

[0118] As described in the above embodiments, to ensure the sparsification effect, a relatively large model is used to train a set of network models in this embodiment. However, due to the limitation of hardware resources in the BMC, the relatively large model cannot be deployed. Therefore, it is considered to embed several small models (for example, a set of network models) in the BMC, and the number and scale of the models used can be automatically adjusted according to the BMC hardware resources. These small models analyze the dense data and output sparse data points. Then, the built-in processing programs in the BMC respectively verify the outputs of these small models, extract the optimal output as the result (the second data set), and store the data in the second data set into a file or a database.

[0119] The above method of using the Teacher-Student method to train small models means using a relatively large model that has been fully trained and has good performance to guide the training of another or multiple smaller models. The advantage of this training method is that based on the already trained large model, several small models of different scales can be output in a short time to meet the requirements under different computing resources.

[0120] As Figure 6 shown, before using the pre-trained large model to train a set of network models, first use the created training data set to train the large model (which can also be understood as the deep learning model to be trained).

[0121] It should be noted that the above training data set is different from the first data set obtained by using the above adaptive sampling algorithm. The sampling step used when sampling the data in this sampling training data set is relatively small, and may even be smaller than the above minimum step S min , for example, sample once every 1 second, and for each sensor, only sample 1 data each time.

[0122] The process of using the above training data set to train the deep learning model to be trained may last for several hours or even several days, which depends on the size of the training data set and the complexity of the model. Through repeated iterations, the large model gradually learns the strategy of effectively sparsifying the time-series data and forms the target network model.

[0123] After training the target network model, use it as the teacher model. This target network model is not only in the state with the optimal structural parameters, but also contains rich data processing experience and can effectively guide the training of the student model.

[0124] Using the target network model as the teacher model, a group of network models to be trained (i.e., student models) are trained. The student models are structurally identical to the teacher model but have different parameters. The training process involves inputting the training dataset into the student models multiple times, and through algorithms such as gradient descent, gradually adjusting the network parameters to sparsify the data while retaining the key information. The role of the teacher model is to provide the direction and criteria for parameter updates, helping the student models to approximate the performance of the teacher model as closely as possible under limited resources.

[0125] After training, the model group composed of N student models will form a set of smaller-scale but diverse-capability deep learning models. These models can be deployed to the restricted environment of the BMC for actual data sparsification processing.

[0126] In the above manner, not only is a powerful target network model trained, but also a group of network models adapted to the BMC hardware environment is successfully trained. The integration and use of this group of models not only solve the problem of big data storage but also maintain high data processing quality and efficiency under various hardware conditions, demonstrating the great potential of deep learning in the field of server monitoring.

[0127] In addition, the teacher-student model training strategy ensures that even under limited BMC resources, a batch of small network models with strong sparsification processing capabilities can be trained, providing an effective and flexible data management solution for the operation and maintenance of the data center.

[0128] In an exemplary embodiment, the above-mentioned training of the deep learning model to be trained using the training dataset to obtain the target network model includes: drawing a first data curve based on the temporal information and numerical values of each data in the training dataset; obtaining a set of data features by inputting the first data curve into the recurrent neural network structure in the deep learning model to be trained; determining the key time points related to the target monitoring task from a series of consecutive time points indicated by the first data curve by inputting the set of data features into the attention layer in the deep learning model to be trained; obtaining a second data curve output by the deep learning model to be trained based on the set of data corresponding to the key time points, where the training dataset includes a set of data; determining the target loss function of the deep learning model to be trained based on the first data curve and the second data curve; and stopping the training when the value of the target loss function satisfies the training end condition, and determining the deep learning model to be trained after stopping the training as the target network model.

[0129] In this embodiment, the training of the deep learning model to be trained can be achieved but is not limited to the following methods:

[0130] S21. Based on the training dataset, plot the first data curve;

[0131] Based on the time series information (acquisition time) and specific sensor values of each data in the training dataset, plot the first data curve. Specifically, you can refer to Figure 6 the input curve on the left. This curve intuitively represents the change trend of time series data and provides a basis for subsequent feature extraction and key time point location.

[0132] In this embodiment, the first data curve can be but is not limited to a Bezier curve. A Bezier curve is a parametric curve widely used in computer graphics to generate smooth curve paths. It defines the shape of the curve through a set of control points and has high flexibility and controllability. In computer graphics, Bezier curves are usually drawn through the de Casteljau algorithm, which generates points on the curve through recursive linear interpolation.

[0133] S22. Use the LSTM structure in the deep learning model to extract features from the data points in the first data curve;

[0134] Specifically, by inputting the first data curve into the deep learning model to be trained, use a special recurrent neural network (LSTM) structure to deeply analyze the data and obtain a set of data features. The LSTM structure can capture the long-term dependencies of sequential data and is an ideal tool for processing time series data.

[0135] S23. Use the Attention layer in the deep learning model to further analyze the data features extracted by the LSTM structure;

[0136] For example, determine the key time points related to the target monitoring task from a continuous number of time points indicated by the first data curve. The Attention layer can enable the model to focus on the important parts in the time series and plays a key role in improving the processing efficiency and accuracy of the model.

[0137] S24. Based on a set of data corresponding to the key time points, generate the second data curve;

[0138] Based on the data corresponding to the above key time points, the model outputs the second data curve after sparsification processing, specifically as Figure 6 the data curve on the right as shown. This data curve contains the key information in the first data curve but has been optimized in terms of data point density to reduce storage requirements.

[0139] S25. Based on the first data curve and the second data curve, calculate the value of the loss function during the model training process and determine whether the training stop condition is met according to the value of the loss function.

[0140] During the iterative training process of the model, the above steps S21 - S25 are repeated until the value of the loss function meets the preset training termination condition. After the training is completed, the target network model should be able to effectively sparsify the time series data while retaining the representative trends and key abnormal events of the data.

[0141] By using the above model training method, not only can a deep learning model capable of processing sensor data be trained, but also the efficiency and accuracy of the model in processing data are ensured. This model is not only applicable to server monitoring but also can be extended to other time series data processing fields, providing strong technical support for data analysis and storage in the big data era.

[0142] In an exemplary embodiment, determining the target loss function of the deep learning model to be trained based on the first data curve and the second data curve includes: based on the time series information of each point in the first data curve, finding a set of predicted output values corresponding to the points in the second data curve with the same time series information; based on a set of input values and a set of predicted output values corresponding to the time series information of each point in the first data curve, determining the mean squared error, where the mean squared error is used to represent the curve similarity between the first data curve and the second data curve; based on a set of predicted output values and a set of input values, determining the mean absolute error, where the mean absolute error is used to represent the result of sparsifying the training data set using the deep learning model to be trained; performing a weighted sum of the mean squared error and the mean absolute error to obtain the value of the target loss function.

[0143] During the training process of the above deep learning model, by setting the training task corresponding to the target loss function, it can be determined whether the performance of the deep learning model (large model) meets the preset requirements. Among them, the value of the loss function includes but is not limited to two parts, the first part is the similarity, and the second part is the sparsity.

[0144] Among them, the calculation of the similarity can be but is not limited to using the Bezier curve used for continuous discrete points on the web page. The mean squared error MSE calculated between the input data set (the first data set) and its corresponding points on the Bezier curve is used as the measurement standard for the similarity.

[0145] For example, assume that there are 100 data points in the first data set. Through the input Figure 6 shown deep learning model, using the processing of the LSTM and Attention layers, 20 key time points are obtained. According to these 20 key time points, the input data curve (the second data curve) is drawn. That is to say, the purpose of using the Attention layer is to draw an output curve as similar as possible to the first data curve using as few data points as possible.

[0146] Obtain the coordinates (x1, y1) to (x 100 , y 100 ) corresponding to 100 data points in the first dataset, and then determine the y' 100 corresponding to x i from the second data curve according to the abscissas x1 to x i . A total of y'1 to y' 100 are obtained. According to y1 to y 100 and y'1 to y' 100 , calculate the mean squared error MSE. MSE not only measures the deviation between the model's predicted values and the actual values but also reflects the model's ability to maintain the data trend. The values on the x-axis represent time series information, such as a time point, and the values on the y-axis represent sensor data collected at a time point.

[0147] The measurement of sparsity is carried out using the mean absolute error MAE, that is, the second part of the loss function L1 shown in Figure 6 . The value of MAE is also calculated using the above y1 to y 100 and y'1 to y' 100 . MAE directly reflects the accuracy of the model in data sparsification processing, that is, the faithfulness of the model to the original data when reducing the number of data points.

[0148] Obviously, it is easy to understand that the better the model performance, the closer the shape of the output second data curve is to the first data curve, and the closer the values (y i and y' i ) of the respective data points corresponding to the same time point (x i ) on the two data curves are.

[0149] The value of the target loss function can be determined by but is not limited to the weighted sum of the above mean squared error and mean absolute error. For example, L = MSE + αL1. According to the value of the target loss function, determine whether the deep learning model meets the condition for stopping training, and determine the model at the end of training as the target network model.

[0150] Additionally, since the N learning models in the group of network models mentioned in the above embodiments are trained under the guidance of the deep learning model in this embodiment, the process of processing the Figure 5 -shown dense dataset by the N learning models is the same as the data processing process of the deep learning model.

[0151] For example, when Figure 5When there are N learning models in the dense dataset input model cluster as shown, the Attention layer in each learning model (small model) is used to locate N groups of key time points, and then according to each group of key time points, an output data curve corresponding to each learning model (i.e., N second data curves) is drawn.

[0152] Using each group of key time points in the N groups of key time points, a group of data points can be filtered out from the first dataset, and each group of data points is determined as a sparse dataset, so that N sparse datasets determined by N learning models can be obtained. Then, the target dataset is determined based on the number and / or similarity of data points in the N sparse datasets as mentioned in the above embodiments, thereby obtaining the second dataset.

[0153] In the same way, when Figure 6 shown deep learning model is used to train the N learning models, the calculation of the loss function is the same as the calculation method of the target loss function in the process of using the training dataset to train the to-be-trained deep learning model, and it is also obtained by weighted summation of similarity and sparsity.

[0154] Through the above method, not only can the training effect of the deep learning model be evaluated, but also the optimization of model parameters can be guided, and finally a high-quality model that can maintain trend similarity and achieve efficient data sparsification in time series data processing can be produced. This not only improves the efficiency of data storage in the BMC system, but also provides a more refined solution for server monitoring and other time series data processing scenarios.

[0155] To more clearly understand the technical solution of the present application, the above data processing method will be further explained below in combination with Figure 7 the overall schematic diagram shown.

[0156] As Figure 7 shown, the data processing process includes a data acquisition stage and a data processing stage. Among them, the data acquisition stage includes two parts: collecting sensor data according to the current step size and adjusting the step size of the next time window according to the newly collected data.

[0157] When adjusting the step size of the next time window according to the newly collected data, assume that step size 2 is the current time window, and according to the characteristics of the target sensor, select the data collected in some time windows within the historical period to predict the data that may be collected by step size 2, obtain the predicted data, and determine the error value between the predicted data and the actual data actually collected within step size 2.

[0158] When the error value is within the allowable range, increase the duration of step size 3, otherwise decrease the duration of step size 3. The increase and decrease of the step size adopt the idea of TCP congestion control, and set the slow start threshold

[0159] As Figure 3 shown, when the comparison result indicates that the error value is less than the second threshold δ, further determine whether the sampling step S of the current time window old is less than If it is less than then increase the sampling step of the next time window, and specifically determine the sampling step of the next time window, that is, the second duration, according to the calculation method in the following embodiments.

[0160] Conversely, as Figure 3 shown, if the comparison result indicates that the error value is greater than or equal to the second threshold δ, that is, the error between the new data and the predicted value exceeds the allowable range, then S new = S min , and at the same time update the slow start threshold S t to

[0161] Obviously, when the error value is less than the second threshold, it means that the error between the predicted value and the actual data is within the allowable range, and at this time the step size will be increased; conversely, the step size will be decreased.

[0162] After adjusting the duration of the step size 3 (the third time window) according to the above method, the collected data 3 is obtained. In the same way, adaptively adjust the duration of each subsequent time window, and when the number of the collected data meets the target processing condition, determine the data collected within the time window sequence as the dense data set.

[0163] Input the dense data set into a group of pre-trained learning models, and use each learning model to sparsify the data in the dense data set to obtain N sparse data sets. By using the number and / or similarity of data points in the N sparse data sets mentioned in the above embodiments to determine the target data set, the second data set is obtained.

[0164] Adopting the technical solution provided by the present application has at least the following beneficial effects:

[0165] 1. Improve the anomaly detection ability: Through adaptive sampling, the sampling frequency can be quickly adjusted when the sensor data changes significantly, ensuring that short-term abnormal situations are captured, and enhancing the recording ability of transient events.

[0166] 2. Save storage space: Adopt a compression strategy for stable data, only save the key change points of the data, and through the sparsification process of the deep learning model, the storage space required is greatly reduced, and the data storage efficiency is optimized.

[0167] 3. Adaptive enhancement: Dynamically adjust the sampling strategy according to the characteristics of sensor data, set different sampling parameters for sensors with different importance and stability, and improve the pertinence and efficiency of data collection.

[0168] 4. Efficient use of resources: In the face of BMC hardware resource limitations, through the small model cluster and teacher-student training method, an effective data processing model is deployed with limited resources while maintaining the quality of data processing.

[0169] 5. Balance data integrity and economy: While reducing storage costs, maintain the representativeness of data through a carefully designed algorithm, ensure that the data can still reflect the trends and important changes of the original data after compression, and maintain the accuracy of data analysis.

[0170] Through the description of the above implementation manners, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. However, in many cases, the former is a better implementation manner.

[0171] According to another aspect of the embodiments of the present application, a data processing device is further provided. The data processing device can be used to implement the data processing method provided in the above embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0172] Figure 8 is a structural block diagram of an optional data processing device according to the embodiments of the present application. As Figure 6 shown in, the data processing device includes a first acquisition unit 802, a first processing unit 804, a second acquisition unit 806, a creation unit 808, and a second processing unit 810.

[0173] The first acquisition unit 802 is configured to acquire historical data collected within a historical time window during the operation of the server, where the historical data is data monitored by a target sensor and used to describe the hardware state information of the server.

[0174] The first processing unit 804 is configured to determine the predicted data of the target sensor within the current time window based on the historical data.

[0175] A second acquisition unit 806, configured to obtain an error value between predicted data and actual data, and adjust a first duration of a current time window to a second duration of a next time window adjacent to the current time window by comparing the error value with a duration adjustment parameter, where the duration adjustment parameter is used to represent a duration change rate of the next time window.

[0176] A creation unit 808, configured to create a first data set based on data collected within a historical time window, the current time window, and the next time window.

[0177] A second processing unit 810, configured to use a set of network models to screen data in the first data set to obtain a second data set, where each network model in the set of network models includes a recurrent neural network structure and an attention layer.

[0178] It should be noted that the first acquisition unit 802 in this embodiment may be used to execute the above step S202, the first processing unit 804 in this embodiment may be used to execute the above step S204, the second acquisition unit 806 in this embodiment may be used to execute the above step S206, the creation unit 808 in this embodiment is used to execute the above step S208, and the second processing unit 810 in this embodiment may be used to execute the above step S210.

[0179] Through the embodiment provided in this application, according to historical data collected within a historical time window, predicted data monitored by a target sensor within a current time window is predicted, and based on a comparison result between an error value between the predicted data and the actual data and a duration adjustment parameter, the duration of the next time window is adjusted in real time, achieving the effect of adaptively adjusting a sampling step. At the same time, by using a set of network models to screen data in the first data set obtained by adaptive sampling, a second data set strongly associated with the current monitoring task is obtained. In other words, by adopting a combination of adaptive sampling frequency adjustment and data screening, the amount of collected time-series data is reduced, the requirement for data storage space is lowered, the problem of excessive storage space consumption of time-series data in the related art is solved, and the technical effect of improving the storage space utilization rate of time-series data is achieved.

[0180] In an exemplary embodiment, the above first processing unit 804 includes: a first processing module, configured to determine a first hyperparameter based on configuration information of a target sensor, where the first hyperparameter is used to represent the number of partial time windows closest to the current time window in the historical time window; a second processing module, configured to determine partial data collected within partial time windows from historical data based on the first hyperparameter; and a third processing module, configured to average the partial data to obtain predicted data collected within the current window.

[0181] In an exemplary embodiment, the second acquisition unit 806 described above includes: a fourth processing module configured to determine a first interval time and a second interval time during the process of acquiring the timing data of the target sensor based on the configuration information of the target sensor; a fifth processing module configured to determine a duration adjustment parameter based on the first interval time and the second interval time; a first adjustment module configured to, when the error value is less than the second threshold, adjust the first duration by comparing the first duration with the duration adjustment parameter to obtain a second duration for the next time window.

[0182] In an exemplary embodiment, the first adjustment module described above includes: a first processing sub-module configured to, when the first duration is less than the duration adjustment parameter, determine the product of the first duration and a second hyperparameter as the second duration; a second processing sub-module configured to, when the first duration is greater than or equal to the duration adjustment parameter, determine the sum of the first duration and a third hyperparameter as the second duration.

[0183] In an exemplary embodiment, the second processing unit 810 described above includes: a sixth processing module configured to, when a set of network models includes N learning models, sparsify a first data set by inputting the first data set into the N learning models respectively to obtain N sparse data sets, where N is a positive integer greater than or equal to 2, and the N learning models are trained by using a pre-trained target network model as a teacher model, and the target network model has the same structure as the N learning models but different structural parameters; a screening module configured to screen out a target data set from the N sparse data sets based on the data included in each sparse data set, where the target data set is a second data set.

[0184] In an exemplary embodiment, the above-mentioned screening module includes: a third processing sub-module, configured to obtain the number of data included in each sparse data set to obtain N values; determine the sparse data set corresponding to the smallest value among the N values as the target data set; or based on the timing information of the data in each sparse data set, determine N similarities between the output data in each sparse data set and the input data in the first data set at the same time point; determine the target similarity with the largest value from the N similarities, and determine the sparse data set corresponding to the target similarity as the target data set; or obtain the number of data included in each sparse data set to obtain N values; and based on the timing information of the data in each sparse data set, determine N similarities between the output data in each sparse data set and the input data in the first data set at the same time point; based on the N values and the N similarities, determine the sparse data set with the smallest number of data and the highest similarity between the output data and the input data from the N sparse data sets; determine the sparse data set with the smallest number of data and the highest similarity between the output data and the input data as the target data set.

[0185] In an exemplary embodiment, the above-mentioned device further includes: a creation unit, configured to create a training data set before screening the data in the first data set by using a set of network models to obtain a second data set, where the data in the training data set is data collected by a target sensor according to a preset duration, and the preset duration is less than the first duration and the second duration; a first training unit, configured to use the training data set to train a deep learning model to be trained to obtain a target network model; a second training unit, configured to use the target network model as a teacher model and a set of network models to be trained as student models to train the set of network models to be trained to obtain a trained set of network models, where the target network model has the same structure as the set of network models.

[0186] In an exemplary embodiment, the above-mentioned first training unit includes: a first drawing module, configured to draw a first data curve based on the timing information and numerical values of each data in the training dataset; a seventh processing module, configured to obtain a set of data features by inputting the first data curve into a recurrent neural network structure in the deep learning model to be trained; an eighth processing module, configured to determine, from a plurality of consecutive time points indicated by the first data curve, key time points related to the target monitoring task by inputting the set of data features into an attention layer in the deep learning model to be trained; a ninth processing module, configured to obtain a second data curve output by the deep learning model to be trained based on a set of data corresponding to the key time points, where the training dataset includes a set of data; a tenth processing module, configured to determine a target loss function of the deep learning model to be trained based on the first data curve and the second data curve; an eleventh processing module, configured to stop training when the value of the target loss function satisfies the training end condition, and determine the deep learning model to be trained after stopping training as the target network model.

[0187] In an exemplary embodiment, the above-mentioned tenth processing module includes: a search sub-module, configured to search, based on the timing information of each point in the first data curve, for a set of predicted output values corresponding to the points with the same timing information in the second data curve; a fourth processing sub-module, configured to determine a mean square error based on a set of input values corresponding to the timing information of each point in the first data curve and the set of predicted output values, where the mean square error is used to represent the curve similarity between the first data curve and the second data curve; a fifth processing sub-module, configured to determine an average absolute error based on the set of predicted output values and the set of input values, where the average absolute error is used to represent the result of sparsifying the training dataset using the deep learning model to be trained; a sixth processing sub-module, configured to perform a weighted sum of the mean square error and the average absolute error to obtain the value of the target loss function.

[0188] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited thereto: the above-mentioned modules are all located in the same processor; or, the above-mentioned various modules are respectively located in different processors in any combination form.

[0189] According to another aspect of the embodiments of the present application, an electronic device is further provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above-mentioned data processing method embodiments.

[0190] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data processing method embodiments when running.

[0191] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), external hard drives, magnetic disks, or optical discs that can store computer programs.

[0192] According to another aspect of the embodiments of the present application, there is also provided a computer program product, the computer program product including a computer program that, when executed by a processor, implements the steps in any of the above-described data processing method embodiments.

[0193] The embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described data processing method embodiments.

[0194] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled artisans can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0195] The above has introduced in detail a data processing method provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data processing method, characterized in that, comprising: During the operation of the server, obtaining historical data collected within a historical time window, where the historical data is data monitored by a target sensor and used to describe the hardware status information of the server; Based on the historical data, determining predicted data of the target sensor within a current time window; Obtaining an error value between the predicted data and actual data, and adjusting a first duration of the current time window to a second duration of a next time window adjacent to the current time window by comparing the error value and a duration adjustment parameter, where the duration adjustment parameter is used to represent a duration change rate of the next time window; Based on the data collected within the historical time window, the current time window, and the next time window, creating a first data set; Using a set of network models to screen the data in the first data set to obtain a second data set, where each network model in the set of network models includes a recurrent neural network structure and an attention layer.

2. The method according to claim 1, characterized in that, The determining the predicted data of the target sensor within the current time window based on the historical data includes: Based on the configuration information of the target sensor, determining a first hyperparameter, where the first hyperparameter is used to represent the number of partial time windows in the historical time window that are closest to the current time window in terms of time; Based on the first hyperparameter, determining partial data collected within the partial time windows from the historical data; Taking an average of the partial data to obtain the predicted data collected within the current window.

3. The method according to claim 1, characterized in that, The obtaining the error value between the predicted data and actual data, and adjusting the first duration of the current time window to the second duration of the next time window adjacent to the current time window by comparing the error value and the duration adjustment parameter includes: Based on the configuration information of the target sensor, determining a first interval time and a second interval time during the process of collecting the time series data of the target sensor; Based on the first interval time and the second interval time, determining the duration adjustment parameter; In a case where the error value is less than a second threshold, adjusting the first duration by comparing the first duration with the duration adjustment parameter to obtain the second duration of the next time window.

4. The method according to claim 3, characterized in that, The adjusting the first duration by comparing the first duration with the duration adjustment parameter to obtain the second duration of the next time window includes: In a case where the first duration is less than the duration adjustment parameter, determining a product of the first duration and a second hyperparameter as the second duration; In a case where the first duration is greater than or equal to the duration adjustment parameter, determining a sum of the first duration and a third hyperparameter as the second duration.

5. The method according to claim 1, characterized in that, Using a set of network models to screen the data in the first data set to obtain a second data set, including: When the set of network models includes N learning models, by inputting the first data set into the N learning models respectively, sparsifying the first data set to obtain N sparse data sets, where N is a positive integer greater than or equal to 2, and the N learning models are trained with a pre-trained target network model as the teacher model, and the target network model has the same structure as the N learning models but different structural parameters; Based on the data included in each sparse data set, screening out a target data set from the N sparse data sets, where the target data set is the second data set.

6. The method according to claim 5, wherein The screening out a target data set from the N sparse data sets based on the data included in each sparse data set includes: Obtaining the number of data included in each sparse data set to obtain N values; Determining the sparse data set corresponding to the smallest value among the N values as the target data set; or Based on the time series information of the data in each sparse data set, determining N similarities between the output data in each sparse data set and the input data in the first data set at the same time point; Determining the target similarity with the largest value from the N similarities, and determining the sparse data set corresponding to the target similarity as the target data set; or Obtaining the number of data included in each sparse data set to obtain N values; and based on the time series information of the data in each sparse data set, determining N similarities between the output data in each sparse data set and the input data in the first data set at the same time point; Based on the N values and the N similarities, determining from the N sparse data sets the sparse data set with the fewest number of data and the highest similarity between the output data and the input data; Determining the sparse data set with the fewest number of data and the highest similarity between the output data and the input data as the target data set.

7. The method according to any one of claims 1 to 6, wherein Before using a set of network models to screen the data in the first data set to obtain a second data set, the method further includes: Creating a training data set, where the data in the training data set is the data monitored by the target sensor collected at a preset duration, and the preset duration is less than the first duration and the second duration; Using the training data set to train a deep learning model to be trained to obtain a target network model; Using the target network model as the teacher model and a set of network models to be trained as the student model to train the set of network models to be trained to obtain the trained set of network models, where the target network model has the same structure as the set of network models.

8. The method according to claim 7, wherein Using the training data set to train a deep learning model to be trained to obtain a target network model, including: Based on the time series information and numerical values of each data in the training data set, draw a first data curve; By inputting the first data curve into a recurrent neural network structure in the deep learning model to be trained, obtain a set of data features; By inputting the set of data features into an attention layer in the deep learning model to be trained, determine key time points related to the target monitoring task from a continuous plurality of time points indicated by the first data curve; Based on a set of data corresponding to the key time points, obtain a second data curve output by the deep learning model to be trained, where the training data set includes the set of data; Based on the first data curve and the second data curve, determine the target loss function of the deep learning model to be trained; When the value of the target loss function satisfies the training end condition, stop training, and determine the deep learning model to be trained after stopping training as the target network model.

9. The method according to claim 8, wherein The determining the target loss function of the deep learning model to be trained based on the first data curve and the second data curve includes: Based on the time series information of each point in the first data curve, find a set of predicted output values corresponding to the points in the second data curve having the same time series information; Based on a set of input values corresponding to the time series information of each point in the first data curve and the set of predicted output values, determine the mean square error, where the mean square error is used to represent the curve similarity between the first data curve and the second data curve; Based on a set of predicted output values and the set of input values, determine the mean absolute error, where the mean absolute error is used to represent the result of sparsifying the training data set using the deep learning model to be trained; Perform weighted summation on the mean square error and the mean absolute error to obtain the value of the target loss function.

10. An electronic device, characterized in that Comprising: A memory for storing a computer program; A processor for implementing the steps of the data processing method according to any one of claims 1 to 9 when executing the computer program.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 9.

Images