A data processing method, device, equipment and medium

Abstract

The present application provides a data processing method, comprising: a first device acquires multiple data, then the first device obtains a target fitting model based on the multiple data, and then the first device sends the target fitting model to a second device, where the target fitting model is used to restore at least one of the multiple data. In this way, the amount of data transmitted and the amount of data stored by the second device can be reduced, thereby alleviating data transmission pressure and data storage pressure.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the advent of the information age, massive amounts of data have also been generated. In order to make full use of the massive amounts of data, it is often necessary to transmit and store the data. Taking the Internet of things (IoT) as an example, more and more devices such as TVs, air conditioners, speakers, routers, cameras, etc. can be independently addressed to form an interconnected network, thereby realizing intelligent identification, positioning, tracking, monitoring and management of devices.

[0003] In order to achieve intelligent perception, identification, positioning, tracking, monitoring or management, IoT devices generate a large amount of time series data. The so-called time series data refers to indicator data that is statistically analyzed over time. The time series data can be resource utilization, electrocardiogram, stock price, etc. In actual applications, the amount of time series data generated by IoT devices per minute can reach hundreds of millions, which leads to high pressure on data transmission and storage.

[0004] Based on this, the industry urgently needs to provide a data processing method to alleviate the pressure of data transmission and storage. Summary of the invention

[0005] The present application provides a data processing method, which achieves data compression by fitting a model to the data, thereby solving the problem of high data transmission and storage pressure. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

[0006] In the first aspect, the present application provides a data processing method. Specifically, a first device as a data sender can obtain multiple data, and then obtain a target fitting model based on the multiple data, and then send the above target fitting model to a second device, instead of directly sending multiple data, the second device can restore at least one of the multiple data through the above target fitting model as required. In this way, the amount of data transmitted and the amount of data stored by the second device can be reduced, thereby alleviating the pressure of data transmission and data storage.

[0007] In some possible implementations, the first device may obtain the target fitting model in the following manner: Specifically, the first device may perform model fitting on multiple fitting models according to multiple data, so as to select a fitting model from the multiple fitting models as the target fitting model.

[0008] The multiple fitting models may be models pre-negotiated by the first device and the second device, and the first device may use the multiple data as training data to train the multiple fitting models until a training end condition is met. The training end condition may be that the number of parameter iterations reaches a maximum number, or that a loss value determined based on a loss function is less than a preset value. After the training is completed, the first device determines a target fitting model based on the loss values ​​of each fitting model.

[0009] In specific implementation, the first device may use the fitting model with the smallest loss value as the target fitting model, or use the fitting model with a loss value less than a preset value as the target fitting model, or use the fitting model with a convergent loss value as the target fitting model.

[0010] In some possible implementations, the multiple fitting models may specifically be at least two of a linear model, a polynomial model, and a neural network model. Considering the applicable scope of the model, the multiple fitting models may include a linear model, a polynomial model, and a neural network model. In this way, by fitting more models, the fitting accuracy can be improved, which is conducive to improving the accuracy of data transmission.

[0011] In some possible implementations, when the first device sends the target fitting model to the second device, it may send the identifier of the fitting model selected by the first device and the corresponding model parameters to the second device. Specifically, when the first device and the second device negotiate the model, they may pre-agree on the model identifier model ID, such as the model ID of the linear model may be 1, the model ID of the polynomial model may be 2, and the model ID of the neural network model may be 3. The model can generally be represented by a function, and the model parameters are the parameters of the function expression.

[0012] Taking the linear model as an example, its function expression is y=ax+b. Among them, x is the independent variable, y is the function value, and a and b are the parameters of the linear model. When the target fitting model is a linear model, the first device can send the fitting model identifier "1" and the model parameters "a" and "b" to the second device.

[0013] Sending model identifiers and parameters can further reduce the amount of data transmitted and stored, reduce transmission resource overhead and storage resource overhead, and save costs.

[0014] In some possible implementations, when the multiple data acquired by the first device are associated with other data, the first device may also determine the association relationship between the multiple data and the other data, and the association relationship may be represented by a relationship function. Based on this, the first device may send the relationship function associated with the target fitting model to the second device, so that the relationship function may be used to determine another target fitting model based on the target fitting model, and at least one of the other data associated with the multiple data may be restored through the other target fitting model.

[0015] By sending the above-mentioned relationship function instead of directly sending other data associated with multiple data, the amount of transmitted data and the amount of stored data are further reduced, the compression performance is improved, and the transmission pressure and storage pressure are alleviated.

[0016] In some possible implementations, the first device may further send a difference to the second device. The difference is a difference between the multiple data acquired by the first device and the multiple data restored by the first device based on the target fitting model. The difference may be used to restore at least one of the multiple data in combination with the target fitting model. By superimposing the difference on the data restored by the target fitting model, lossless data recovery can be achieved, thereby improving the accuracy of the transmitted data.

[0017] In some possible implementations, the above-mentioned multiple data may be multiple time series data within a time window. The time window may be set according to actual needs, for example, it may be set to one minute. Time series data refers to time series data, that is, a data column recorded in chronological order for the same indicator. In an example, the time window is one minute and the collection period is one second, then 60 data may be collected within one time window.

[0018] In a second aspect, the present application provides a data processing method. Specifically, a second device as a data receiver receives a target fitting model from a first device, where the target fitting model is obtained by the first device based on multiple acquired data, and the second device restores at least one of the multiple data according to the target fitting model.

[0019] Compared with directly receiving multiple data, receiving the target fitting model and storing the target fitting model greatly reduces the pressure of data transmission and data storage. Moreover, the second device can choose to restore at least one of the multiple data, so as to meet the personalized needs of different users.

[0020] In some possible implementations, the second device may also receive a relationship function associated with the target fitting model, and determine another target fitting model based on the relationship function and the target fitting model, so that the second device can recover at least one of the other data associated with the multiple data according to the other target fitting model.

[0021] Compared with directly receiving multiple data of related indicators, receiving the relationship function associated with the target fitting model can further reduce the amount of transmitted data and the amount of stored data, thereby alleviating the pressure of data transmission and data storage.

[0022] In some possible implementations, the second device can choose to perform lossless recovery or lossy recovery on the data according to demand. For data with high data accuracy requirements, lossless recovery can be selected, and for data with relatively low data accuracy requirements, lossy recovery can be selected to save costs.

[0023] In a specific implementation, the second device directly restores the data according to the target fitting model, which is lossy recovery. When the second device also receives a difference from the first device, the difference is the difference between the multiple data obtained by the first device and the multiple data restored by the first device based on the target fitting model, the second device can restore at least one of the multiple data according to the target fitting model and the difference. Specifically, the second device can first restore the data according to the target fitting model, and then superimpose the above difference on this basis to obtain the original data, thereby achieving lossless data recovery.

[0024] In a third aspect, the present application provides a data processing device, the data processing device comprising:

[0025] A communication module, used for acquiring multiple data;

[0026] A fitting module, used for obtaining a target fitting model based on the plurality of data;

[0027] The communication module is further used to send the target fitting model to the second device, where the target fitting model is used to restore at least one data among the multiple data.

[0028] In some possible implementations, the fitting module is specifically used to:

[0029] Model fitting is performed on a plurality of fitting models according to the plurality of data, so as to select a fitting model from the plurality of fitting models as the target fitting model.

[0030] In some possible implementations, the multiple fitting models include at least two of a linear model, a polynomial model, and a neural network model.

[0031] In some possible implementations, the communication module is specifically used to:

[0032] The identifier of the fitting model selected by the fitting module and the corresponding model parameters are sent to the second device.

[0033] In some possible implementations, the communication module is further configured to:

[0034] A relationship function associated with the target fitting model is sent to the second device, where the relationship function is used to determine another target fitting model based on the target fitting model, and the another target fitting model is used to restore at least one data among other data associated with the multiple data.

[0035] In some possible implementations, the communication module is further configured to:

[0036] A difference value is sent to the second device, where the difference value is used to restore at least one of the multiple data in combination with the target fitting model, and the difference value is a difference between the multiple data acquired by the first device and the multiple data restored by the first device based on the target fitting model.

[0037] In some possible implementations, the multiple data include multiple time series data within a time window.

[0038] In a fourth aspect, the present application provides a data processing device, the device comprising:

[0039] A communication module, configured to receive a target fitting model from a first device, wherein the target fitting model is obtained by the first device based on a plurality of acquired data;

[0040] A recovery module is used to recover at least one of the multiple data according to the target fitting model.

[0041] In some possible implementations, the communication module is further configured to:

[0042] receiving a relationship function associated with the target fitting model;

[0043] The device also includes:

[0044] A determination module, used for determining another target fitting model according to the relationship function and the target fitting model;

[0045] The recovery module is also used for:

[0046] At least one of the other data associated with the plurality of data is restored according to the another target fitting model.

[0047] In some possible implementations, the communication module is further configured to:

[0048] a communication module, configured to receive a difference from a first device, wherein the difference is a difference between the plurality of data acquired by the first device and the plurality of data restored by the first device based on the target fitting model;

[0049] The recovery module is specifically used for:

[0050] At least one of the plurality of data is restored according to the target fitting model and the difference value.

[0051] In a fifth aspect, the present application provides a device, the device comprising a processor and a memory;

[0052] The processor is used to execute instructions stored in the memory so that the processor performs the data processing method as described in the first aspect or the second aspect.

[0053] In a sixth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a device, enables the device to execute the data processing method as described in the first aspect or the second aspect.

[0054] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a device, enables the device to execute the data processing method described in the first or second aspect above.

[0055] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0057] Figure 1 A schematic diagram of a system architecture of a data processing method provided in an embodiment of the present application;

[0058] Figure 2 An interactive flow chart of a data processing method provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0064] Figure 8 A schematic diagram of the structure of a device provided in an embodiment of the present application;

[0065] Fig. 9 A schematic diagram of the structure of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0067] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0068] Figure 1 A possible application scenario of an embodiment of the present application is shown. In this application scenario, device 102 and device 104 are network node devices. Among them, device 102 can be a network node device that sends data, and device 104 is a network node device that receives the data. Among them, the data sent by device 102 can be data generated by device 102 itself, or it can be data obtained by device 102 from other devices. In some implementations, device 102 can also be used as a device for receiving data, and device 104 can also be used as a device for sending data. This application is only illustrated by an example of data being sent from device 102 to device 104, and does not constitute a limitation on the technical solution of this application.

[0069] like Figure 1 As shown, device 102 can be a mobile terminal device such as a smart phone, a smart bracelet, a tablet computer, etc., which can monitor any one or more indicators such as position and heartbeat through a position sensor, a heartbeat sensor, etc. to generate monitoring data. In some embodiments, device 102 can also be a device deployed with a cloud application, which can monitor any one or more indicators such as the throughput and latency of the application to generate monitoring data. Of course, device 102 can also be an end-side device of the Internet of Things (IoT), such as a smart refrigerator, a smart light bulb, etc., which can monitor any one or more indicators such as temperature and brightness to generate monitoring data.

[0070] Correspondingly, the device 104 may be a data transfer device. For example, in a cloud service scenario, the device 104 may be an edge device of the cloud, such as a gateway device (GW), a switch device, etc. In some embodiments, the device 104 may also be a data consumption device, such as a server, etc.

[0071] If device 102 directly sends the collected data to device 104, when the data is large, on the one hand, a large bandwidth resource is required, and on the other hand, device 104 needs to have sufficient storage space, which puts forward high requirements for network transmission and storage. Based on this, the present application provides a data processing method, which supports device 102 as a data sender to first obtain multiple data, and then obtain a target fitting model based on the multiple data, and send the target fitting model instead of directly sending multiple data. In this way, the amount of data sent can be reduced, the requirements for network transmission can be reduced, and the transmission pressure can be reduced. In addition, the recipient can store the target fitting model, and when the data needs to be used, restore at least one of the multiple data based on the target fitting model, which also reduces the requirements for data storage, reduces storage pressure, and improves storage performance.

[0072] In order to make the technical solution of the present application clearer and easier to understand, the data processing method provided in the embodiment of the present application will be described in detail from the perspective of the interaction between the device 102 and the device 104.

[0073] See also Figure 2 The interactive flow chart of the data processing method shown in FIG. 1 includes:

[0074] S202: The device 102 obtains multiple data.

[0075] In a specific implementation, the device 102 may generate multiple data or obtain multiple data from other devices. Among them, the multiple data obtained by the device 102 (such as data generated by the device 102 itself, data obtained by the device 102 from other devices) may be time series data, that is, data of an indicator within a time period. Of course, the multiple data obtained by the device 102 may also be non-time series data, such as data related to location.

[0076] To facilitate understanding, the following description is given with reference to specific examples.

[0077] In one example, the multiple data acquired by the device 102 may be the CPU usage rate of the virtual machine in a time period and / or the memory usage rate of the virtual machine in a time period obtained by the device monitoring one or more of the indicators such as the central process unit (CPU) usage rate and the memory usage rate of the virtual machine. Of course, the multiple data acquired by the device 102 may also be the throughput and / or latency of the application in a time period obtained by the device monitoring one or more of the indicators such as the throughput and latency of the application.

[0078] It should be noted that the CPU usage rate, memory usage rate, throughput, and latency within a time period described in the embodiment of the present application specifically refer to the CPU usage rate, memory usage rate, throughput, and latency corresponding to different time points within a time period. Based on this, the device 102 and the device 104 that perform data interaction can pre-negotiate the data collection rules. For example, when the data is time series data, the device 102 and the device 104 can pre-negotiate the data collection cycle, that is, the collection time interval. Of course, the device 102 can also send collection rules to the device 104 when sending data, such as the collection cycle for sending data.

[0079] In another example, the multiple data acquired by device 102 may also be data generated by monitoring indicators such as rainfall at different locations. This data is non-time series data, which may change with different geographical locations. Similar to time series data, device 102 and device 104 may pre-negotiate the data collection location interval. For example, the collection location interval may be measured by longitude and latitude, such as 1 longitude or one dimension. In some possible implementations, the collection location interval may also be measured by length or height, such as 5 kilometers (km), etc.

[0080] When the device 102 obtains multiple data, it is often necessary to transmit metadata describing the data when transmitting the data. Metadata is specifically data that describes the data, which can be at least one attribute of the data. For example, for data such as CPU usage, its metadata may include the indicator name, the project identifier of the project to which it belongs, the service name, etc. The metadata of multiple data corresponding to the same indicator are often consistent. Therefore, compared with directly transmitting the data and its metadata, obtaining multiple data can achieve the aggregation of metadata, and only one copy of the metadata needs to be transmitted. In this way, data compression can be achieved.

[0081] After the device 102 obtains the multiple data, it can further process the multiple data, such as compressing them, so as to reduce the transmission overhead and storage overhead. In the specific implementation, the device 102 can obtain multiple data according to the pre-configured acquisition quantity N, that is, the device 102 can obtain N data at a time. When the data obtained by the device 102 is time series data, the device 102 can obtain multiple time series data within the time window Δt. Among them, the acquisition quantity N or the time window Δt can be set according to the empirical value, and the embodiment of the present application does not limit this.

[0082] The data in the embodiments of the present application may be numerical values. In some cases, the data may also be characters, such as characters A, B, C, D, etc. representing levels.

[0083] S204: The device 102 obtains a target fitting model based on the multiple data.

[0084] In a specific implementation, the device 102 may perform model fitting on multiple data, and determine a fitting model that meets preset conditions as a target fitting model. In some implementations, in order to reduce the amount of calculation for model fitting and improve the efficiency of model fitting, the device 102 and the device 104 may negotiate multiple fitting models in advance, so that the device 102 may perform model fitting on the multiple fitting models according to the multiple data, so as to select a fitting model from the multiple fitting models as the target fitting model.

[0085] The multiple fitting models negotiated by the device 102 and the device 104 may be models that are easy to fit. In some implementations, the multiple fitting models may be at least two of a linear model, a polynomial model, and a neural network model. The neural network model may be a three-layer neural network model, which can reduce the complexity of the model and make the model easier to fit.

[0086] When the device 102 simulates and fits multiple fitting models according to multiple data, it can calculate the loss value of each fitting model according to the multiple data and the data obtained by model fitting, and then select a fitting model from the multiple fitting models as the target fitting model according to the loss value.

[0087] Specifically, the device 102 may mark multiple data as training data, and then train the fitting model based on the training data in combination with the loss function to optimize the fitting model parameters until the training end condition is met. The training end condition may be that the number of parameter iterations reaches a maximum number, or that the loss value determined based on the loss function is less than a preset value. After the training is completed, the device 102 determines the target fitting model according to the loss value of each fitting model.

[0088] In some possible implementations, the device 102 may use the fitting model with the smallest loss value as the target fitting model, or use the fitting model with a loss value less than a preset value as the target fitting model, or use the fitting model with a convergent loss value as the target fitting model.

[0089] S206: The device 102 determines the difference between the multiple data acquired by it and the multiple data restored based on the target fitting model.

[0090] Specifically, the multiple data acquired by the device 102 and the multiple data restored based on the target fitting model correspond one to one. For example, if the device 102 acquires N data, the N data can be restored based on the target fitting model. The device 102 can perform a difference between each of the acquired multiple data and the data restored by the corresponding target fitting model, thereby obtaining multiple difference values, which are used to achieve lossless data recovery, that is, accurately restore the multiple data acquired by the first device according to the target fitting model and the difference values.

[0091] It should be understood that when executing the data processing method provided in the embodiment of the present application, the device 102 may not execute the above S206. For example, in a scenario where lossy data compression can meet the requirements, the device 102 may not execute the above S206.

[0092] S208 : The device 102 sends the target fitting model to the device 104 .

[0093] In a specific implementation, the device 102 and the device 104 may agree on the model identifiers of the negotiated multiple fitting models. In this way, when the device 102 selects a fitting model from the multiple fitting models as the target fitting model, it may send the identifier of the selected fitting model and the corresponding model parameters to the device 104, thereby sending the target fitting model to the device 104. Based on this, the transmission overhead can be further reduced.

[0094] For ease of understanding, this application also provides a specific example for illustration.

[0095] In this example, the identifier model_ID of the linear model may be 1, the identifier model_ID of the polynomial model may be 2, and the identifier model_ID of the neural network model may be 3. The linear model may be expressed as:

[0096] y=ax+b (1)

[0097] Wherein, a and b are parameters of the linear model, indicating the slope and intercept respectively. x can be time or longitude and latitude (indicating a region), and y can be data recovered by the device 104 .

[0098] When device 102 selects a linear model as a target fitting model, device 102 may send a sequence or array including the following data, such as {1, a, b} or [1, a, b], to send the target fitting model to device 104 .

[0099] Assuming that the time window includes 60 data, compared with sending 60 data, in an embodiment of the present application, the device 102 only needs to send a very short series or array. For example, for a linear model, it only needs to send the model identifier and the model parameters, a total of three parameters, which is equivalent to compressing 95% of the data, greatly reducing the amount of transmitted data and the amount of stored data.

[0100] It should be noted that the execution order of S206 and S208 can be arbitrary, for example, they can be executed in parallel, or they can be executed successively according to a set order.

[0101] S210: The device 102 sends to the device 104 the difference between the multiple data acquired by the device 102 and the multiple data restored based on the target fitting model.

[0102] When the device 102 sends the difference value and the target fitting model, it may send them in one message or in multiple messages.

[0103] The device 102 may execute the above S208 and S210 in parallel, or may execute them sequentially in a set order.

[0104] It should be understood that, similar to S206, when executing the data processing method provided in the embodiment of the present application, the device 102 may not execute the above S210. For example, in a scenario where lossy data compression can meet the requirements, the device 102 may not execute the above S210.

[0105] S212: The device 104 restores a plurality of data according to the target fitting model and the difference.

[0106] For ease of understanding, the data acquired by device 102 may be denoted as y0, the data restored by device 102 based on the target fitting model may be denoted as y1, and the difference between the data acquired by device 102 and the data y1 restored by device 102 based on the target fitting model may be denoted as Δy.

[0107] When device 104 receives the target fitting model and the difference Δy, it can first restore y1 according to the target fitting model. For example, for time series data, device 104 can substitute time into the expression of the target fitting model to restore y1. It should be noted that for ordered time series data, such as periodic time series data, device 104 can substitute the corresponding time into the expression of the target fitting model based directly on the agreed acquisition cycle to restore y1. For non-ordered time series data, device 102 can also send time to device 104, such as "1, 2, 4, 5", so that device 104 can substitute the time sent by device 102 into the expression of the target fitting model to restore y1.

[0108] Furthermore, after restoring y1, the device 104 may superimpose y1 with the corresponding difference Δy, thereby restoring y0 and achieving lossless data recovery.

[0109] It should be understood that, similar to S206 and S210, when executing the data processing method provided in the embodiment of the present application, in a scenario where lossy data compression can meet the requirements, the device 104 may not perform the step of superimposing y1 with the corresponding difference Δy to restore y0. That is, the device 104 may restore multiple data according to the target fitting model.

[0110] Based on the above description, the embodiment of the present application provides a data processing method, which supports the device 102 as the data sender to first obtain multiple data, and then obtain a target fitting model based on the multiple data, and send the target fitting model instead of directly sending multiple data, so that the amount of data sent can be reduced, the requirements for network transmission can be reduced, and the transmission pressure can be reduced. And the device 104 as the receiving party can store the target fitting model, and when the data needs to be used, it can restore multiple data based on the target fitting model, which also reduces the requirements for data storage, reduces storage pressure, and improves storage performance.

[0111] Furthermore, the data processing method provided by the present application also supports lossy and lossless compression for user selection. For the lossy compression method, the device 102 sends the target fitting model, and the device 104 restores the data acquired by the device 102 according to the target fitting model. For the lossless compression method, the device 102 also sends the difference between the data acquired by the device 102 and the data restored based on the target fitting model, and the device 104 accurately restores the data acquired by the device 102 according to the target fitting model and the above difference. In addition, since the difference sent by the device 102 is generally a very small value, it will occupy fewer bits in encoding than a large value, so the compression effect can also be obtained, reducing the transmission and storage pressure.

[0112] For ease of understanding, the present application embodiment also provides a schematic diagram of the data processing method. Figure 3 As shown, the device 102 can obtain multiple data in the time window, and then perform model fitting based on the obtained multiple data to obtain a target fitting model. Figure 3 The data is time series data and the target fitting model is a sinusoidal model for illustration. In other possible implementations, other types of data and other types of models may also be used, which are not limited in this embodiment.

[0113] Then, the device 102 may send the target fitting model and the difference between the acquired data and the target fitting model recovery data to the device 104. In this example, the difference is specifically [0.01, 0.12, 0.03, ..., 0.06]. The device 104 may perform lossy data recovery according to the target fitting model. When the data accuracy requirement is high, the device 104 may also superimpose the above difference on the basis of the lossy data recovery result, thereby achieving lossless data recovery.

[0114] Figure 3 The example is described by collecting one data at each moment, that is, each timestamp corresponds to a value. However, in many scenarios, the values ​​of multiple indicators can be collected at one moment. For example, the device 102 collects environmental information once a minute, and the environmental information may include two indicators: temperature and humidity.

[0115] The multiple indicators collected by the device 102 may be independent of each other or may be associated with each other. For the indicators with an associated relationship, the data may be further compressed based on the association to further reduce the transmission overhead and storage overhead.

[0116] Specifically, for multiple indicators with an associated relationship, the device 102 may perform model fitting only for multiple data corresponding to one indicator. For the convenience of description, the indicator for model fitting may be referred to as a basic indicator. The device 102 may determine the association relationship between other indicators in the multiple indicators and the above basic indicator based on the data of the multiple indicators. The association relationship may be characterized by a relationship function. For example, when the association relationship is linear, it may be characterized by a linear function.

[0117] Based on this, other indicators of device 102 send the relationship function associated with the target fitting model to device 104, and device 104 can determine another target fitting model based on the relationship function and the target fitting model. Specifically, device 104 substitutes the expression of the target fitting model into the above relationship function to obtain the expression of another target fitting model. Then device 104 can restore other multiple data associated with multiple data according to another target fitting model, that is, multiple data corresponding to other indicators associated with the basic indicators. Among them, the specific implementation of device 104 restoring other multiple data associated according to another target fitting model is similar to restoring multiple data according to one target model. For details, please refer to the relevant content description above, which will not be repeated here.

[0118] For ease of understanding, the present application also provides a specific example to illustrate the process of restoring multiple associated data.

[0119] like Figure 4 As shown, for data including multiple indicators, the device 102 can determine the correlation between the indicators according to the multiple data corresponding to each indicator. In this example, the multiple data of 8 indicators in the time window can be divided into two groups, the indicators within the group have a correlation relationship, and the indicators between the groups do not have a correlation relationship.

[0120] For indicators that have no correlation, you can use the following method: Figure 2 The data processing method provided by the embodiment shown is compressed. In this example, the device 102 can perform model fitting on multiple data corresponding to one indicator in each group of indicators to obtain a corresponding target fitting model. Then the device 102 sends the target fitting model and the relationship function representing the association relationship to the device 104. Figure 4As shown, the device 102 sends two target fitting models and three relationship functions associated with each target fitting model. The device 104 can restore multiple data of the corresponding indicators according to the target fitting models. In addition, the device 104 can determine the target fitting models corresponding to the indicators with the associated relationship according to the target fitting models and the relationship functions, and restore the associated multiple data based on the target fitting models corresponding to the indicators with the associated relationship.

[0121] The data processing method of the present application is introduced below by taking the data processing of the two indicators of CPU usage and memory usage as an example.

[0122] like Figure 5 As shown, the device 102 collects CPU usage and memory usage at a collection period of 1 second. In this example, the device 102 needs to collect data within 10 minutes and send the data within 10 minutes to the device 104. In the traditional method, the device 102 sends its data and corresponding metadata to the device 104 every time it collects a piece of data.

[0123] See also Figure 5 , the metadata sent by device 102 includes the project identifier project_id of the project to which the CPU belongs, the cluster identifier cluster_id, the service name service_name, the service instance name service_instance_name, and the metric name metric_name. Since the data sent by device 102 is time series data, it is necessary to send the timestamp timestamp and the data value value when sending the data. Based on this, the data of an indicator at a time point can be a data block composed of the above metadata and data. When device 102 sends the data of one indicator within 10 minutes, it sends 600 data blocks.

[0124] Among them, the metadata corresponding to an indicator at different times is often consistent. By obtaining multiple data (such as caching multiple data), the metadata of multiple data can be aggregated. Therefore, the device 102 only needs to send one data block, in which the data part can include the start timestamp timestamp_from and the end timestamp timestamp timestamp_to. In addition, the data value is also changed from a single value to an array of values ​​corresponding to multiple moments. Since there is no need to repeatedly send repeated metadata, the device 104 does not need to store repeated metadata, thereby reducing transmission pressure and storage pressure.

[0125] Furthermore, the device 102 can perform model fitting based on the acquired multiple data, specifically based on the aggregated array as training data, and fit several pre-set fitting models, such as linear models, polynomial models, and neural network models, respectively, calculate the loss value of each fitting model, and determine the target fitting model. The device 102 can record the model identifier and model parameters in the value field. In this way, the value field can be reduced from an array including 60 data to an array including 3 data, further reducing the amount of data, reducing the transmission pressure and storage pressure. The device 104 receives the above data and stores it. Correspondingly, when the device 104 has a demand for some data of CPU usage, the device 104 can read the model identifier and model parameters of the value field, restore the target fitting model, and then restore at least one of the multiple CPU usage data based on the target fitting model.

[0126] When the memory usage rate and the CPU usage rate are correlated, the data to be sent can be further compressed. Specifically, for the correlation indicator memory usage rate, the device 102 can send a data block consisting of simplified metadata and simplified data. Figure 5 As shown, the simplified metadata includes the indicator name metric_name, which is memory in this example. Other metadata of the data can refer to the associated indicators, also called reference indicators. The simplified data includes a reference indicator (reference, ref) field and a value field, wherein the field value of the reference indicator field is CPU, and the value field is used to identify the relationship function between the indicator and the reference indicator, wherein the value field includes several numerical values ​​including a function identifier and function parameters. In this way, the amount of data that needs to be transmitted for the associated indicator is greatly reduced, and the transmission pressure and storage pressure are reduced. Correspondingly, when the device 104 has a demand for some data on memory usage, the device 104 can read the function identifier and function parameters of the value field, thereby restoring the relationship function between memory usage and CPU usage, restoring the target fitting model of memory usage based on the relationship function and the target fitting model of CPU usage, and then restoring at least one of the multiple memory usage data based on the target fitting model.

[0127] Combination of the above Figures 1 to 5 The data processing method provided in the embodiment of the present application has been introduced. The data processing device and equipment provided in the embodiment of the present application are introduced below in conjunction with the accompanying drawings.

[0128] See also Figure 6 The data processing device 300 shown in FIG. 1 includes:

[0129] A communication module 302, used to obtain multiple data;

[0130] A fitting module 304, configured to obtain a target fitting model based on the plurality of data;

[0131] The communication module 302 is further used to send the target fitting model to the second device 102, where the target fitting model is used to restore at least one data among the multiple data.

[0132] In some possible implementations, the fitting module 304 is specifically configured to:

[0133] Model fitting is performed on a plurality of fitting models according to the plurality of data, so as to select a fitting model from the plurality of fitting models as the target fitting model.

[0134] In some possible implementations, the multiple fitting models include at least two of a linear model, a polynomial model, and a neural network model.

[0135] In some possible implementations, the communication module 302 is specifically configured to:

[0136] The identifier of the fitting model selected by the fitting module 304 and the corresponding model parameters are sent to the second device.

[0137] In some possible implementations, the communication module 302 is further configured to:

[0138] A relationship function associated with the target fitting model is sent to the second device, where the relationship function is used to determine another target fitting model based on the target fitting model, and the another target fitting model is used to restore at least one data among other data associated with the multiple data.

[0139] In some possible implementations, the communication module is further configured to:

[0140] A difference value is sent to the second device, where the difference value is used to restore at least one of the multiple data in combination with the target fitting model, and the difference value is a difference between the multiple data acquired by the first device and the multiple data restored by the first device based on the target fitting model.

[0141] The data processing device 300 according to the embodiment of the present application may be corresponding to the method described in the embodiment of the present application, and the above and other operations and / or functions of each module in the data processing device 300 are respectively to implement Figure 2 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0142] Next, see Figure 7The data processing device 400 shown in FIG. 1 includes:

[0143] The communication module 402 is used to receive a target fitting model from a first device, where the target fitting model is obtained by the first device based on a plurality of acquired data;

[0144] The recovery module 404 is used to recover at least one of the multiple data according to the target fitting model.

[0145] In some possible implementations, the communication module 402 is further configured to:

[0146] receiving a relationship function associated with the target fitting model;

[0147] The device 400 further includes:

[0148] A determination module, used for determining another target fitting model according to the relationship function and the target fitting model;

[0149] The recovery module 404 is also used for:

[0150] At least one of the other data associated with the plurality of data is restored according to the another target fitting model.

[0151] In some possible implementations, the communication module is further configured to:

[0152] A communication module 402 is configured to receive a difference from a first device, where the difference is a difference between the plurality of data acquired by the first device and the plurality of data restored by the first device based on the target fitting model;

[0153] The recovery module 404 is specifically used for:

[0154] At least one of the plurality of data is restored according to the target fitting model and the difference value.

[0155] The data processing device 400 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each module in the data processing device 400 are respectively to implement Figure 2 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0156] Figures 8 to 9 A device is also provided. Figure 8 The device 500 shown can be used to implement the above Figure 6 The functions of the data processing device 300 in the embodiment shown are: Fig. 9 The device 600 shown can be used to implement the above Figure 7The functionality of the data processing device 400 in the illustrated embodiment.

[0157] Device 500 includes bus 501, processor 502, communication interface 503 and memory 504. Processor 502, memory 504 and communication interface 503 communicate through bus 501. Bus 501 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The communication interface 503 is used for external communication, such as obtaining a plurality of data, sending a target fitting model to a second device, etc.

[0158] The processor 502 may be a central processing unit (CPU). The memory 504 may include a volatile memory, such as a random access memory (RAM). The memory 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD or a SSD.

[0159] The memory 504 stores executable codes, and the processor 502 executes the executable codes to perform the aforementioned data processing method.

[0160] Specifically, in implementing Figure 6 In the case of the embodiment shown, and Figure 6 When each module described in the embodiment is implemented by software, Figure 6 The software or program code required for the fitting module 304 function is stored in the memory 504. The communication module 302 function is implemented through the communication interface 503. The processor 502 is used to execute the instructions in the memory 504 and execute the data processing method applied to the data processing device 300.

[0161] The device 600 includes a bus 601, a processor 602, a communication interface 603, and a memory 604. The processor 602, the memory 604, and the communication interface 603 communicate with each other via the bus 601. Figure 7 In the case of the embodiment shown, and Figure 7When each module described in the embodiment is implemented by software, the software or program code required for the recovery module 404 in Execution 7 is stored in the memory 604. The communication module 402 function is implemented through the communication interface 603. The processor 602 is used to execute the instructions in the memory 604 and execute the data processing method applied to the data processing device 400.

[0162] The embodiment of the present application also provides a computer-readable storage medium, including instructions, which, when executed on a device, enables the device to execute the above-mentioned data processing method applied to the data processing apparatus 300 or the data processing apparatus 400.

[0163] The embodiment of the present application also provides a computer program product, and when the computer program product is executed by a computer, the computer executes any of the aforementioned data processing methods. The computer program product may be a software installation package, and when any of the aforementioned data processing methods is required, the computer program product may be downloaded and executed on a computer.

Claims

1. A data processing method, characterized in that: The method comprises: The first device obtains multiple data and metadata of the multiple data, where metadata of the same indicator in the multiple data are the same and are aggregated into one metadata, and the metadata is at least one attribute of the data; The first device obtains a target fitting model based on the plurality of data; The first device sends the target fitting model, aggregated metadata and difference to the second device, the target fitting model is used to recover at least one of the multiple data, the difference is the difference between the multiple data acquired by the first device and the multiple data recovered by the first device based on the target fitting model, and is used to recover at least one of the multiple data in combination with the target fitting model.

2. The method according to claim 1, characterized in that The first device obtains a target fitting model based on the multiple data, including: Model fitting is performed on a plurality of fitting models according to the plurality of data, so as to select a fitting model from the plurality of fitting models as the target fitting model.

3. The method according to claim 2, characterized in that The plurality of fitting models include at least two of a linear model, a polynomial model, and a neural network model.

4. The method according to claim 2, characterized in that: The first device sending the target fitting model to the second device includes: The first device sends an identifier of the fitting model selected by the first device and corresponding model parameters to the second device.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The first device sends a relationship function associated with the target fitting model to the second device, where the relationship function is used to determine another target fitting model based on the target fitting model, and the another target fitting model is used to restore at least one data among other data associated with the multiple data.

6. The method according to any one of claims 1 to 4, characterized in that: The multiple data include multiple time series data within a time window.

7. A data processing method, characterized in that: The method comprises: The second device receives a target fitting model, aggregated metadata, and a difference value from the first device, wherein the target fitting model is obtained by the first device based on multiple data acquired, the aggregated metadata is obtained by aggregating metadata of a same indicator in the multiple data, the metadata of the same indicator in the multiple data are the same, the metadata is at least one attribute of the data, and the difference value is a difference between the multiple data acquired by the first device and the multiple data restored by the first device based on the target fitting model; The second device restores at least one data among the plurality of data according to the target fitting model and the difference value.

8. The method according to claim 7, characterized in that The method further comprises: The second device receives a relationship function associated with the target fitting model; The second device determines another target fitting model according to the relationship function and the target fitting model; The second device restores at least one data among other data associated with the plurality of data according to the another target fitting model.

9. A data processing device, characterized in that: The device comprises: A communication module, used to obtain multiple data and metadata of the multiple data, wherein metadata of the same indicator in the multiple data are the same and are aggregated into one metadata, and the metadata is at least one attribute of the multiple data; A fitting module, used for obtaining a target fitting model based on the plurality of data; The communication module is also used to send the target fitting model, aggregated metadata and difference to the second device, the target fitting model is used to restore at least one of the multiple data, the difference is the difference between the multiple data acquired by the first device and the multiple data restored by the first device based on the target fitting model, and is used to restore at least one of the multiple data in combination with the target fitting model.

10. The device according to claim 9, characterized in that The fitting module is specifically used for: Model fitting is performed on a plurality of fitting models according to the plurality of data, so as to select a fitting model from the plurality of fitting models as the target fitting model.

11. The device according to claim 10, characterized in that The plurality of fitting models include at least two of a linear model, a polynomial model, and a neural network model.

12. The device according to claim 10, characterized in that The communication module is specifically used for: The identifier of the fitting model selected by the fitting module and the corresponding model parameters are sent to the second device.

13. The device according to any one of claims 9 to 12, characterized in that The communication module is also used for: A relationship function associated with the target fitting model is sent to the second device, where the relationship function is used to determine another target fitting model based on the target fitting model, and the another target fitting model is used to restore at least one data among other data associated with the multiple data.

14. The device according to any one of claims 9 to 12, characterized in that The multiple data include multiple time series data within a time window.

15. A data processing device, characterized in that: The device comprises: a communication module, configured to receive a target fitting model, aggregated metadata, and a difference value from a first device, wherein the target fitting model is obtained by the first device based on the plurality of data acquired, the aggregated metadata is obtained by aggregating metadata of a same indicator in the plurality of data, the metadata of the same indicator in the plurality of data are the same, the metadata is at least one attribute of the plurality of data, and the difference value is a difference between the plurality of data acquired by the first device and the plurality of data restored by the first device based on the target fitting model; A recovery module is used to recover at least one of the multiple data according to the target fitting model and the difference.

16. The device according to claim 15, characterized in that The communication module is also used for: receiving a relationship function associated with the target fitting model; The device also includes: A determination module, used for determining another target fitting model according to the relationship function and the target fitting model; The recovery module is also used for: At least one of the other data associated with the plurality of data is restored according to the another target fitting model.

17. A device, characterized in that The device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the processor executes the method according to any one of claims 1 to 8.

18. A computer-readable storage medium comprising instructions, which, when executed on a device, causes the device to perform the method according to any one of claims 1 to 8.

Images