Data management method and device, computer equipment and storage medium

Abstract

The invention relates to a data management method and device, computer equipment and a storage medium. The method comprises the steps of obtaining operation and maintenance data obtained by monitoring target equipment; obtaining the number of data buckets associated with the target data dimension as a target number; dividing the operation and maintenance data to form a target number of data buckets; determining operation and maintenance data in the data buckets, wherein the data relevance between the operation and maintenance data and the adjacent data buckets conforms to expected relevance, and taking the operation and maintenance data as target data; constructing a representation data set representing the operation and maintenance data; wherein the representation data set comprises initial data, ending data and target data of the operation and maintenance data; and storing the operation and maintenance data to the first data cluster, and storing the representation data set to the second data cluster. By adopting the method, the data management efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to a data management method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the rapid development of technologies such as the Internet of Things and cloud computing, the scale of data has exploded and is often accompanied by high dimensionality and high noise. Therefore, a large amount of computing resources are required to process and analyze the data, resulting in problems such as low management efficiency and high resource consumption when managing data on a large scale. Summary of the Invention

[0003] Based on this, it is necessary to provide a data management method, device, computer equipment and storage medium that can improve management efficiency in response to the above technical problems.

[0004] In order to solve the above technical problems, in a first aspect, a data management method is provided, the method comprising:

[0005] Obtain operation and maintenance data obtained by monitoring the target device;

[0006] Obtain the number of data buckets associated with the target data dimension as the target number; divide the operation and maintenance data to form the target number of data buckets;

[0007] Determine the operation and maintenance data whose data correlation between the data in the data bucket and its adjacent data buckets meets the expected correlation, and use it as the target data;

[0008] Constructing a characterization data set representing the operation and maintenance data; wherein the characterization data set includes initial data, end data, and target data of the operation and maintenance data;

[0009] The operation and maintenance data is stored in the first data cluster, and the characterization data set is stored in the second data cluster.

[0010] In one embodiment, obtaining the number of data buckets associated with the target data dimension includes:

[0011] Obtain operation and maintenance data, clean the operation and maintenance data, remove invalid data in the operation and maintenance data, and obtain collected data;

[0012] A preset time dimension is obtained, and the preset time dimension is used as a target sampling point. The collected data is reduced in dimension based on the target sampling point to obtain reduced-dimensional data.

[0013] In one embodiment, obtaining a number of data buckets associated with a target data dimension as a target number; and dividing the operation and maintenance data to form the target number of data buckets includes:

[0014] Get the number of data points corresponding to the dimensionality reduction data and the target data dimension;

[0015] Take the target data dimension as the minuend and the number of the first and last data points corresponding to the operation and maintenance data as the subtrahend to perform a subtraction operation to obtain the target number;

[0016] The number of data points corresponding to the dimensionality reduction data is used as the dividend, and the target number is used as the divisor to perform division operation to obtain the quotient and remainder;

[0017] Assign the corresponding number of data points to each data bucket;

[0018] Select the number of data buckets corresponding to the remainder as distribution buckets, and evenly distribute the number of data points corresponding to the remainder to each distribution bucket.

[0019] In one embodiment, the operation and maintenance data that determines whether the data correlation between a data bucket and its adjacent data buckets meets the expected correlation, as target data, includes:

[0020] Determine a current data bucket, a first adjacent data bucket that is in the previous order of the current data bucket, and a second adjacent data bucket that is in the next order of the current data bucket;

[0021] Select any data point from the first adjacent data bucket as the first data point;

[0022] Sequentially select the data points in the current data bucket as the second data point;

[0023] Calculating a target value based on the data values ​​of all data points in the second adjacent data bucket, and generating a third data point according to the target value;

[0024] Traverse the data points in the current data bucket and calculate the area of ​​the target triangle formed by connecting the first data point, the second data point, and the third data point;

[0025] The second data point corresponding to the largest target triangle area is used as the target data in the current data bucket.

[0026] In one embodiment, calculating a target value based on data values ​​of all data points in the second adjacent data bucket, and generating a third data point according to the target value includes:

[0027] Obtain the sum of the data values ​​of all data points in the second adjacent data bucket to calculate the first average value;

[0028] The target trend curve is constructed with the order of all data points in the second adjacent data bucket as the X-axis and the data values ​​of all data points in the second adjacent data bucket as the Y-axis;

[0029] Obtaining a trajectory mutation point of the target trend curve, and calculating a second average value based on the sum of data values ​​of data points corresponding to the trajectory mutation point;

[0030] The first average value or the second average value is taken as a target value, and the target value is taken as a data value of the third data point.

[0031] In one embodiment, calculating the target triangle area formed by connecting the first data point, the second data point, and the third data point includes:

[0032] Respectively acquiring time values and data values corresponding to the first data point, the second data point, and the third data point;

[0033] Inputting the time values and the data values corresponding to the first data point, the second data point, and the third data point into a preset calculation formula to obtain the target triangle area formed by connecting the first data point, the second data point, and the third data point;

[0034] The preset calculation formula is as follows:

[0035]

[0036] Wherein, S represents the target triangle area, P1X represents the time value corresponding to the second data point, P1Y represents the data value corresponding to the second data point, P0X represents the time value corresponding to the first data point, P0Y represents the data value corresponding to the second data point, P2X represents the time value corresponding to the third data point, and P2Y represents the data value corresponding to the third data point.

[0037] In one embodiment, the data management method further includes:

[0038] In response to receiving the data management instruction, acquiring a data amount corresponding to the to-be-queried data;

[0039] Based on the data amount corresponding to the to-be-queried data and the data management speed, predicting a predicted query time corresponding to the to-be-queried data;

[0040] Acquiring a target query time range matched with the predicted query time;

[0041] Acquiring a query data cluster corresponding to the target query time range, the query data cluster being a first data cluster or a second data cluster.

[0042] To solve the above technical problems, a second aspect provides a data management device, the device comprising:

[0043] The division module is configured to acquire operation and maintenance data obtained by a monitoring target device; acquire a data bucket number associated with a target data dimension as a target number; and divide the operation and maintenance data to form the target number of data buckets.

[0044] A determination module is used to determine the operation and maintenance data whose data correlation between the data in the data bucket and its adjacent data buckets meets the expected correlation as the target data;

[0045] The storage module is used to construct a characterization data set representing the operation and maintenance data; wherein the characterization data set includes the initial data, end data and target data of the operation and maintenance data; store the operation and maintenance data in the first data cluster, and store the characterization data set in the second data cluster.

[0046] In order to solve the above technical problems, in the third aspect, a computer device is provided, including a memory, a processor and a computer program stored in the memory and runnable on the processor, and the processor implements the following steps when executing the computer program: the processor implements the steps of the above-mentioned first aspect method when executing the computer program.

[0047] In order to solve the above technical problems, in a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method of the above first aspect are implemented.

[0048] Different from the related art, the present application obtains the operation and maintenance data obtained by monitoring the target device; obtains the number of data buckets associated with the target data dimension as the target number; divides the operation and maintenance data to form the target number of data buckets; determines the operation and maintenance data whose data correlation between the data bucket and its adjacent data buckets meets the expected correlation as the target data; constructs a characterization data set that characterizes the operation and maintenance data; wherein the characterization data set includes the initial data, end data and target data of the operation and maintenance data; stores the operation and maintenance data in the first data cluster, and stores the characterization data set in the second data cluster. Applying the above method, in the present application, by performing dimensionality reduction processing on the operation and maintenance data, characterization data is obtained, which can retain the characteristics of the operation and maintenance data while reducing the dimensionality of the operation and maintenance data, and store the operation and maintenance data and characterization data in the first data cluster and the second data cluster respectively, which can improve data management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 1 is a flow chart of a data management method according to an embodiment;

[0050] Figure 2 Schematic diagram of the target triangle area in one embodiment;

[0051] Figure 3 is a flow chart of a data management method according to another embodiment;

[0052] Figure 4 is a structural block diagram of a data management device in one embodiment;

[0053] Figure 5FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] In one embodiment, Figure 1 As shown, the present application provides a data management method, which specifically includes the following steps:

[0056] Step 101: Acquire operation and maintenance data obtained by monitoring the target device.

[0057] Operation and maintenance data can be data from monitoring devices that monitor target devices. For example, it can be time-based time series data such as server performance indicators and network traffic obtained from operation and maintenance monitoring. This data is typically characterized by high throughput, high dimensionality, and high noise.

[0058] After obtaining the operation and maintenance data obtained from monitoring the target device, the operation and maintenance data can be cleaned, denoised, and invalid data removed.

[0059] For example, the Apache Flink real-time stream processing framework can be used to clean the collected time series data, remove noise and invalid data, and obtain cleaned operation and maintenance data, that is, collected data. Compared with traditional offline processing data cleaning methods, using the Apache Flink real-time stream processing framework to clean the collected time series data can meet the needs of real-time monitoring of data.

[0060] In one embodiment, the operation and maintenance data can also be segmented using the Jieba word segmentation technology to obtain segmented data, and the segmented data can be cleaned, denoised, and invalid data removed. For example, taking the operation and maintenance data "the startup time of the server is A" as an example, the result of segmenting the text using the Jieba word segmentation technology is "the server / startup / time / is / A / ". The Jieba word segmentation technology can count the probability of each segmentation result, that is, the rationality of each segmentation result. And because words are the smallest units with semantic information in Chinese sentences, using the Jieba word segmentation technology for word segmentation can make the operation and maintenance data after word segmentation conform to the characteristics of Chinese and improve the accuracy of word segmentation. Optionally, the operation and maintenance data can also be segmented using language processing tools, which are not limited to Jieba, snowNLP, thulac, etc. The auxiliary words in the word system data can be deleted as invalid data with no actual meaning, and the nouns and verbs in the segmented data are retained to obtain the operation and maintenance data after data cleaning, that is, the collected data.

[0061] After obtaining the collected data, the preset time dimension can be obtained, and the preset time dimension is used as the target sampling point. The collected data is reduced in dimension based on the target sampling point to obtain reduced-dimensional data. The preset time dimension can be year, quarter, month, day, hour, minute, and so on. For example, assuming that the operation and maintenance data is a sampling point per minute, it can be aggregated through a 10-minute window, and statistics such as sum or count in each window can be calculated, thereby compressing the data into a sampling point of 10 minutes (preset time dimension) to achieve dimensionality reduction. The preset time dimension here can be set according to actual needs, and the specific time of the preset time dimension is not limited in this application.

[0062] Step 102: Obtain the number of data buckets associated with the target data dimension as the target number; and divide the operation and maintenance data to form the target number of data buckets.

[0063] Obtain the number of data points corresponding to the dimensionality reduction data and the target data dimension; perform a subtraction operation with the target data dimension as the minuend and the number of first and last data points corresponding to the operation and maintenance data as the subtrahend to obtain the target number; perform a division operation with the number of data points corresponding to the dimensionality reduction data as the dividend and the target number as the divisor to obtain the quotient and remainder; allocate the number of data points corresponding to the quotient to each data bucket; select the number of data buckets corresponding to the remainder as distribution buckets, and evenly distribute the number of data points corresponding to the remainder to each distribution bucket.

[0064] Specifically, the number of data points corresponding to the reduced-dimensional data is the target number of sampling points, and the target data dimension is the number of data points to be reduced. Here, the data points corresponding to the reduced-dimensional data can be bucketed, with the number of data points corresponding to the target data dimension as the target dimensionality reduction value. The data points corresponding to the reduced-dimensional data are distributed across the number of data buckets corresponding to the target dimensionality reduction value, and a representative data point is selected from each data bucket to further achieve data dimensionality reduction.

[0065] Set the number of data buckets corresponding to the target dimensionality reduction value and sort each data bucket. Place the first data point corresponding to the operation and maintenance data into the first data bucket set as the representative data point of the first data bucket, and place the last data point corresponding to the operation and maintenance data into the last data bucket set as the representative data point of the last data bucket. In this way, the first and last data points of the original operation and maintenance data are used as the representative data points of the first and last data buckets after dimensionality reduction, which can achieve data dimensionality reduction while retaining the main features of the original data.

[0066] The target data dimension is further subtracted from the number of first and last data points corresponding to the reduced data as the subtrahend to obtain the target number. Here, the target number is the number of data buckets excluding the first and second data buckets. The number of data points corresponding to the reduced data is divided by the target number as the divisor to obtain the quotient and remainder. The number of data points corresponding to the quotient is assigned to each data bucket. The number of data buckets corresponding to the remainder is selected as the distribution buckets, and the number of data points corresponding to the remainder is evenly distributed to each distribution bucket.

[0067] In a specific example, assuming that the number of data points corresponding to the dimensionality reduction data is 100 and the number of data points corresponding to the target data dimension is 20, 20 data buckets are set, and the first data point corresponding to the operation and maintenance data is placed in the first data bucket of the 20 data buckets, and the last data point corresponding to the operation and maintenance data is placed in the last data bucket of the 20 data buckets. The number of first and last data points is the sum of the first data point corresponding to the operation and maintenance data and the last data point corresponding to the operation and maintenance data. The number of first and last data points is 2. After determining the first data bucket and the last data bucket, the data points corresponding to the dimensionality reduction data need to be allocated to the remaining (20-2) buckets.

[0068] The number of data points corresponding to the dimensionality reduction data (100) is used as the dividend, and the target number (20-2) is used as the divisor to perform a division operation, obtaining the quotient (5) and the remainder (10); the number of data points corresponding to the quotient (5) is allocated to each data bucket, that is, 5 data points are allocated to each data bucket in the data bucket corresponding to the target number; the number of data buckets corresponding to the remainder (10) is selected as the distribution bucket, that is, the data bucket corresponding to the remainder value is selected from the data bucket corresponding to the target number as the distribution bucket, and the number of data points corresponding to the remainder is evenly distributed to each distribution bucket. Finally, in the 20 data buckets obtained, the first data bucket and the last data bucket each store a data point corresponding to the operation and maintenance data, the distribution bucket stores (quotient + 1) data points corresponding to the dimensionality reduction data, and the remaining data buckets store data points corresponding to the quotient of the dimensionality reduction data. At this point, multiple data points are stored in multiple data buckets other than the first data bucket and the last data bucket. In order to achieve further dimensionality reduction, it is necessary to determine the target data from the multiple data points in each data bucket to represent the multiple data points in this data bucket.

[0069] Step 103 : Determine the operation and maintenance data whose data correlation between the data bucket and its adjacent data buckets meets the expected correlation, and use it as the target data.

[0070] Specifically, determine the current data bucket, the first adjacent data bucket in the previous order of the current data bucket, and the second adjacent data bucket in the subsequent order of the current data bucket; obtain the first data point from the first adjacent data bucket, and the first data point is the representative data point of the first adjacent data bucket; select the data points in the current data bucket as the second data point in sequence; calculate the target value based on the data values ​​of all data points in the second adjacent data bucket, and generate the third data point according to the target value; traverse the data points in the current data bucket, and calculate the target triangle area formed by connecting the first data point, the second data point, and the third data point; and use the second data point corresponding to the largest target triangle area as the target data in the current data bucket.

[0071] As shown in the above example, assuming that 20 data buckets are set, the first data point corresponding to the operation and maintenance data is the representative data point of the first data bucket among the 20 data buckets. Assuming that the current data bucket is the second data bucket, the first data bucket is the first adjacent data bucket of the current data bucket, and the third data bucket is the second adjacent data bucket of the current data bucket. The representative data point of the first adjacent data bucket is used as the first data point, and the data points in the current data bucket are selected in sequence as the second data points.

[0072] See also Figure 2, you can draw a diagram of the area of ​​the triangle formed by connecting the first data point (P1 candidate point), the second data point (P0 current bucket representative point), and the third data point (P2 back bucket average point). The X-axis represents time and the Y-axis represents data value. Calculate the Y value range of all data points in each data bucket. The Y value range here is different from the Y value. The Y value range refers to the trend of the maximum value minus the minimum value of the data value in each data bucket. When determining the target data of each data bucket later, it is necessary to ensure that the trend of the target data of each data bucket is consistent with the trend represented by the calculated Y value range. In this way, the distortion rate of the dimensionality reduction data can be reduced.

[0073] Obtain the data values ​​of all data points in the second adjacent data bucket to calculate the target value, set a virtual data point, assign a value to the virtual data point according to the target value, and generate a third data point. Traverse the data points in the current data bucket, select the data points in the current data bucket in turn as the second data point, calculate the target triangle area formed by the first data point, the second data point, and the third data point, and use the second data point corresponding to the calculated maximum triangle area as the target data in the current data bucket. According to the order of the set data buckets, determine the current data bucket in turn, and based on the first data point of the first adjacent data bucket of the current data bucket, the third data point of the second adjacent data bucket, and the second data point of the current data bucket, the target data of all current data buckets can be determined.

[0074] In one embodiment, a target value is calculated based on the data values ​​of all data points in the second adjacent data bucket, and generating a third data point according to the target value includes: obtaining the sum of the data values ​​of all data points in the second adjacent data bucket to calculate a first average value; constructing a target trend curve with the order of all data points in the second adjacent data bucket as the X-axis and the data values ​​of all data points in the second adjacent data bucket as the Y-axis; obtaining the trajectory mutation point of the target trend curve, and calculating a second average value based on the sum of the data values ​​of the data points corresponding to the trajectory mutation point; using the first average value or the second average value as the target value, and using the target value as the data value of the third data point.

[0075] The trajectory mutation point here refers to the data point where there is a significant fluctuation or change in the target trend curve. For example, if the data value of any data point in the target trend curve increases or decreases significantly compared to its adjacent data points, this point is the trajectory mutation point.

[0076] This application provides two methods for calculating the target value. The first method is to calculate the first average value of the data values ​​of all data points in the second adjacent data bucket. The second method is to construct a target trend curve based on all data points in the second adjacent data bucket and obtain the second average value of the data values ​​corresponding to the target trajectory mutation point. The first average value and the second average value here reflect the trend of the data values ​​in the second adjacent data bucket.

[0077] By calculating the target value based on the data value of the data point in the second adjacent data bucket and generating a virtual third data point based on the target value to participate in the calculation of the target triangle area, the calculation time can be saved.

[0078] In an embodiment, the calculation of the target triangle area formed by connecting the first data point, the second data point and the third data point can include: obtaining the time value and the data value corresponding to the first data point, the second data point and the third data point respectively; inputting the time value and the data value corresponding to the first data point, the second data point and the third data point into a preset calculation formula to obtain the target triangle area formed by connecting the first data point, the second data point and the third data point.

[0079] The preset calculation formula is as follows:

[0080]

[0081] Wherein, S represents the target triangle area, P1X represents the time value corresponding to the second data point, P1Y represents the data value corresponding to the second data point, P0X represents the time value corresponding to the first data point, P0Y represents the data value corresponding to the second data point, P2X represents the time value corresponding to the third data point, and P2Y represents the data value corresponding to the third data point.

[0082] Step 104, constructing a set of representative data representing the operation and maintenance data; wherein the set of representative data includes initial data, end data and target data of the operation and maintenance data.

[0083] The representative data set stored here is the representative data in each storage bucket set, which specifically includes the first data point (initial data) of the operation and maintenance data stored in the first data bucket, the last data point (end data) of the operation and maintenance data stored in the last data bucket, and the target data of other data buckets calculated.

[0084] Step 105, storing the operation and maintenance data to the first data cluster and storing the set of representative data to the second data cluster.

[0085] In this application, the original operation and maintenance data will be stored in the first data cluster, and the set of representative data obtained by reducing the dimension of the original operation and maintenance data will be stored in the second data cluster. Subsequently, data is selectively obtained from the first data cluster or the second data cluster according to different situations.

[0086] In one embodiment, the data management method further includes: in response to receiving a data management instruction, obtaining the data volume corresponding to the data to be queried; predicting the predicted query time corresponding to the data to be queried based on the data volume corresponding to the data to be queried and the data management speed; obtaining a target query time range that matches the predicted query time; obtaining a query data cluster corresponding to the target query time range, the query data cluster being the first data cluster or the second data cluster.

[0087] In response to receiving the data management instruction, the predicted query time corresponding to the data to be queried is obtained, and it is determined in which query time range the predicted query time is located. The target query time range here is the first query time range or the second query time range. A corresponding relationship between the first query time range and the first data cluster is established, and a corresponding relationship between the second query time range and the second data cluster is established.

[0088] If the target query time range that matches the predicted query time is the first query time range, the data to be queried is retrieved from the first data cluster. If the target query time range that matches the predicted query time is the second query time range, the data to be queried is retrieved from the second data cluster. The first and second query time ranges here reflect the time required to query the data. Generally, when the query time is long, the data to be queried is retrieved from the second data cluster (i.e., the data cluster after dimensionality reduction) to reduce system resource consumption. When the query time is short, the data to be queried is retrieved from the first data cluster (i.e., the original data cluster).

[0089] like Figure 3 As shown, the data management method provided in this application specifically includes technical means for data storage and data query.

[0090] After receiving the operation and maintenance data, the operation and maintenance data will be stored in the first data cluster, and the operation and maintenance data will flow into the data preprocessing and dimensionality reduction module to perform dimensionality reduction processing on the operation and maintenance data to obtain a representative data set, and the representative data set will be stored in the second data cluster.

[0091] After receiving the data management instruction, a data query operation is executed to selectively obtain the data to be queried from the first data cluster or the second data cluster according to the data query time range.

[0092] When returning data, you can determine the size of the data set. If the data set is large, you need to use LTTB to downsample and return the data. If the data set is not large, return the data directly.

[0093] This application realizes efficient query and optimization of operation and maintenance monitoring time series data through the combination of Flink and LTTB algorithms, as well as the data dimensionality reduction function based on the time dimension; it can realize queries for large time ranges, such as data aggregation queries by year, month, day, and hour (data aggregation is performed on 1 minute data through sum / count), and the LTTB dimensionality reduction module will perform corresponding dimensionality reduction processing on the data according to the time granularity of the query. For example, when querying by year, the dimensionality of each year's data will be reduced; when querying by month, the dimensionality of each month's data will be reduced; according to the user's query request, the appropriate index structure and pre-aggregated data are selected to improve query efficiency, reduce unnecessary disk access and data decompression operations, and reduce system resource consumption; for requests for queries by year, month, day, and hour, the query optimization module will give priority to pre-calculated aggregated data to avoid expensive aggregation operations on the original data, and at the same time display the query results to the user in the form of charts, curves, etc., so that the user can understand and analyze the data, reduce unnecessary disk access and data decompression operations, and reduce system resource consumption.

[0094] It should be understood that although Figure 1 、 Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0095] In one embodiment, Figure 4 As shown, a data management device is provided, including: a division module 20, a determination module 21 and a storage module 22, wherein:

[0096] A partitioning module is used to obtain operation and maintenance data obtained by monitoring the target device; obtain the number of data buckets associated with the target data dimension as the target number; and partition the operation and maintenance data to form the target number of data buckets;

[0097] A determination module is used to determine the operation and maintenance data whose data correlation between the data in the data bucket and its adjacent data buckets meets the expected correlation as the target data;

[0098] The storage module is used to construct a characterization data set representing the operation and maintenance data; wherein the characterization data set includes the initial data, end data and target data of the operation and maintenance data; store the operation and maintenance data in the first data cluster, and store the characterization data set in the second data cluster.

[0099] In one embodiment, another implementation of the data management method implemented by the above device includes the following specific steps:

[0100] Get the number of buckets associated with the target data dimension before including:

[0101] Obtain operation and maintenance data, clean the operation and maintenance data, remove invalid data in the operation and maintenance data, and obtain collected data;

[0102] A preset time dimension is obtained, and the preset time dimension is used as a target sampling point. The collected data is reduced in dimension based on the target sampling point to obtain reduced-dimensional data.

[0103] In one embodiment, another implementation of the data management method implemented by the above device includes the following specific steps:

[0104] The number of data buckets associated with the target data dimension is obtained as the target number; the operation and maintenance data is divided to form the target number of data buckets including:

[0105] Get the number of data points corresponding to the dimensionality reduction data and the target data dimension;

[0106] Take the target data dimension as the minuend and the number of the first and last data points corresponding to the operation and maintenance data as the subtrahend to perform a subtraction operation to obtain the target number;

[0107] The number of data points corresponding to the dimensionality reduction data is used as the dividend, and the target number is used as the divisor to perform division operation to obtain the quotient and remainder;

[0108] Assign the corresponding number of data points to each data bucket;

[0109] Select the number of data buckets corresponding to the remainder as distribution buckets, and evenly distribute the number of data points corresponding to the remainder to each distribution bucket.

[0110] In one embodiment, another implementation of the data management method implemented by the above device includes the following specific steps:

[0111] Determine whether the data correlation between a bucket and its adjacent buckets meets the expected correlation. The target data includes:

[0112] Determine a current data bucket, a first adjacent data bucket that is in the previous order of the current data bucket, and a second adjacent data bucket that is in the next order of the current data bucket;

[0113] selecting any data point in the first adjacent data bucket as a first data point;

[0114] selecting a data point in the current data bucket as a second data point in turn;

[0115] calculating a target value based on data values of all data points in the second adjacent data bucket, and generating a third data point according to the target value;

[0116] traversing the data points in the current data bucket, and calculating a target triangle area formed by the first data point, the second data point and the third data point;

[0117] taking the second data point corresponding to the maximum target triangle area as a target data in the current data bucket.

[0118] In one embodiment, another implementation of the data management method that the above device can implement has the following specific steps:

[0119] calculating a target value based on data values of all data points in the second adjacent data bucket, and generating a third data point according to the target value includes:

[0120] calculating a first average value based on a sum of data values of all data points in the second adjacent data bucket;

[0121] constructing a target trend curve with the order of all data points in the second adjacent data bucket as the X axis and the data values of all data points in the second adjacent data bucket as the Y axis;

[0122] obtaining a trajectory mutation point of the target trend curve, and calculating a second average value based on a sum of data values of data points corresponding to the trajectory mutation point;

[0123] taking the first average value or the second average value as the target value, and taking the target value as the data value of the third data point.

[0124] In one embodiment, another implementation of the data management method that the above device can implement has the following specific steps:

[0125] calculating a target triangle area formed by the first data point, the second data point and the third data point includes:

[0126] respectively obtaining time values and data values corresponding to the first data point, the second data point and the third data point;

[0127] inputting the time values and the data values corresponding to the first data point, the second data point and the third data point into a preset calculation formula to obtain the target triangle area formed by the first data point, the second data point and the third data point;

[0128] wherein, the preset calculation formula is as follows:

[0129]

[0130] wherein S represents a target triangle area, P1X represents a time value corresponding to the second data point, P1Y represents a data value corresponding to the second data point, P0X represents a time value corresponding to the first data point, P0Y represents a data value corresponding to the second data point, P2X represents a time value corresponding to the third data point, and P2Y represents a data value corresponding to the third data point.

[0131] In one embodiment, another implementation of the data management method that can be implemented by the above-described device has the following specific steps:

[0132] The data management method further includes:

[0133] In response to receiving the data management instruction, a data amount corresponding to the to-be-queried data is obtained;

[0134] A predicted query time corresponding to the to-be-queried data is predicted based on the data amount corresponding to the to-be-queried data and the data management speed;

[0135] A target query time range matching the predicted query time is obtained;

[0136] A query data cluster corresponding to the target query time range is obtained, and the query data cluster is the first data cluster or the second data cluster.

[0137] The specific limitations on the data management device can be referred to the limitations on the data management method in the foregoing, and will not be described herein. Each module in the above-described data management device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-described modules can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform the operations corresponding to each module.

[0138] In an embodiment, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions, when the program instructions are executed by a computer, the computer can execute the data management method provided by each method.

[0139] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data applied in the data management method. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a data management method.

[0140] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0141] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0142] Step 101, obtaining operation and maintenance data obtained by a monitoring target device.

[0143] Step 102, obtaining the number of data buckets associated with the target data dimension as the target number; dividing the operation and maintenance data to form a target number of data buckets.

[0144] Step 103, determining operation and maintenance data whose data correlation between the data bucket and its adjacent data bucket meets the expected correlation as target data.

[0145] Step 104, constructing a representation data set representing the operation and maintenance data; wherein the representation data set includes initial data, end data and target data of the operation and maintenance data.

[0146] Step 105, storing the operation and maintenance data to a first data cluster, and storing the representation data set to a second data cluster.

[0147] In one embodiment, the processor further implements the following steps when executing the computer program:

[0148] Before obtaining the number of data buckets associated with the target data dimension, it includes:

[0149] Obtain operation and maintenance data, perform data cleaning on the operation and maintenance data, remove invalid data in the operation and maintenance data, and obtain collected data;

[0150] acquire a preset time dimension, target sampling points with the preset time dimension, reduce the dimension of the collected data based on the target sampling points, and obtain reduced dimension data.

[0151] In one embodiment, the processor also implements the following steps when executing the computer program:

[0152] acquire the number of data buckets associated with the target data dimension as the target number; and divide the operation and maintenance data to form the target number of data buckets, including:

[0153] acquire the number of data points corresponding to the reduced dimension data and the target data dimension;

[0154] perform subtraction operation with the target data dimension as the minuend and the number of first and last data points corresponding to the operation and maintenance data as the subtrahend to obtain the target number;

[0155] perform division operation with the number of data points corresponding to the reduced dimension data as the dividend and the target number as the divisor to obtain quotient and remainder;

[0156] allocate the number of data points corresponding to the quotient to each data bucket;

[0157] select the number of data buckets corresponding to the remainder as allocation buckets, and divide the number of data points corresponding to the remainder into each allocation bucket.

[0158] In one embodiment, the processor also implements the following steps when executing the computer program:

[0159] determine the operation and maintenance data whose data association between the data bucket and its adjacent data bucket meets the expected association as the target data, including:

[0160] determine the current data bucket, the first adjacent data bucket belonging to the previous order of the current data bucket, and the second adjacent data bucket of the next order of the current data bucket;

[0161] select any data point in the first adjacent data bucket as the first data point;

[0162] select the data points in the current data bucket as the second data point in turn;

[0163] calculate a target value based on the data values of all data points in the second adjacent data bucket, and generate a third data point according to the target value;

[0164] traverse the data points in the current data bucket, and calculate the area of the target triangle formed by the first data point, the second data point, and the third data point;

[0165] take the second data point corresponding to the maximum target triangle area as the target data in the current data bucket.

[0166] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0167] Calculating a target value based on the data values ​​of all data points in the second adjacent data bucket, and generating a third data point according to the target value includes:

[0168] Obtain the sum of the data values ​​of all data points in the second adjacent data bucket to calculate the first average value;

[0169] The target trend curve is constructed with the order of all data points in the second adjacent data bucket as the X-axis and the data values ​​of all data points in the second adjacent data bucket as the Y-axis;

[0170] Obtaining a trajectory mutation point of the target trend curve, and calculating a second average value based on the sum of data values ​​of data points corresponding to the trajectory mutation point;

[0171] The first average value or the second average value is used as a target value, and the target value is used as a data value of the third data point.

[0172] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0173] Calculating the area of ​​a target triangle formed by connecting the first data point, the second data point, and the third data point includes:

[0174] Obtaining the time value and data value corresponding to the first data point, the second data point, and the third data point respectively;

[0175] Input the time values ​​and data values ​​corresponding to the first data point, the second data point, and the third data point into a preset calculation formula to obtain the target triangle area formed by connecting the first data point, the second data point, and the third data point;

[0176] The preset calculation formula is as follows:

[0177]

[0178] Among them, S represents the area of ​​the target triangle, P1X represents the time value corresponding to the second data point, P1Y represents the data value corresponding to the second data point, P0X represents the time value corresponding to the first data point, P0Y represents the data value corresponding to the second data point, P2X represents the time value corresponding to the third data point, and P2Y represents the data value corresponding to the third data point.

[0179] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0180] Data management methods also include:

[0181] In response to receiving the data management instruction, obtaining the data volume corresponding to the data to be queried;

[0182] predict the predicted query time corresponding to the to-be-queried data based on a data volume corresponding to the to-be-queried data and a data management speed;

[0183] obtain a target query time range matching the predicted query time;

[0184] obtain a query data cluster corresponding to the target query time range, the query data cluster being a first data cluster or a second data cluster.

[0185] In an embodiment, a computer readable storage medium is provided, and the computer readable storage medium has stored thereon a computer program, and the computer program is executed by a processor to implement the following steps:

[0186] Step 101, obtaining operation and maintenance data obtained by a monitoring target device.

[0187] Step 102, obtaining a number of data buckets associated with a target data dimension as a target number, and dividing the operation and maintenance data to form the target number of data buckets.

[0188] Step 103, determining operation and maintenance data in which data correlation between the data bucket and adjacent data buckets thereof meets expected correlation as target data.

[0189] Step 104, constructing a representation data set representing the operation and maintenance data, wherein the representation data set includes initial data, end data, and target data of the operation and maintenance data.

[0190] Step 105, storing the operation and maintenance data to a first data cluster and storing the representation data set to a second data cluster.

[0191] In an embodiment, the computer program is executed by the processor to further implement the following steps:

[0192] Before obtaining the number of data buckets associated with the target data dimension, the following steps are included:

[0193] Obtaining the operation and maintenance data, performing data cleaning on the operation and maintenance data, removing invalid data in the operation and maintenance data, and obtaining collected data.

[0194] Obtaining a preset time dimension, taking the preset time dimension as a target sampling point, performing dimension reduction on the collected data based on the target sampling point, and obtaining reduced data.

[0195] In an embodiment, the computer program is executed by the processor to further implement the following steps:

[0196] Obtaining the number of data buckets associated with the target data dimension as a target number, and dividing the operation and maintenance data to form the target number of data buckets includes:

[0197] obtaining a data point quantity corresponding to the dimension-reduced data and a target data dimension;

[0198] performing subtraction operation with the target data dimension as the minuend and the first and last data point quantities corresponding to the operation and maintenance data as the subtrahend to obtain a target quantity;

[0199] performing division operation with the data point quantity corresponding to the dimension-reduced data as the dividend and the target quantity as the divisor to obtain a quotient and a remainder;

[0200] allocating data points corresponding to the quantity of the quotient to each data bucket;

[0201] selecting a data bucket corresponding to the quantity of the remainder as an allocation bucket and equally distributing data points corresponding to the quantity of the remainder to each allocation bucket.

[0202] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0203] determining operation and maintenance data in the data bucket and between adjacent data buckets thereof that meets an expected correlation as target data includes:

[0204] determining a current data bucket, a first adjacent data bucket belonging to a previous order of the current data bucket, and a second adjacent data bucket belonging to a next order of the current data bucket;

[0205] selecting any data point in the first adjacent data bucket as a first data point;

[0206] selecting data points in the current data bucket in turn as second data points;

[0207] calculating a target value based on data values of all data points in the second adjacent data bucket and generating a third data point according to the target value;

[0208] traversing data points in the current data bucket, calculating a target triangle area formed by the first data point, the second data point, and the third data point;

[0209] selecting the second data point corresponding to the largest target triangle area as target data in the current data bucket.

[0210] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0211] calculating a target value based on data values of all data points in the second adjacent data bucket and generating a third data point according to the target value includes:

[0212] obtaining a sum of data values of all data points in the second adjacent data bucket to calculate a first average value;

[0213] Taking the order of all data points in the second adjacent data bucket as the X axis and the data values of all data points in the second adjacent data bucket as the Y axis, a target trend curve is constructed;

[0214] A trajectory mutation point of the target trend curve is obtained, and a second average value is calculated based on the sum of the data values of the data points corresponding to the trajectory mutation point;

[0215] The first average value or the second average value is taken as a target value, and the target value is taken as the data value of the third data point.

[0216] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0217] The area of the target triangle formed by connecting the first data point, the second data point, and the third data point includes:

[0218] The time values and data values corresponding to the first data point, the second data point, and the third data point are obtained respectively;

[0219] The time values and data values corresponding to the first data point, the second data point, and the third data point are input into a preset calculation formula to obtain the area of the target triangle formed by connecting the first data point, the second data point, and the third data point;

[0220] The preset calculation formula is as follows:

[0221]

[0222] Wherein, S represents the area of the target triangle, P1X represents the time value corresponding to the second data point, P1Y represents the data value corresponding to the second data point, P0X represents the time value corresponding to the first data point, P0Y represents the data value corresponding to the second data point, P2X represents the time value corresponding to the third data point, and P2Y represents the data value corresponding to the third data point.

[0223] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0224] The data management method further includes:

[0225] In response to receiving a data management instruction, the data amount corresponding to the to-be-queried data is obtained;

[0226] The predicted query time corresponding to the to-be-queried data is predicted based on the data amount corresponding to the to-be-queried data and the data management speed;

[0227] A target query time range matching the predicted query time is obtained;

[0228] A query data cluster corresponding to the target query time range is obtained, and the query data cluster is a first data cluster or a second data cluster.

[0229] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0230] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0231] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data management method, characterized in that: The data management method comprises: Obtain operation and maintenance data obtained by monitoring the target device; Obtaining a number of data buckets associated with a target data dimension as a target number; dividing the operation and maintenance data to form the target number of data buckets; Determine the operation and maintenance data whose data correlation between the data in the data bucket and its adjacent data buckets meets the expected correlation as the target data; Constructing a characterization data set representing the operation and maintenance data; wherein the characterization data set includes initial data, end data, and target data of the operation and maintenance data; The operation and maintenance data is stored in a first data cluster, and the characterization data set is stored in a second data cluster.

2. The method according to claim 1, characterized in that The step of obtaining the number of data buckets associated with the target data dimension includes: Acquire operation and maintenance data, perform data cleaning on the operation and maintenance data, remove invalid data in the operation and maintenance data, and obtain collected data; A preset time dimension is obtained, and the preset time dimension is used as a target sampling point. The collected data is reduced in dimension based on the target sampling point to obtain reduced-dimensional data.

3. The method according to claim 1, characterized in that The number of data buckets associated with the target data dimension is obtained as the target number; Dividing the operation and maintenance data to form the target number of data buckets includes: Get the number of data points corresponding to the dimensionality reduction data and the target data dimension; A subtraction operation is performed using the target data dimension as the minuend and the number of first and last data points corresponding to the operation and maintenance data as the subtrahend to obtain the target number; The number of data points corresponding to the dimensionality reduction data is used as the dividend, and the target number is used as the divisor to perform division operation to obtain the quotient and remainder; Assign the corresponding number of data points to each data bucket; Select the number of data buckets corresponding to the remainder as distribution buckets, and evenly distribute the number of data points corresponding to the remainder to each distribution bucket.

4. The method according to claim 1, wherein The operation and maintenance data for determining whether the data correlation between the data in the data bucket and its adjacent data buckets meets the expected correlation as target data includes: Determine the current data bucket, the first adjacent data bucket that is in the previous order of the current data bucket, and the second adjacent data bucket that is in the next order of the current data bucket; Obtain a first data point from a first adjacent data bucket, where the first data point is a representative data point of the first adjacent data bucket; Sequentially select the data points in the current data bucket as the second data point; calculating a target value based on the data values ​​of all data points in the second adjacent data bucket, and generating a third data point according to the target value; Traverse the data points in the current data bucket and calculate the area of ​​a target triangle formed by connecting the first data point, the second data point, and the third data point; The second data point corresponding to the largest target triangle area is used as the target data in the current data bucket.

5. The method according to claim 4, characterized in that Calculating a target value based on the data values ​​of all data points in the second adjacent data bucket and generating a third data point according to the target value includes: Obtain the sum of the data values ​​of all data points in the second adjacent data bucket to calculate the first average value; The target trend curve is constructed with the order of all data points in the second adjacent data bucket as the X-axis and the data values ​​of all data points in the second adjacent data bucket as the Y-axis; Obtaining a trajectory mutation point of the target trend curve, and calculating a second average value based on the sum of data values ​​of data points corresponding to the trajectory mutation point; The first average value or the second average value is used as a target value, and the target value is used as a data value of the third data point.

6. The method according to claim 4, characterized in that Calculating the target triangle area formed by connecting the first data point, the second data point, and the third data point includes: Obtaining the time value and data value corresponding to the first data point, the second data point, and the third data point respectively; Inputting the time values ​​and data values ​​corresponding to the first data point, the second data point, and the third data point into a preset calculation formula to obtain the target triangle area formed by connecting the first data point, the second data point, and the third data point; The preset calculation formula is as follows: Among them, S represents the area of ​​the target triangle, P1X represents the time value corresponding to the second data point, P1Y represents the data value corresponding to the second data point, P0X represents the time value corresponding to the first data point, P0Y represents the data value corresponding to the second data point, P2X represents the time value corresponding to the third data point, and P2Y represents the data value corresponding to the third data point.

7. The method according to claim 1, characterized in that The data management method further includes: In response to receiving the data management instruction, obtaining the data volume corresponding to the data to be queried; Predicting a predicted query time corresponding to the data to be queried based on the data volume and data management speed corresponding to the data to be queried; Obtaining a target query time range that matches the predicted query time; A query data cluster corresponding to a target query time range is obtained, where the query data cluster is a first data cluster or a second data cluster.

8. A data management device, characterized in that: The data management device includes: A partitioning module is configured to obtain operation and maintenance data obtained by monitoring a target device; obtain a number of data buckets associated with a target data dimension as a target number; and partition the operation and maintenance data to form the target number of data buckets; A determination module, configured to determine operation and maintenance data whose data correlation between the data in the data bucket and its adjacent data buckets meets the expected correlation, as target data; A storage module is used to construct a characterization data set that characterizes the operation and maintenance data; wherein the characterization data set includes the initial data, end data and target data of the operation and maintenance data; store the operation and maintenance data in a first data cluster, and store the characterization data set in a second data cluster.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images