Data processing method, device and system

Abstract

The present invention discloses a data processing method, device and system, which relates to the field of big data technology, wherein the method comprises: obtaining performance monitoring data in a monitoring cycle, and generating a performance indicator chart of the performance monitoring data according to a predetermined rule; performing quantitative analysis on the obtained performance monitoring data based on a preset threshold, and determining a key period in the monitoring cycle according to the quantitative analysis result; performing classification processing on the key performance indicator chart corresponding to the key period based on image analysis technology, and determining the chart type of the key performance indicator chart and the performance indicator attribute corresponding to the image type according to the classification processing result; and associating and storing the key period, the chart type of the key performance indicator chart and the performance indicator attribute corresponding to the image type, so as to facilitate the subsequent chart restoration operation. Through the present invention, the efficiency of subsequent data analysis can be improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device and system. Background Art

[0002] With the continuous enrichment of banking services and the surge in the number of users brought by Internet finance, the scale of banking systems has continued to grow. Especially under the distributed transformation, the number of nodes has continued to grow, and the amount of monitoring data generated by massive nodes has expanded dramatically. The existing data archiving, storage, and analysis methods can no longer meet the needs under cost control. There are three specific problems:

[0003] 1. The amount of data is large, and the storage of historical data puts a great cost pressure on data storage.

[0004] 2. Due to the large amount of existing data, the cost of obtaining valuable information from a large amount of data is high, including decompression and retrieval costs.

[0005] 3. Currently, data is mainly stored in the form of images, which ignores many details and is difficult to quantify and analyze. Summary of the invention

[0006] In view of this, the present invention provides a data processing method, device and system to solve at least one of the above-mentioned problems.

[0007] According to a first aspect of the present invention, there is provided a data processing method, the method comprising:

[0008] Acquire performance monitoring data in a monitoring cycle, and generate performance indicator charts of the performance monitoring data according to predetermined rules;

[0009] Perform quantitative analysis on the acquired performance monitoring data based on pre-set thresholds, and determine the key time periods in the monitoring cycle based on the quantitative analysis results;

[0010] Classify the key performance indicator chart corresponding to the key period based on the image analysis technology, and determine the chart type of the key performance indicator chart and the performance indicator attribute corresponding to the image type according to the classification result;

[0011] The key time period, the chart type of the key performance indicator chart, and the performance indicator attribute corresponding to the image type are associated and stored to facilitate subsequent chart restoration operations.

[0012] According to a second aspect of the present invention, there is provided a data processing device, the device comprising:

[0013] A data acquisition unit, used to acquire performance monitoring data in a monitoring cycle;

[0014] A chart generating unit, used to generate a performance indicator chart of the performance monitoring data according to a predetermined rule;

[0015] A key period determination unit, configured to perform quantitative analysis on the acquired performance monitoring data based on a preset threshold value, and determine a key period in a monitoring cycle according to the quantitative analysis result;

[0016] A classification unit, used for classifying the key performance indicator charts corresponding to the key time period based on image analysis technology;

[0017] a chart type determination unit, which determines the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type according to the classification processing result;

[0018] The storage unit is used to store the key time period, the chart type of the key performance indicator chart, and the performance indicator attribute corresponding to the image type in association with each other, so as to facilitate the subsequent chart restoration operation.

[0019] According to a third aspect of the present invention, there is provided a data processing system, the system comprising the above-mentioned data processing device, and a database, the database being used to store the performance monitoring data acquired by the data processing device, as well as the associated stored key time periods, chart types of key performance indicator charts, and performance indicator attributes corresponding to the image type.

[0020] According to a fourth aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0021] According to a fifth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0022] It can be seen from the above technical solution that a performance indicator chart is generated according to the acquired performance monitoring data, and the acquired performance monitoring data is quantitatively analyzed based on a preset threshold to determine the key time period. Subsequently, the key performance indicator chart corresponding to the key time period is classified based on the image analysis technology, and the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type are determined according to the classification processing result. Then, the key time period, the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type are associated and stored to facilitate subsequent chart restoration operations. In this way, by extracting key information from the data and storing it in an associated manner, early analysis of the monitoring data is achieved, which reduces the pressure on data storage and improves the efficiency of subsequent data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention;

[0025] Figure 2 is a detailed flow chart of a data processing method according to an embodiment of the present invention;

[0026] Figure 3 is a structural block diagram of a data processing system according to an embodiment of the present invention;

[0027] Figure 4 is a structural block diagram of a data processing device according to an embodiment of the present invention;

[0028] Figure 5 is an exemplary structural block diagram of a data processing system according to an embodiment of the present invention;

[0029] Figure 6 It is a schematic block diagram of the system structure of the electronic device 600 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Since the existing system monitoring data volume is large, the data storage pressure is large, and there is also the problem of low data analysis efficiency. Based on this, an embodiment of the present invention provides a data processing solution, which extracts important fragments from the data and records key feature values ​​to achieve early analysis of monitoring data, while reducing data storage pressure and providing accurate quantitative data support for subsequent data analysis, thereby improving data analysis efficiency. The following describes the embodiment of the present invention in detail with reference to the accompanying drawings.

[0032] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0033] Step 101, acquiring performance monitoring data in a monitoring period, and generating a performance indicator chart of the performance monitoring data according to a predetermined rule.

[0034] The performance monitoring data here involves CPU (Central Processing Unit) usage, memory usage, IOPS (Input / Output Operations Per Second), etc.

[0035] In actual operation, multiple performance monitoring data can be obtained and respective performance indicator charts can be generated.

[0036] The above-mentioned predetermined rule may be to use predetermined chart generating software to generate the chart. The chart generating software may be selected according to the actual operation situation, and the present invention does not impose any limitation on this.

[0037] Step 102: Perform quantitative analysis on the acquired performance monitoring data based on a preset threshold value, and determine a key period in the monitoring cycle according to the quantitative analysis result.

[0038] Step 103: classify the key performance indicator chart corresponding to the key period based on image analysis technology, and determine the chart type of the key performance indicator chart and the performance indicator attribute corresponding to the image type according to the classification result.

[0039] The chart type here can be determined based on the shape of the chart. For example, the chart types include: flat top, wave, burr, and others.

[0040] Flat tops mainly refer to the type of performance indicators that remain at a certain level for a period of time. Waves mainly refer to the type of performance indicators that change slowly and obviously in a periodic manner. Burrs mainly refer to the type of performance indicators that fluctuate up and down with high frequency and large amplitude.

[0041] In the process of classifying the chart, the key performance indicator chart can be firstly analyzed based on the image analysis technology, and then the key performance indicator chart after the initial analysis is combined with the quantitative analysis result to classify the key performance indicator chart. In this way, the accuracy of chart classification can be improved.

[0042] Step 104 : The key period, the chart type of the key performance indicator chart, and the performance indicator attribute corresponding to the image type are stored in association with each other to facilitate the subsequent chart restoration operation.

[0043] A performance indicator chart is generated according to the acquired performance monitoring data, and a key period is determined by quantitatively analyzing the acquired performance monitoring data based on a preset threshold. Then, the key performance indicator chart corresponding to the key period is classified based on image analysis technology, and the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type are determined according to the classification processing result. Then, the key period, the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type are associated and stored to facilitate subsequent chart restoration operations. In this way, by extracting key information from the data and storing it in an associated manner, early analysis of the monitoring data is achieved, which reduces the pressure on data storage and improves the efficiency of subsequent data analysis.

[0044] In actual operation, the performance monitoring data acquired in step 101 may also be stored in a database based on the node and type to which the data belongs, to facilitate subsequent data query.

[0045] When performing a chart restoration operation, you can first obtain the associated stored key time period, chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type; then, according to the acquired associated stored information, perform a chart restoration operation based on the graphic template corresponding to the chart type.

[0046] Specifically, according to the acquired associated stored information, the chart restoration operation for the critical period can be performed based on the graphic template corresponding to the chart type; then, according to the preset threshold value corresponding to the image type, the chart restoration operation for the non-critical period is performed in combination with the restored chart for the critical period.

[0047] In order to better understand the present invention, the following Figure 2 The detailed data processing flow shown is used to describe the embodiment of the present invention.

[0048] like Figure 2 As shown, the detailed data processing flow includes the following steps 201-207:

[0049] Step 201, collect traditional performance capacity monitoring data, and collect performance capacity monitoring data at the system level and application service level through the Agent deployed on the node. Send it to the storage module in the form of streaming data. The storage module sends the received data to different processing nodes according to the node to which the data belongs and the data type through the processing cluster therein, and processes the monitoring time-sharing data (i.e., the value corresponding to the monitoring item) into the database as the data basis for subsequent quantitative analysis.

[0050] Step 202, based on the monitoring data stored in the storage module (which can be called hotspot monitoring data), draw performance indicator charts based on real monitoring time series data in units of days, mainly including: CPU usage chart, memory usage chart, IOPS chart, transaction volume chart, etc. The above charts are all standard two-dimensional curve charts, which can effectively reuse analysis methods. These charts can serve as the data basis for subsequent graphic analysis.

[0051] Step 203, based on the time-sharing data in the database, quantitative analysis is performed, mainly based on the preset threshold value to distinguish between critical time periods and non-critical time periods. Taking CPU usage as an example, the threshold value is 20% usage. If the time-sharing data does not exceed the threshold value within 1 timing cycle (i.e., monitoring cycle, the timing cycle is not the monitoring sampling rate, and its selection is based on efficiency and storage comprehensive considerations, generally set at 1-4 hours), then the period is assumed to be a non-critical period and will not be output to the subsequent analysis stage. Only data records exceeding 20% ​​will flow into the next analysis stage.

[0052] Step 204, based on the performance indicator chart generated in step 202, a graphic analysis is performed, and the performance indicator chart is classified into types such as flat top, wave, burr, and other types through the image analysis technology of pattern recognition. Since the classification accuracy of the chart by the image analysis technology is low, it is preferably combined with the quantized time-sharing data for secondary classification. That is to say, after the image analysis technology analyzes and classifies the chart, it is necessary to verify it through the quantized monitoring data (time-sharing data).

[0053] For example, a flat-top chart must meet the requirement that 80% of the points change within plus or minus 10% of the mean. A flat top can be understood as a trapezoid with a very long top, so the change should be very close to the mean. 10% is currently an empirical value.

[0054] Wave-type charts must satisfy the Fourier curve fitting correlation coefficient greater than 0.7. Waves can be understood as graphs that approximate wave changes, with obvious peaks and troughs. The quantitative verification method is to use regular curve fitting. The correlation coefficient of 0.7 is a strongly correlated comparison result we obtained in practice.

[0055] The charts belonging to the glitch type need to meet the requirements that the change between two adjacent sampling points is higher than 30% of the maximum value and appears at least 3 times in one timing cycle. The glitch can be understood as a frequent and large-scale change in the graph, showing the shape of many thorns. Therefore, the set rule is that the change between two data collection points is higher than 30% and must appear repeatedly, because the occasional glitch has little reference value. The remaining charts that cannot be classified are classified as other categories.

[0056] Step 205, based on the chart type analyzed in step 204, define different attributes and characteristic values ​​for each chart type, and store the attributes and characteristic values ​​in the private area of ​​the corresponding chart type. The public area includes: date, peak, trough, average, peak time point and other public attributes. The private area contains different attributes according to the chart type. Taking burrs as an example, the private area includes: time period, number of burr occurrences, average burr height, burr height standard deviation (used to determine whether it is regular), time interval standard deviation (used to determine whether it occurs randomly or at a fixed frequency). Based on the above attributes and characteristic values, the corresponding scalar is obtained from the time series results to generate a complete monitoring profile (attribute information, including: attributes and characteristic values) object, for example, the format is JSON (Java Script Object Notation, JS object notation)), and the monitoring profile object is:

[0057]

[0058]

[0059] In actual operation, each object contains one type of monitoring information of one node in units of days.

[0060] Step 206, based on the monitoring profile object generated in step 205, save it to a NoSQL (non-relational database) database based on key:value. Date, node, and monitoring data type are used as key values, and profile object is used as value. After the analysis is completed, the time series data actually monitored is marked as cleanable, and then cleaned up regularly through batch processing.

[0061] Step 207, when it is necessary to trace back the performance capacity, the process of restoring the performance chart is performed based on the monitoring profile object stored in step 205. In the process, the templates of flat top, wave and burr graphics are mainly used to dynamically generate the templates in combination with the attributes and feature values ​​in the private area, and the non-critical time intervals are supplemented with the set thresholds and returned to the front end for user analysis.

[0062] For example, the monitoring profile object is as follows:

[0063]

[0064]

[0065] The monitoring profile object shown above indicates that there is only one period of time that needs attention in the whole day. When restoring the performance chart, first restore the chart according to the content in interval1. For example, based on the stored information, it is found that it is a chart with a regular burr pattern. According to the "period length / number of burr occurrences" as the period, the points where the burr is located are drawn on the two-dimensional chart, and the threshold (20%) is used as the filling value for the non-burr period to restore the monitoring graph. For the time that is not in the interval1 period, it belongs to the low-value interval, which can be directly filled with 0.8 times the threshold (16%, which is an empirical value).

[0066] The embodiment of the present invention analyzes, abstracts, and aggregates the features of the monitoring data obtained from the system and stores them as a processing basis for subsequent analysis, thereby reducing data storage pressure while improving the efficiency of subsequent data analysis.

[0067] Based on similar inventive concepts, an embodiment of the present invention further provides a data processing system, such as Figure 3 As shown, the system includes: a data processing device 31 and a database 32, wherein the data processing device 31 can preferably be used to implement the process in the above method embodiment; the database 32 is used to store the performance monitoring data obtained by the data processing device, as well as the key time periods stored in association, the chart types of the key performance indicator charts and the performance indicator attributes corresponding to the image type.

[0068] In order to clearly describe the embodiment of the present invention, the data processing device 31 is described in detail below.

[0069] Figure 4 is a structural block diagram of the data processing device 31, such as Figure 4 As shown, the data processing device 31 includes: a data acquisition unit 311, a chart generation unit 312, a key period determination unit 313, a classification unit 314, a chart type determination unit 315, and a storage unit 316, wherein:

[0070] The data acquisition unit 311 is used to acquire performance monitoring data in a monitoring period.

[0071] The chart generating unit 312 is used to generate a performance indicator chart of the performance monitoring data according to a predetermined rule.

[0072] The critical period determination unit 313 is used to perform quantitative analysis on the acquired performance monitoring data based on a preset threshold value, and determine the critical period in the monitoring cycle according to the quantitative analysis result.

[0073] The classification unit 314 is used to classify the key performance indicator charts corresponding to the key time period based on image analysis technology.

[0074] Specifically, the classification unit 314 includes: a primary analysis processing module and a classification module, wherein:

[0075] A primary analysis and processing module, used for performing primary analysis and processing on the key performance indicator chart based on image analysis technology;

[0076] The classification module is used to classify the key performance indicator chart after the initial analysis and the key performance indicator chart in combination with the quantitative analysis result.

[0077] The chart type determination unit 315 determines the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type according to the classification processing result.

[0078] The chart type here can be determined based on the shape of the chart. For example, the chart types include: flat top, wave, burr, and others.

[0079] The storage unit 316 is used to store the key period, the chart type of the key performance indicator chart, and the performance indicator attribute corresponding to the image type in association with each other, so as to facilitate the subsequent chart restoration operation.

[0080] A performance indicator chart is generated according to the performance monitoring data acquired by the data acquisition unit 311 through the chart generation unit 312. The key period determination unit 313 determines the key period by quantitatively analyzing the acquired performance monitoring data based on a preset threshold. Then, the classification unit 314 classifies the key performance indicator chart corresponding to the key period based on the image analysis technology. The chart type determination unit 315 determines the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type according to the classification processing result. Then, the storage unit 316 associates and stores the key period, the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type to facilitate the subsequent chart restoration operation. In this way, by extracting key information from the data and storing it in association, early analysis of the monitoring data is achieved, which reduces the pressure on data storage and improves the efficiency of subsequent data analysis.

[0081] In one embodiment, the data processing device 31 further includes: a key information acquisition unit and a chart restoration unit, wherein:

[0082] A key information acquisition unit, used to acquire the key time period and chart type of the key performance indicator chart stored in association and the performance indicator attribute corresponding to the image type;

[0083] The chart restoration unit is used to perform a chart restoration operation based on the graphic template corresponding to the chart type according to the acquired associated stored information. Specifically, the chart restoration operation for the key period can be performed based on the graphic template corresponding to the chart type according to the acquired associated stored information; and then the chart restoration operation for the non-key period can be performed in combination with the restored chart for the key period according to a preset threshold value corresponding to the image type.

[0084] Through the above-mentioned key information acquisition unit and chart restoration unit, historical charts can be restored and analyzed.

[0085] In actual operation, the data processing device 31 may also store the performance monitoring data acquired by the data acquisition unit 311 in a database based on the node and type to which the data belongs, to facilitate subsequent query and analysis.

[0086] The specific execution process of the above-mentioned units and modules can be found in the description of the above-mentioned method embodiment, which will not be repeated here.

[0087] In actual operation, the above-mentioned units and modules may be provided in combination or individually, and the present invention is not limited thereto.

[0088] Figure 5 is an exemplary structural block diagram of a data processing system according to an embodiment of the present invention. Figure 5 As shown, the exemplary system includes: an analysis unit 1, a storage unit 2 and a monitoring chart generation unit 3, wherein:

[0089] The analysis unit includes: a graphic analysis unit and a quantitative analysis unit, which are mainly used to analyze the collected graphical monitoring charts and quantitative monitoring data. Graphic analysis mainly classifies the graphics, designs different elements according to the classification, extracts the corresponding indicators and features from the collected monitoring charts and data, and forms an analyzed monitoring profile.

[0090] The storage unit includes: a hot data module and a historical data module. The hot data module contains the original monitoring charts and monitoring data. The data stored in the historical data module is classified in the analysis unit. The analysis results of different classifications contain different characteristics. The module is responsible for the storage and efficient retrieval of analysis results of different characteristics.

[0091] The monitoring chart generation unit includes: a hotspot chart module 301 and a historical chart module 302, which are mainly responsible for drawing monitoring charts for front-end display, and need to support the drawing of hotspot data and historical data at the same time.

[0092] In order to further understand this example, each unit is described in detail below.

[0093] (1) Analysis Unit 1

[0094] Analysis unit 1 is a component for monitoring data analysis and processing. It obtains the monitored quantitative data from the storage unit, judges the importance and validity of the data through thresholds, and distinguishes between critical periods and non-critical periods. At the same time, it obtains the monitoring curve graph from the hotspot chart module of the monitoring chart generation unit, intercepts the critical period for graphic analysis, and combines the results of graphic analysis and quantitative analysis to classify the monitoring results. The categories include: flat top, wave, burr, and others. Flat top mainly refers to the type in which the performance indicator is maintained at a certain level for a period of time; wave mainly refers to the type in which the performance indicator obviously changes periodically and slowly; burr mainly refers to the type in which the performance indicator fluctuates up and down with high frequency and large amplitude; and other types that cannot be classified into the first three categories. Different types correspond to different private data areas, and different private data areas store different attributes and characteristics. According to the unique attributes and characteristics, the corresponding scalar is extracted from the results of quantitative analysis to generate a monitoring profile for the interval.

[0095] (2) Storage unit 2

[0096] Storage unit 2 is mainly used to store performance and capacity monitoring data, which is divided into hot data module and historical data module. The hot data module stores various performance and capacity indicators in different time periods (which can be stored in a time series database, such as influxdb). Currently, it can support relational tables and non-relational key: value modes to archive monitoring time series data within 30 days. The historical data module is mainly used to store the monitoring profile output by the analysis unit, which can be stored in a non-relational key: value mode.

[0097] (3) Monitoring chart generation unit 3

[0098] The hotspot chart module 301 mainly determines the time-sharing points in the coordinates according to the (time point + monitoring scalar) mode based on the monitoring hotspot data stored in the storage unit within 30 days, and then fits the points to obtain the hotspot chart. That is, according to the time-sharing monitoring data stored in the time series database, each point in the coordinate graph (such as CPU usage, IOPS) is restored, and then these points are fitted through tool drawing to form a relatively smooth monitoring chart. The graphics generated by the hotspot chart module are all obtained from real monitoring data, which is an accurate production performance capacity chart for recent performance capacity analysis and backtracking.

[0099] The historical chart module 302 mainly generates historical monitoring charts based on the monitoring profile stored in the storage unit. The chart is generated based on the attributes and features in the profile and the graphic template corresponding to the chart type to restore the chart. The graphics generated by the historical chart module are all generated based on the feature values ​​in the template and profile, which is an approximate restoration of the actual production performance and capacity, and is used for historical data backtracking and long-term trend analysis.

[0100] The performance monitoring data analysis and storage solution proposed in the embodiment of the present invention realizes the early analysis of the monitoring data by extracting important fragments and recording key feature values, reduces storage usage and provides accurate quantitative data support for subsequent data analysis, thereby improving analysis efficiency and accuracy. At the same time, the monitoring graph can be approximately restored through the recorded key features, which is convenient for the user's analysis and diagnosis.

[0101] This embodiment also provides an electronic device, which may be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited thereto. In this embodiment, the electronic device may be implemented with reference to the above method embodiment and the data processing device / system embodiment, the contents of which are incorporated herein, and the repeated parts are not repeated.

[0102] Figure 6 FIG. 6 is a schematic block diagram of a system structure of an electronic device 600 according to an embodiment of the present invention. Figure 6 As shown, the electronic device 600 may include a central processor 100 and a memory 140; the memory 140 is coupled to the central processor 100. It should be noted that this figure is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0103] In one embodiment, the data processing function may be integrated into the central processing unit 100. The central processing unit 100 may be configured to perform the following control:

[0104] Acquire performance monitoring data in a monitoring cycle, and generate performance indicator charts of the performance monitoring data according to predetermined rules;

[0105] Perform quantitative analysis on the acquired performance monitoring data based on pre-set thresholds, and determine the key time periods in the monitoring cycle based on the quantitative analysis results;

[0106] Classify the key performance indicator chart corresponding to the key period based on the image analysis technology, and determine the chart type of the key performance indicator chart and the performance indicator attribute corresponding to the image type according to the classification result;

[0107] The key time period, the chart type of the key performance indicator chart, and the performance indicator attribute corresponding to the image type are associated and stored to facilitate subsequent chart restoration operations.

[0108] From the above description, it can be seen that the electronic device provided in the embodiment of the present application generates a performance indicator chart based on the acquired performance monitoring data, and at the same time, determines the key time period by quantitatively analyzing the acquired performance monitoring data based on a preset threshold, and then classifies the key performance indicator chart corresponding to the key time period based on image analysis technology, and determines the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type according to the classification processing result, and then stores the key time period, the chart type of the key performance indicator chart, and the performance indicator attributes corresponding to the image type in association with each other to facilitate subsequent chart restoration operations. In this way, by extracting key information from the data and storing it in association, early analysis of the monitoring data is achieved, which reduces the pressure on data storage and improves the efficiency of subsequent data analysis.

[0109] In another embodiment, the data processing device / system may be configured separately from the central processing unit 100. For example, the data processing device / system may be configured as a chip connected to the central processing unit 100, and the data processing function is implemented under the control of the central processing unit.

[0110] like Figure 6 As shown, the electronic device 600 may further include: a communication module 110, an input unit 120, an audio processing unit 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily have to include Figure 6 In addition, the electronic device 600 may also include Figure 6 For components not shown, reference may be made to the prior art.

[0111] like Figure 6 As shown, the central processor 100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The central processor 100 receives inputs and controls the operations of various components of the electronic device 600.

[0112] The memory 140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 100 may execute the program stored in the memory 140 to implement information storage or processing.

[0113] The input unit 120 provides input to the CPU 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0114] The memory 140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 140 may also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 600 through the central processor 100.

[0115] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0116] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processor 100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0117] Based on different communication technologies, multiple communication modules 110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide an audio output via the speaker 131 and receive an audio input from the microphone 132, thereby realizing a common telecommunication function. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 130 is also coupled to the central processor 100, so that the sound can be recorded on the local machine through the microphone 132, and the sound stored on the local machine can be played through the speaker 131.

[0118] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0119] In summary, the embodiment of the present invention provides a data processing solution. Since the amount of performance monitoring data is large and the value of a single piece of data is low, the life cycle management of historical data through this solution can effectively save costs. At the same time, this solution extracts key summary information from the monitoring data in advance, which is convenient for subsequent analysis and processing, improves the retrieval speed and processing efficiency, and by using a combination of image analysis and quantitative analysis, it can also avoid the problems of inaccurate image analysis and unintuitive quantitative analysis.

[0120] The preferred embodiments of the present invention are described above with reference to the accompanying drawings. Many features and advantages of these embodiments are clear from this detailed description, and therefore the claims are intended to cover all such features and advantages of these embodiments that fall within their true spirit and scope. In addition, since many modifications and changes are easily conceivable to those skilled in the art, it is not intended to limit the embodiments of the present invention to the precise structure and operation illustrated and described, but all suitable modifications and equivalents falling within its scope may be covered.

[0121] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire performance monitoring data in a monitoring cycle, and generate performance indicator charts of the performance monitoring data according to predetermined rules; the performance indicator charts include: CPU usage chart, memory usage chart, IOPS chart and transaction volume chart; Perform quantitative analysis on the acquired performance monitoring data based on pre-set thresholds, and determine the key time periods in the monitoring cycle based on the quantitative analysis results; Classify the key performance indicator chart corresponding to the key period based on the image analysis technology, and determine the chart type of the key performance indicator chart and the performance indicator attribute corresponding to the image type according to the classification result; storing the key period, the chart type of the key performance indicator chart, and the performance indicator attribute corresponding to the image type in association with each other, so as to facilitate a subsequent chart restoration operation; The chart restoration operation includes: Obtaining the chart type of the key period and key performance indicator chart associated with the storage and the performance indicator attributes corresponding to the image type; According to the acquired associated stored information, a chart restoration operation is performed based on a graphic template corresponding to the chart type, including: According to the acquired associated stored information, a chart restoration operation for a key period is performed based on a graphic template corresponding to the chart type; According to a preset threshold value corresponding to the image type, a chart restoration operation for a non-critical period is performed in combination with the restored chart for the critical period.

2. The method according to claim 1, characterized in that Classifying the key performance indicator charts corresponding to the key time period based on the image analysis technology includes: Performing initial analysis and processing on the key performance indicator chart based on image analysis technology; The key performance indicator chart after the initial analysis is combined with the quantitative analysis result to classify the key performance indicator chart.

3. The method according to claim 1, characterized in that: After acquiring the performance monitoring data in the monitoring cycle, the method further includes: The acquired performance monitoring data is stored in the database based on the node and type to which the data belongs.

4. A data processing device, characterized in that: The device comprises: A data acquisition unit, used to acquire performance monitoring data in a monitoring cycle; A chart generating unit, used to generate a performance indicator chart of the performance monitoring data according to a predetermined rule; the performance indicator chart includes: a CPU usage chart, a memory usage chart, an IOPS chart and a transaction volume chart; A key period determination unit, configured to perform quantitative analysis on the acquired performance monitoring data based on a preset threshold value, and determine a key period in a monitoring cycle according to the quantitative analysis result; A classification unit, used for classifying the key performance indicator charts corresponding to the key time period based on image analysis technology; a chart type determination unit, which determines the chart type of the key performance indicator chart and the performance indicator attributes corresponding to the image type according to the classification processing result; A storage unit, used to associate and store the key period, the chart type of the key performance indicator chart, and the performance indicator attribute corresponding to the image type, so as to facilitate a subsequent chart restoration operation; A key information acquisition unit, used to acquire the key time period and chart type of the key performance indicator chart stored in association and the performance indicator attribute corresponding to the image type; A chart restoring unit, configured to perform a chart restoring operation based on a graphic template corresponding to the chart type according to the acquired associated stored information; The chart restoration unit is specifically used for: According to the acquired associated stored information, a chart restoration operation for a key period is performed based on a graphic template corresponding to the chart type; According to a preset threshold value corresponding to the image type, a chart restoration operation for a non-critical period is performed in combination with the restored chart for the critical period.

5. The device according to claim 4, characterized in that The taxonomic units include: A primary analysis and processing module, used for performing primary analysis and processing on the key performance indicator chart based on image analysis technology; The classification module is used to classify the key performance indicator chart after the initial analysis and the key performance indicator chart in combination with the quantitative analysis result.

6. A data processing system, characterized in that: The system includes: a data processing device as described in any one of claims 4 to 5, and a database, wherein the database is used to store the performance monitoring data acquired by the data processing device, as well as the associated stored key time periods, chart types of key performance indicator charts, and performance indicator attributes corresponding to the image type.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images