Data storage optimization method, device, electronic device and storage medium

Abstract

The present invention discloses a data storage optimization method, device, electronic device and storage medium. By acquiring multiple pieces of collected data in a data collector in real time, and instructing a time series database to store each piece of collected data; instructing an aggregation computing engine to acquire each piece of collected data from a time series database, and performing aggregating operations on the collected data according to a pre-set joint aggregation rule, to obtain data aggregation results of different granularities; instructing the time series database to acquire each piece of data aggregation results, and storing each piece of data aggregation results in each time granularity table respectively; instructing the time series database to perform data compression processing on each time granularity table, to obtain a data aggregation storage result, and to complete the storage optimization operation of the collected data according to the data aggregation storage result. The problem of large data volume and inconvenient query caused by directly storing the original data collected is solved, the data storage cost is reduced, and the efficiency of fast data aggregation and accelerated data query is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data storage optimization method, device, electronic device and storage medium. Background Art

[0002] With the advent of the information age, more and more data has appeared. Especially in the process of processing page data, data query and data storage are also needed. How to reasonably store data, save storage space and improve data query rate is crucial.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following defects: At present, most of the data stored in the database is raw data, and the raw data is aggregated and stored again. The double storage leads to too much data and slow query speed. In addition, the historical data stored in the database is generally rarely used, but it occupies a huge storage cost. Summary of the invention

[0004] The present invention provides a data storage optimization method, device, electronic device and storage medium, so as to reduce the data storage cost and improve the efficiency of fast data aggregation and accelerated data query.

[0005] According to one aspect of the present invention, a data storage optimization method is provided, which includes:

[0006] Acquire multiple pieces of collected data from the data collector in real time, and instruct the time series database to store each of the collected data;

[0007] Instructing the aggregation calculation engine to obtain each of the collected data from the time series database, and to perform an aggregation operation on the collected data according to a preset joint aggregation rule, so as to obtain data aggregation results of different granularities;

[0008] Instruct the time series database to obtain each data aggregation result, and store each data aggregation result in each pre-constructed time granularity table;

[0009] The time series database is instructed to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

[0010] According to another aspect of the present invention, there is provided a data storage optimization device, comprising:

[0011] A data acquisition module is used to acquire multiple pieces of collected data in the data collector in real time, and instruct the time series database to store each of the collected data;

[0012] A data aggregation result determination module is used to instruct the aggregation calculation engine to obtain each of the collected data from the time series database, and perform an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities;

[0013] A data aggregation result storage module is used to instruct the time series database to obtain each data aggregation result and store each data aggregation result in each pre-constructed time granularity table;

[0014] The data aggregation storage result determination module is used to instruct the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data storage optimization method described in any embodiment of the present invention when executing the computer program.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data storage optimization method described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention obtains multiple collected data in the data collector in real time, and instructs the preset time series database to store each of the collected data; instructs the preset aggregation calculation engine to obtain each of the collected data from the time series database, and performs the collection data aggregation operation according to the preset joint aggregation rules to obtain data aggregation results of different granularities; instructs the time series database to obtain each data aggregation result, and stores each data aggregation result in each pre-constructed time granularity table; instructs the time series database to perform data compression processing on each time granularity table to obtain data aggregation storage results, so as to complete the storage optimization operation of the collected data according to the data aggregation storage results. The problem of large data volume and inconvenient query caused by directly storing the original data of the collected data is solved, the data storage cost is reduced, and the efficiency of fast data aggregation and accelerated data query is improved.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flow chart of a data storage optimization method provided according to Embodiment 1 of the present invention;

[0021] Figure 2 is a structural schematic diagram of a data storage optimization device provided according to Embodiment 2 of the present invention;

[0022] Figure 3 It is a structural schematic diagram of an electronic device provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Embodiment 1

[0026] Figure 1 A flowchart of a data storage optimization method is provided for embodiment 1 of the present invention. This embodiment is applicable to situations where storage optimization and query of collected data are performed. The method can be executed by a data storage optimization device, and the data storage optimization device can be implemented in the form of hardware and / or software.

[0027] Correspondingly, such as Figure 1 As shown, the method includes:

[0028] S110, acquiring multiple pieces of collected data from the data collector in real time, and instructing the time series database to store each piece of the collected data.

[0029] The data collector may be a device for collecting data, and may send the collected multiple data to a time series database.

[0030] Specifically, the time series database can receive the collected data and store the data. The time series database can be an Influx DB database, which is not specifically limited here. In detail, the collected data is stored in the first time granularity table in the time series database. Assume that the first time granularity table corresponds to a 1-minute time granularity table, that is, in the time series database, data is stored at a smaller granularity.

[0031] S120, instructing the aggregation calculation engine to obtain each of the collected data from the time series database, and perform an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities.

[0032] The aggregation computing engine may be an engine that acquires and aggregates the collected data. Specifically, the aggregation computing engine may be Apache Spark, which is not specifically limited here.

[0033] The joint aggregation rule may be a pre-set rule for aggregating the collected data, and the data aggregation result may be an aggregation result obtained by aggregating the collected data.

[0034] In this embodiment, it is assumed that the aggregation computing engine is Apache Spark and the time series database is Influx DB. The collected data in the Influx DB database is obtained through Apache Spark, and the collected data is the data in the first time granularity table.

[0035] Furthermore, the collected data can be aggregated according to the joint aggregation rules to obtain data aggregation results of different granularities.

[0036] Optionally, the joint aggregation rules include a first joint aggregation rule, a second joint aggregation rule and a third joint aggregation rule; and performing data aggregation operations according to pre-set joint aggregation rules to obtain data aggregation results of different granularities includes: performing data aggregation operations according to the first joint aggregation rule to obtain a first data aggregation result; performing data aggregation operations according to the second joint aggregation rule to obtain a second data aggregation result; and performing data aggregation operations according to the third joint aggregation rule to obtain a third data aggregation result.

[0037] The first joint aggregation rule may be a rule for aggregating data in the first time granularity table by the first time. The second joint aggregation rule may be a rule for aggregating data in the first time granularity table by the second time. The third joint aggregation rule may be a rule for aggregating data in the first time granularity table by the third time. Here, the first time is less than the second time, and the second time is less than the third time.

[0038] Exemplarily, it is assumed that the first time is 5 minutes, the second time is 30 minutes, and the third time is 60 minutes.

[0039] Specifically, for the collected data in the first time granularity table (1-minute time granularity table), data aggregation is performed every 5 minutes, and the average value of the aggregated results is taken to obtain the first data aggregation result, and the current storage time needs to be recorded and sent to the Influx DB database.

[0040] Similarly, for the collected data in the first time granularity table (1-minute time granularity table), data aggregation is performed every 30 minutes, and the average value of the aggregated results is taken to obtain the second data aggregation result, and the current storage time needs to be recorded and sent to the Influx DB database.

[0041] Correspondingly, for the collected data in the first time granularity table (1-minute time granularity table), data aggregation is performed every 60 minutes, and the average value of the aggregated results is taken to obtain the third data aggregation result, and the current storage time needs to be recorded and sent to the Influx DB database.

[0042] S130, instructing the time series database to obtain each data aggregation result, and store each data aggregation result in each pre-constructed time granularity table.

[0043] The time granularity table includes a first time granularity table, a second time granularity table, a third time granularity table and a fourth time granularity table. The time granularity of the first time granularity table is smaller than that of the second time granularity table; the time granularity of the second time granularity table is smaller than that of the third time granularity table; the time granularity of the third time granularity table is smaller than that of the fourth time granularity table.

[0044] Continuing with the previous example, the first time granularity table can be set to a 1-minute time granularity table; the second time granularity table can be set to a 5-minute time granularity table; the third time granularity table can be set to a 30-minute time granularity table; and the fourth time granularity table can be set to a 60-minute time granularity table.

[0045] Optionally, storing each data aggregation result in each pre-constructed time granularity table respectively includes: storing the first data aggregation result in the second time granularity table; storing the second data aggregation result in the third time granularity table; storing the third data aggregation result in the fourth time granularity table.

[0046] Continuing with the previous example, the first data aggregation result obtained by performing data aggregation once every 5 minutes is stored in the second time granularity table, and the corresponding current storage time needs to be recorded. The second data aggregation result obtained by performing data aggregation once every 30 minutes is stored in the third time granularity table, and the corresponding current storage time needs to be recorded. The third data aggregation result obtained by performing data aggregation once every 60 minutes is stored in the fourth time granularity table, and the corresponding current storage time needs to be recorded.

[0047] S140, instructing the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

[0048] The data aggregation storage result may be a processing result obtained by performing data compression processing on the collected data or collected aggregated data in each time granularity table.

[0049] Optionally, after instructing the time series database to perform data compression processing on each time granularity table to obtain the data aggregation storage result, it also includes: obtaining a current time granularity table in each time granularity table in turn; obtaining the current storage time corresponding to each collected data or each aggregated data in the current time granularity table, and obtaining a preset deletion policy that matches the current time granularity table; deleting each collected data or each aggregated data according to the current storage time and the current deletion policy; returning to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.

[0050] The deletion strategy may be a strategy for deleting the collected data in tables of different time granularities.

[0051] Among them, the first time granularity table corresponds to the first deletion policy; the second time granularity table corresponds to the second deletion policy; the third time granularity table corresponds to the third deletion policy; and the fourth time granularity table corresponds to the fourth deletion policy.

[0052] In this embodiment, since the data stored in the first time granularity table is unaggregated data, that is, collected data, it is necessary to obtain the current storage time corresponding to each collected data and determine whether it satisfies the first deletion policy. In addition, since the second time granularity table, the third time granularity table, and the fourth time granularity table are aggregated data, that is, aggregated data, it is necessary to obtain the current storage time corresponding to each aggregated data and determine whether it satisfies the corresponding deletion policy.

[0053] Exemplarily, it is assumed that the first deletion policy may be a policy for deleting data if the current storage time corresponding to the collected data meets 1 month. It is assumed that the second deletion policy may be a policy for deleting data if the current storage time corresponding to the aggregated data meets 3 months. It is assumed that the third deletion policy may be a policy for deleting data if the current storage time corresponding to the aggregated data meets 6 months. It is assumed that the fourth deletion policy may be a policy for deleting data if the current storage time corresponding to the aggregated data meets 12 months.

[0054] Specifically, in each of the time granularity tables, a current time granularity table is obtained in turn. Assuming that the current time granularity table is the first time granularity table, the current storage time corresponding to each collected data in the first time granularity table is obtained, and the first deletion policy (a policy for performing data deletion processing when the current storage time meets 1 month) is obtained; then, the deletion operation of each collected data is performed according to the current storage time and the current deletion policy.

[0055] In addition, assuming that the current time granularity table is the second time granularity table, obtain the current storage time corresponding to each aggregate data in the second time granularity table, and obtain the second deletion policy (a policy for data deletion if the current storage time meets 3 months); then perform deletion operations on each collected data according to the current storage time and the current deletion policy.

[0056] Optionally, the deletion operation of each collected data or each aggregated data is performed according to the current storage time and the current deletion policy, including: judging whether each current storage time can match the current deletion policy, and if so, deleting the current collected data or each aggregated data; if not, returning to execute the operation of judging whether each storage time can match the preset deletion policy, until the traversal of each collected data or each aggregated data is completed.

[0057] Continuing with the previous example, the current storage time corresponding to each collected data in the first time granularity table is obtained (assuming that the current storage time is 25 days), and the first deletion policy (a policy for data deletion if the current storage time meets 1 month). Since the current storage time does not meet the first deletion policy, the operation of determining whether each storage time can match the preset deletion policy is returned until all collected data are traversed. In addition, assuming that the current storage time is 35 days, since the current storage time meets the first deletion policy, the current collected data is deleted.

[0058] In addition, the current storage time corresponding to each aggregate data in the second time granularity table is obtained (assuming that the current storage time is 50 days), and the second deletion policy (a policy for data deletion if the current storage time meets 3 months). Since the current storage time does not meet the second deletion policy, the operation of determining whether each storage time can match the preset deletion policy is returned until all aggregate data are traversed. In addition, assuming that the current storage time is 95 days, since the current storage time meets the second deletion policy, the current aggregate data is deleted.

[0059] Optionally, after performing the deletion operation of each collected data or each aggregated data according to the current storage time and the current deletion policy, it also includes: determining whether the current time granularity table is the first time granularity table, and if so, instructing the time series database to send each collected data corresponding to the current time granularity table to the cold database; if not, returning to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.

[0060] In this embodiment, if the collected data in the first time granularity table meets the first deletion policy, it is not only necessary to delete the collected data from the time series database, but also to send each collected data to the cold database. Since the cold database is a database that is not frequently used, when it is necessary to trace the historical collected data, the collected data can be obtained from the cold database, so that data can be better stored and the flexibility of data storage can be improved.

[0061] Optionally, before the real-time acquisition of multiple collected data in the data collector, it also includes: performing point-of-sale processing on the front-end query page to obtain various historical collected data; wherein the historical collected data includes: historical query granularity and historical time range; analyzing and clustering the historical query granularity and historical time range to construct different time granularity tables, and configuring corresponding deletion strategies and joint aggregation rules for each time granularity table.

[0062] The historical query granularity may be the size of the query granularity used to describe the data collected at the historical moment. The historical time range may be the size of the time range used to describe the data collected at the historical moment.

[0063] In this embodiment, the collected data of historical moments can be obtained by performing point embedding processing on the front-end query page, and the collected data may include historical query granularity and historical time range. By performing analysis and clustering processing on the historical query granularity and historical time range corresponding to different historical collected data, the corresponding time granularity tables can be obtained according to the clustering results, and the corresponding deletion strategies and joint aggregation rules can be determined according to the various time granularity tables.

[0064] The advantage of this setting is that different time granularity tables, deletion strategies, and joint aggregation rules can be determined through analysis and clustering of historically collected data. This can be more suitable for data storage optimization of the current page and improve the efficiency and accuracy of data storage.

[0065] Optionally, after storing each data aggregation result in each pre-constructed time granularity table respectively, it also includes: obtaining the current front-end query request corresponding to the front-end query page in real time; parsing the current front-end query request to obtain the current query granularity and the current time range; querying in each of the time granularity tables according to the current query granularity and the current time range to determine the current front-end query request result, and feeding back the current front-end query request result to the front-end query page.

[0066] In this embodiment, after the data storage optimization is completed, it is necessary to perform query processing on the stored data. Among them, the current front-end query request can be a request to query the data. After parsing the current front-end query request, the corresponding current query granularity and current time range can be obtained.

[0067] Furthermore, the corresponding current query granularity and current time range can be matched in each time granularity table in the time series database to obtain the current front-end query request result, and the current front-end query request result can be fed back to the front-end query page for page display to the user, thereby improving the accuracy and speed of data query and improving the user's data query experience.

[0068] The technical solution of the embodiment of the present invention obtains multiple collected data in the data collector in real time, and instructs the preset time series database to store each of the collected data; instructs the preset aggregation calculation engine to obtain each of the collected data from the time series database, and performs the collection data aggregation operation according to the preset joint aggregation rules to obtain data aggregation results of different granularities; instructs the time series database to obtain each data aggregation result, and stores each data aggregation result in each pre-constructed time granularity table; instructs the time series database to perform data compression processing on each time granularity table to obtain data aggregation storage results, so as to complete the storage optimization operation of the collected data according to the data aggregation storage results. The problem of large data volume and inconvenient query caused by directly storing the original data of the collected data is solved, the data storage cost is reduced, and the efficiency of fast data aggregation and accelerated data query is improved.

[0069] Embodiment 2

[0070] Figure 2 This is a schematic diagram of the structure of a data storage optimization device provided in Embodiment 2 of the present invention. The data storage optimization device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or a server to implement a data storage optimization method in an embodiment of the present invention. Figure 2 As shown, the device includes: a data acquisition module 210, a data aggregation result determination module 220, a data aggregation result storage module 230 and a data aggregation storage result determination module 240.

[0071] The collected data acquisition module 210 is used to acquire multiple collected data in the data collector in real time and instruct the time series database to store each of the collected data;

[0072] The data aggregation result determination module 220 is used to instruct the aggregation calculation engine to obtain each of the collected data from the time series database, and perform an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities;

[0073] The data aggregation result storage module 230 is used to instruct the time series database to obtain each data aggregation result, and store each data aggregation result in each pre-constructed time granularity table;

[0074] The data aggregation storage result determination module 240 is used to instruct the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

[0075] The technical solution of the embodiment of the present invention obtains multiple collected data in the data collector in real time, and instructs the preset time series database to store each of the collected data; instructs the preset aggregation calculation engine to obtain each of the collected data from the time series database, and performs the collection data aggregation operation according to the preset joint aggregation rules to obtain data aggregation results of different granularities; instructs the time series database to obtain each data aggregation result, and stores each data aggregation result in each pre-constructed time granularity table; instructs the time series database to perform data compression processing on each time granularity table to obtain data aggregation storage results, so as to complete the storage optimization operation of the collected data according to the data aggregation storage results. The problem of large data volume and inconvenient query caused by directly storing the original data of the collected data is solved, the data storage cost is reduced, and the efficiency of fast data aggregation and accelerated data query is improved.

[0076] Based on the above embodiments, the time granularity table includes a first time granularity table, a second time granularity table, a third time granularity table and a fourth time granularity table.

[0077] On the basis of the above embodiments, it also includes a collection data deletion module, including: a current time granularity table acquisition unit, which can be specifically used to: after the time series database is instructed to perform data compression processing on each time granularity table and obtain the data aggregation storage result, a current time granularity table is sequentially acquired in each time granularity table; a deletion policy acquisition unit, which can be specifically used to: acquire the current storage time corresponding to each collection data or each aggregation data in the current time granularity table, and acquire a preset deletion policy matching the current time granularity table; wherein the first time granularity table corresponds to the first deletion policy; the second time granularity table corresponds to the first deletion policy; The granularity table corresponds to the second deletion strategy; the third time granularity table corresponds to the third deletion strategy; the fourth time granularity table corresponds to the fourth deletion strategy; the time granularity of the first time granularity table is smaller than that of the second time granularity table; the time granularity of the second time granularity table is smaller than that of the third time granularity table; the time granularity of the third time granularity table is smaller than that of the fourth time granularity table; the collected data deletion unit can be specifically used to: perform deletion operations on each collected data or each aggregated data according to the current storage time and the current deletion strategy; the return execution unit can be specifically used to: return to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.

[0078] Based on the above embodiments, the collected data deletion unit can be specifically used to: determine whether each current storage time can match the current deletion policy, and if so, delete the current collected data or each aggregated data; if not, return to execute the operation of determining whether each storage time can match the preset deletion policy until all collected data or each aggregated data are traversed.

[0079] On the basis of the above embodiments, it also includes a first time granularity table judgment unit, which can be specifically used to: after the deletion operation of each collected data is performed according to the current storage time and the current deletion policy, determine whether the current time granularity table is the first time granularity table, and if so, instruct the time series database to send each collected data corresponding to the current time granularity table to the cold database; if not, return to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.

[0080] On the basis of the above embodiments, the joint aggregation rule includes a first joint aggregation rule, a second joint aggregation rule and a third joint aggregation rule.

[0081] Based on the above embodiments, the data aggregation result determination module 220 can be specifically used to: perform the data aggregation operation through the first joint aggregation rule to obtain the first data aggregation result; perform the data aggregation operation through the second joint aggregation rule to obtain the second data aggregation result; perform the data aggregation operation through the third joint aggregation rule to obtain the third data aggregation result.

[0082] Based on the above embodiments, the data aggregation result storage module 230 can be specifically used to: store the first data aggregation result in the second time granularity table; store the second data aggregation result in the third time granularity table; store the third data aggregation result in the fourth time granularity table.

[0083] On the basis of the above embodiments, it also includes a time granularity table construction module, which can be specifically used for: before obtaining multiple collected data in the real-time data collector, performing point-of-sale processing on the front-end query page to obtain various historical collected data; wherein, the historical collected data includes: historical query granularity and historical time range; analyzing and clustering the historical query granularity and the historical time range to construct different time granularity tables, and configuring corresponding deletion strategies and joint aggregation rules for each time granularity table.

[0084] On the basis of the above embodiments, it also includes a current front-end query request result determination module, which can be specifically used to: after storing each data aggregation result in each pre-built time granularity table respectively, obtain the current front-end query request corresponding to the front-end query page in real time; parse the current front-end query request to obtain the current query granularity and the current time range; query in each of the time granularity tables according to the current query granularity and the current time range to determine the current front-end query request result, and feed back the current front-end query request result to the front-end query page.

[0085] The data storage optimization device provided in the embodiment of the present invention can execute the data storage optimization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0086] Embodiment 3

[0087] Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement the third embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0088] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0089] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0090] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data storage optimization method.

[0091] In some embodiments, the data storage optimization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data storage optimization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data storage optimization method in any other appropriate manner (e.g., by means of firmware).

[0092] The method includes: acquiring multiple collected data in a data collector in real time, and instructing a time series database to store each of the collected data; instructing an aggregation computing engine to acquire each of the collected data from the time series database, and perform aggregating operations on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities; instructing the time series database to acquire each data aggregation result, and store each data aggregation result in a pre-constructed time granularity table respectively; instructing the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

[0093] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0095] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0098] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0099] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0100] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0101] Embodiment 4

[0102] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a data storage optimization method when executed by a computer processor, the method comprising: acquiring multiple collected data in a data collector in real time, and instructing a time series database to store each of the collected data; instructing an aggregation computing engine to acquire each of the collected data from the time series database, and perform a collection data aggregation operation according to a preset joint aggregation rule to obtain data aggregation results of different granularities; instructing the time series database to acquire each data aggregation result, and store each data aggregation result in a pre-constructed time granularity table respectively; instructing the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

[0103] Of course, the computer-readable storage medium provided in the embodiment of the present invention has computer executable instructions that are not limited to the method operations described above, and can also execute related operations in the data storage optimization method provided in any embodiment of the present invention.

[0104] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0105] It is worth noting that in the embodiment of the above-mentioned data storage optimization device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0106] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data storage optimization method, characterized in that: include: Acquire multiple pieces of collected data from the data collector in real time, and instruct the time series database to store each of the collected data; Instructing the aggregation calculation engine to obtain each of the collected data from the time series database, and to perform an aggregation operation on the collected data according to a preset joint aggregation rule, so as to obtain data aggregation results of different granularities; The collecting data aggregation operation is performed according to the preset joint aggregation rules to obtain data aggregation results of different granularities, including: performing data aggregation according to different time granularities and different times, and taking the average value of the aggregated results, recording the current storage time, and obtaining the data aggregation results; Instruct the time series database to obtain each data aggregation result, and store each data aggregation result in each pre-constructed time granularity table; The time series database is instructed to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

2. The method according to claim 1, characterized in that The time granularity table includes a first time granularity table, a second time granularity table, a third time granularity table and a fourth time granularity table; After instructing the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, the method further includes: In each of the time granularity tables, a current time granularity table is obtained in turn; Obtaining the current storage time corresponding to each collected data or each aggregated data in the current time granularity table, and obtaining a preset deletion policy matching the current time granularity table; Among them, the first time granularity table corresponds to the first deletion policy; the second time granularity table corresponds to the second deletion policy; the third time granularity table corresponds to the third deletion policy; the fourth time granularity table corresponds to the fourth deletion policy; the time granularity of the first time granularity table is smaller than that of the second time granularity table; the time granularity of the second time granularity table is smaller than that of the third time granularity table; the time granularity of the third time granularity table is smaller than that of the fourth time granularity table; Deleting each collected data or each aggregated data according to the current storage time and current deletion policy; Return to execute the operation of sequentially acquiring a current time granularity table until all the time granularity tables are traversed.

3. The method according to claim 2, characterized in that The deleting operation of each collected data or each aggregated data according to the current storage time and the current deletion policy includes: Determine whether the current storage time matches the current deletion policy, and if so, delete the current collected data or the current aggregated data; If not, the process returns to executing the operation of judging whether each storage time can match the preset deletion policy, until all collected data or aggregated data are traversed.

4. The method according to claim 3, characterized in that After performing the deletion operation of each collected data or each aggregated data according to the current storage time and the current deletion policy, the method further includes: Determine whether the current time granularity table is the first time granularity table, and if so, instruct the time series database to send each collected data corresponding to the current time granularity table to the cold database; If no, the process returns to executing the operation of obtaining a current time granularity table in sequence until all the time granularity tables are traversed.

5. The method according to claim 4, characterized in that The joint aggregation rules include a first joint aggregation rule, a second joint aggregation rule and a third joint aggregation rule; The data aggregation operation is performed according to the preset joint aggregation rules to obtain data aggregation results of different granularities, including: Performing a collection data aggregation operation by using the first joint aggregation rule to obtain a first data aggregation result; Performing a collection data aggregation operation by using the second joint aggregation rule to obtain a second data aggregation result; Performing a collection data aggregation operation by using the third joint aggregation rule to obtain a third data aggregation result; The data aggregation results are stored in pre-built time granularity tables, including: The first data aggregation result is stored in the second time granularity table; the second data aggregation result is stored in the third time granularity table; and the third data aggregation result is stored in the fourth time granularity table.

6. The method according to claim 5, characterized in that Before the real-time acquisition of multiple pieces of collected data in the data collector, the method further includes: Perform point-of-sale processing on the front-end query page to obtain various historical collection data; The historical collection data includes: historical query granularity and historical time range; The historical query granularity and historical time range are analyzed and clustered to construct different time granularity tables, and the corresponding deletion strategy and joint aggregation rules are configured for each time granularity table.

7. The method according to claim 6, characterized in that After storing the data aggregation results in the pre-built time granularity tables respectively, the method further includes: Obtain the current front-end query request corresponding to the front-end query page in real time; Parsing the current front-end query request to obtain the current query granularity and the current time range; According to the current query granularity and the current time range, a query is performed in each of the time granularity tables to determine the current front-end query request result, and the current front-end query request result is fed back to the front-end query page.

8. A data storage optimization device, characterized in that: include: A data acquisition module is used to acquire multiple pieces of collected data in the data collector in real time, and instruct the time series database to store each of the collected data; A data aggregation result determination module is used to instruct the aggregation calculation engine to obtain each of the collected data from the time series database, and perform an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities; wherein, performing an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities includes: aggregating data according to different time granularities and different times, and taking an average value of the aggregated results, recording the current storage time, and obtaining the data aggregation result; A data aggregation result storage module is used to instruct the time series database to obtain each data aggregation result and store each data aggregation result in each pre-constructed time granularity table; The data aggregation storage result determination module is used to instruct the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data storage optimization method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data storage optimization method according to any one of claims 1 to 7 when executed.

Images