Dynamic adjustment method, system and equipment for aggregation window of database

Abstract

The invention discloses a method, a system and equipment for dynamically adjusting an aggregation window of a database, and relates to the technical field of databases. The method comprises the following steps: acquiring current flow parameters, data density and load parameters of a database and query demand data of a user; calculating an adjustment factor of the aggregation window by using a preset formula based on the flow parameter, the data density and the load parameter, and adjusting the size of the initial aggregation window of the database based on the adjustment factor to obtain the size of a target aggregation window; determining a target aggregation granularity of the aggregation window based on the query demand data; and setting a new aggregation window based on the target aggregation window size and the target aggregation granularity, and performing aggregation query by using the new aggregation window. Therefore, by dynamically adjusting the window, the aggregation window is more suitable for the real-time change of the data, and the flexibility of aggregation query is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method, system and device for dynamically adjusting an aggregation window of a database. Background Art

[0002] Aggregation query is a key function of time series databases, which is often used to count and summarize large amounts of high-frequency data. Common aggregation operations include average (AVG), maximum (MAX), minimum (MIN), count (COUNT), etc. In order to improve query efficiency, time series databases usually aggregate data in advance according to a set time window.

[0003] Currently, most time series databases use fixed windows to aggregate data when performing aggregate queries. This fixed time window aggregation mechanism means that the user or system needs to define a fixed time range in advance, such as data aggregation every 1 minute or every 5 minutes. This approach works well in static or relatively stable data flow scenarios, but when the data writing speed or query requirements change dynamically, the fixed window mechanism will have a series of problems such as being unable to adapt to real-time data changes, reduced query efficiency, waste of computing resources, waste of storage space, response delays, etc. Summary of the invention

[0004] The main purpose of the present application is to provide a method, system and device for dynamically adjusting the aggregation window of a database, so that the aggregation window can be more adaptable to real-time changes in data and the flexibility of aggregation query can be improved by dynamically adjusting the window.

[0005] To achieve the above object, the present application provides a method for dynamically adjusting an aggregation window of a database, comprising:

[0006] Obtain the database's current traffic parameters, data density, load parameters, and user query demand data;

[0007] Calculating an adjustment factor of the aggregation window based on the traffic parameter, the data density, and the load parameter using a preset formula, and adjusting an initial aggregation window size of the database based on the adjustment factor to obtain a target aggregation window size;

[0008] Determining a target aggregation granularity of the aggregation window based on the query demand data;

[0009] A new aggregation window is set based on the target aggregation window size and the target aggregation granularity, and an aggregation query is performed using the new aggregation window.

[0010] Optionally, it is characterized in that the traffic parameter includes a data writing rate, and the preset formula is:

[0011] A=w1*R+w2*D+w3*L

[0012] In the formula, A is the adjustment factor, and its value range is [-1,1]; R is the data write rate; w1 is the weight of the data write rate; D is the data density; w2 is the weight of the data density; L is the load parameter; and w3 is the weight of the load parameter.

[0013] Optionally, before using a preset formula and calculating the adjustment factor of the aggregation window based on the traffic parameter, the data density and the load parameter, the method also includes: determining the current scenario type of the database; based on the current scenario type of the database, dynamically adjusting the weight of the data write rate, the weight of the data density and the weight of the load parameter.

[0014] Optionally, the scenario types include high-frequency writing scenarios, density change scenarios and resource-scarce scenarios; the dynamic adjustment of the weight of the data write rate, the weight of the data density and the weight of the load parameter includes: if the scenario type is the high-frequency writing scenario, the weight of the data write rate is greater than the weight of the data density, and the weight of the data density is greater than the weight of the load parameter; if the scenario type is the density change scenario, the weight of the data density is greater than the weight of the data write rate, and the weight of the data write rate is greater than the weight of the load parameter; if the scenario type is the resource-scarce scenario, the weight of the load parameter is greater than the weight of the data write rate, and the weight of the data write rate is greater than the weight of the data density.

[0015] Optionally, obtaining the current data density of the database includes: obtaining the number of data points in a preset time period of the database; and determining the data density based on the preset time period and the number of data points.

[0016] Optionally, the query demand data includes a query time range; determining the target aggregation granularity of the aggregation window based on the query demand data includes: if the query time range meets the first preset range, determining the target aggregation window granularity to be the first granularity; if the query time range meets the second preset range, determining the target aggregation window granularity to be the second granularity; the second granularity is smaller than the first granularity.

[0017] Optionally, after obtaining the current traffic parameters, data density, load parameters of the database and the user's query demand data, the method further includes: determining whether a preset adjustment trigger condition is met; if it is determined that the preset adjustment trigger condition is met, using a preset formula and based on the traffic parameters, the data density and the load parameters to calculate the adjustment factor of the aggregation window; wherein the preset adjustment trigger condition includes a preset interval time and / or a change in the data write rate exceeding a preset change value.

[0018] Optionally, the current traffic parameters of the database also include current data traffic, traffic peak value within a preset time period, traffic average value within a preset time period, and total write volume within multiple consecutive unit time periods; the method also includes: judging whether the database is in a traffic peak period based on the current data traffic and the traffic peak value within the preset time period, and directing the data traffic to a low-load node of the database when it is determined that the database is in the traffic peak period; determining whether there is an anomaly in the data flow of the database based on the traffic peak value within the preset time period, the traffic average value, a first preset threshold, and a second preset threshold; wherein the first preset threshold is dynamically determined based on historical peak data, and the second preset threshold is dynamically determined based on historical average data; determining the change pattern of the total write volume based on the total write volume within each of the unit time periods, and optimizing the aggregation query strategy based on the change pattern.

[0019] In addition, to achieve the above-mentioned purpose, the present application also provides a database aggregation window dynamic adjustment system, including: an acquisition module, used to obtain the current traffic parameters, data density, load parameters of the database and the user's query demand data; a window size adjustment module, used to use a preset formula and calculate the adjustment factor of the aggregation window based on the traffic parameters, the data density and the load parameters, and adjust the initial aggregation window size of the database based on the adjustment factor to obtain the target aggregation window size; a window granularity determination module, used to determine the target aggregation granularity of the aggregation window based on the query demand data; an aggregation query module, used to set a new aggregation window based on the target aggregation window size and the target aggregation granularity, and use the new aggregation window to perform aggregation query.

[0020] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for dynamically adjusting the aggregation window of the database as described in any one of the above is implemented.

[0021] The method for dynamically adjusting the aggregation window of a database of the present application obtains the current traffic parameters, data density, load parameters of the database and the query demand data of the user; then uses a preset formula and calculates the adjustment factor of the aggregation window based on the traffic parameters, data density and load parameters, and adjusts the initial aggregation window size of the database based on the adjustment factor, so that the aggregation window size adapts to the current state of the database and the changes in the data flow; then determines the target aggregation granularity of the aggregation window based on the query demand data, so that the granularity of the aggregation window is more adapted to user needs; finally, sets a new aggregation window based on the target aggregation window size and the target aggregation granularity, and uses the new aggregation window to perform aggregation query, thereby making the aggregation window more adaptable to real-time changes in data and improving the flexibility of aggregation query. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is one of the flow charts of the method for dynamically adjusting the aggregation window of a database in an embodiment of the present application;

[0023] Figure 2 This is the second flowchart of the method for dynamically adjusting the aggregation window of a database in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a database aggregation window dynamic adjustment system according to an embodiment of the present application;

[0025] Figure 4 An example of a physical structure diagram of an electronic device is provided;

[0026] In the figure, 310, acquisition module; 320, window size adjustment module; 330, window granularity determination module; 340, aggregation query module; 410, processor; 420, communication interface; 430, memory; 440, communication bus.

[0027] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] Time-Series Database (TSDB) is a database specifically designed to store and manage serialized data that is generated over time. With the widespread application of the Internet of Things (IoT), smart manufacturing, financial market data, industrial monitoring and other fields, the scale and speed of data processed by time-series databases are increasing.

[0030] Aggregation query is a key function of time series databases, and is often used to count and summarize large amounts of high-frequency data. Common aggregation operations include average (AVG), maximum (MAX), minimum (MIN), count (COUNT), etc. In order to improve query efficiency, time series databases usually aggregate data in advance according to a set time window, such as 1 minute, 5 minutes, 1 hour, etc.

[0031] Currently, most time series databases use fixed windows to aggregate data when performing aggregate queries. This fixed time window aggregation mechanism means that the user or system needs to define a fixed time range in advance, such as aggregating data every 1 minute or every 5 minutes. This approach works well in static or relatively stable data flow scenarios, but when the data writing speed or query requirements change dynamically, the fixed window mechanism will expose the following limitations:

[0032] 1. Query performance bottleneck: Fixed window size cannot be adjusted dynamically according to real-time changes in data. For example, when the data writing speed suddenly increases, aggregation operations using a longer time window may cause excessive data redundancy and the query results may not be real-time enough. Conversely, if the amount of data written suddenly decreases, but the time window remains unchanged, the query efficiency may decrease, wasting computing resources.

[0033] 2. Waste of storage space: Aggregation of data under a fixed window may generate unnecessary redundant data in some scenarios. For example, when the data in a certain period of time is very sparse, the system will still aggregate it according to the fixed window, resulting in a waste of storage space. This waste will be magnified, especially in large-scale time series databases.

[0034] 3. Inflexible query results: User query requirements often change dynamically. Some scenarios require more fine-grained aggregate data, while other scenarios may require more coarse-grained aggregate data. If the database's aggregation window is fixed and cannot be quickly adjusted to respond to different query requirements, it will affect the user experience.

[0035] 4. Fluctuation of system load: In scenarios with high concurrency and large amounts of data, fixed window aggregation may cause unexpected fluctuations in system load. When data writes surge, the processing capacity of the fixed window may not be able to respond in time, resulting in increased query latency or even system crashes.

[0036] Based on the limitations of the above aggregate queries, the existing aggregate query methods for time series databases have the following problems:

[0037] 1. Unable to adapt to real-time data changes: In dynamic data environments such as the Internet of Things and financial markets, the speed of data generation may fluctuate significantly over time. Fixed aggregation windows cannot dynamically adapt to these changes. For example, when the financial market opens, the amount of data suddenly surges. Without a dynamic aggregation window adjustment mechanism, queries will become very slow, affecting users' ability to obtain real-time data.

[0038] 2. Resource waste: When data traffic decreases, fixed windows still perform aggregation operations at a set time granularity, resulting in ineffective computing and storage waste. In these cases, resource utilization efficiency is low, especially in cloud computing environments, where wasted computing resources directly affect cost-effectiveness.

[0039] 3. Inconsistent query response time: There may be huge differences in query response time during peak and off-peak periods. The fixed window design is difficult to quickly respond to data query needs during peak periods, resulting in increased latency. During off-peak periods, fixed window aggregation may generate too much useless aggregated data, increasing query processing time.

[0040] 4. Lack of flexibility for different query requirements: User query requirements are dynamic and may require data of different granularities depending on the business scenario. However, fixed window aggregation methods are difficult to provide flexible support for these requirements. For example, one user may need 5-minute aggregated data for the past day, while another user needs 1-minute aggregated data for the past hour. Fixed windows cannot efficiently meet different requirements.

[0041] 5. Unable to cope with abnormal data situations: When abnormal situations occur in the data stream (such as sudden increase in data flow or data accumulation caused by network delay), fixed windows cannot cope with these sudden abnormalities. Without dynamic aggregation window adjustment, the accuracy of data query may decrease, or real-time information cannot be obtained on time.

[0042] 6. Increased maintenance complexity: For developers or operation and maintenance personnel, the fixed window aggregation design needs to be manually configured and constantly adjusted to adapt to changing scenarios, which increases the maintenance cost of the system. In addition, different application scenarios may require different window settings, resulting in more complex configuration work.

[0043] Based on this, the embodiments of the present application provide a method, system and device for dynamically adjusting the aggregation window of a database, by dynamically adjusting the size and granularity of the aggregation window according to aspects such as data flow changes, data density, system load conditions and user query requirements of the time series database, so that the aggregation window can better adapt to real-time changes in data, improve the flexibility and efficiency of aggregation queries, achieve resource optimization, and improve the accuracy of data aggregation and user experience.

[0044] Figure 1 This is one of the flow charts of the method for dynamically adjusting the aggregation window of a database in an embodiment of the present application.

[0045] like Figure 1 As shown, the method for dynamically adjusting the aggregation window of the database may include the following steps:

[0046] Step 110: Obtain the current traffic parameters, data density, load parameters of the database and the user's query demand data.

[0047] It should be noted that the database described in the embodiment of the present application can be a time series database; in addition, the method for dynamically adjusting the aggregation window of the database in the embodiment of the present application can be executed by the processor of the system where the database is located. For example, if the database is the database of an urban traffic management system, then the method can be executed by the processor of the urban traffic management system.

[0048] The method for dynamically adjusting the aggregation window of a database in an embodiment of the present application can be applied to various application scenarios such as the Internet of Things (IoT), financial market monitoring, intelligent manufacturing, urban traffic management, environmental monitoring, and health monitoring.

[0049] In this embodiment, the database can collect time series data from various data sources in real time and write the time series data into the database. Data sources include IoT devices, sensors, applications, etc. The database can receive data through a protocol adaptation layer (such as MQTT protocol, HTTP protocol, TCP protocol, etc.).

[0050] After receiving the data, the database can pre-process the received data, such as deduplication, formatting, and data cleaning, to ensure data quality. After the database pre-processes the received data, it writes the data into the time series database according to the timestamp of the data to ensure that the data is stored in chronological order.

[0051] When writing time series data to the database, you can monitor the current traffic parameters, data density, and load parameters of the database in real time. The following is a detailed introduction to these three parameters:

[0052] In some embodiments, the current traffic parameters of the database may include a data write rate. The data write rate can reflect the short-term dynamics of the database traffic. When the data write rate is fast, it indicates that the data flow may fluctuate significantly over time. If a longer time window is used for aggregation, data redundancy may result, and the returned query results may not be real-time enough. When the data write rate is slow, it indicates that the data traffic is small. If the time window size remains unchanged, the query efficiency may decrease, wasting computing resources. Therefore, this embodiment obtains the data write rate and adjusts the window size based on the data write rate.

[0053] The data write rate can be obtained by using the performance monitoring tools or commands provided by the database. For MySQL, the SHOW GLOBAL STATUS command can be used in combination with specific status variables (such as Com_insert, Com_update, Com_delete, etc.) to calculate the write rate. The data write rate can also be obtained by using third-party monitoring tools and services. There are many monitoring tools designed specifically for databases, such as the Prometheus+Grafana combination, Zabbix, Nagios, etc. These tools can not only collect information about the data write rate, but also provide rich visual reports. It should be noted that other existing acquisition methods can also be used to obtain the data write rate, and the method for obtaining the data write rate is not specifically limited here.

[0054] In some implementations, the current traffic parameters of the database also include the current data traffic, the traffic peak value within a preset time period, the traffic average value within a preset time period, and the total write volume within multiple consecutive unit time periods.

[0055] It should be noted that both the preset time period and the unit time period can be set by the staff according to actual needs. For example, the preset time period can be 5 days, and the unit time period can be 1 hour, 1 day, etc. There is no specific limitation on the preset time period and unit time period here.

[0056] In this embodiment, the traffic peak value and traffic average value of the database in this time period can be recorded regularly, and the total data write amount can be recorded once every unit time period. For example, the traffic peak value and traffic average value of the week can be recorded once a week, and the total data write amount can be recorded once a day.

[0057] The method for dynamically adjusting the aggregation window of the database may also include: judging whether the database is in a traffic peak period based on the current data traffic and the traffic peak within a preset time period, and directing the data traffic to the low-load node of the database when it is determined that the database is in a traffic peak period; determining whether there is an anomaly in the data flow of the database based on the traffic peak, the average traffic value, the first preset threshold and the second preset threshold within the preset time period; wherein the first preset threshold is dynamically determined based on the historical peak data, and the second preset threshold is dynamically determined based on the historical average data; determining the change pattern of the total write volume based on the total write volume in each unit time period, and optimizing the aggregation query strategy based on the change pattern.

[0058] Specifically, the traffic peak value can reflect the maximum pressure of the database system in a certain period of time, and is the core basis for adjusting the upper limit of the aggregation window. For example, when the data traffic reaches the peak, the aggregation window can be appropriately increased to reduce the system load. In addition, the traffic peak value can also be used to predict the maximum resource demand of the system, thereby helping to reasonably allocate memory, CPU and storage resources.

[0059] The average traffic value can reflect the normal operation level of the data flow and can be used as a benchmark value for adjusting the aggregation window. When running smoothly for a long time, the aggregation window can be kept stable and frequent adjustments of the aggregation window can be avoided. The average value can also be used to estimate the regular resource requirements of the system, optimize the efficiency of resource use, and avoid excessive or insufficient resource use.

[0060] In a cluster environment, by recording the peak and average traffic flow, the distribution characteristics of the data flow can be identified, so as to make load balancing adjustments. Specifically, after obtaining the current data flow, the current data flow can be compared with the peak flow. If the current data flow is higher than the peak flow, it is determined that the database is currently in a peak flow period. At this time, the peak flow can be directed to the database's low-load nodes to balance the overall pressure.

[0061] Furthermore, it is also possible to determine whether there is an anomaly in the data stream (for example, a sudden data write or a traffic attack) based on the traffic peak value, the traffic average value, the first preset threshold value, and the second preset threshold value. Specifically, a first preset threshold value for comparison can be determined using the historical peak value data, and then a second preset threshold value for comparison can be determined using the historical average value data. Then the current traffic peak value is compared with the first preset threshold value, and the current traffic average value is compared with the second preset threshold value. If the current traffic peak value deviates significantly from the first preset threshold value, or the current traffic average value deviates significantly from the second preset threshold value, it is determined that there is an anomaly in the data stream. The traffic peak value and the traffic average value can provide important monitoring indicators for the system, so that problems in the system can be discovered in a timely manner through the traffic peak value and the traffic average value.

[0062] It is understandable that although monitoring the data write rate can reflect short-term dynamics, monitoring only the data write rate may not be sufficient to meet the needs of system optimization.

[0063] Therefore, the embodiment of the present application also monitors the total amount of data written to the database within a unit time period, and the total amount of data written is an important indicator for evaluating data integrity. By counting the total number of written data, it is possible to check whether the data is missed or written repeatedly, especially for application scenarios that require strong consistency. The total amount of written data in the database also directly affects the storage capacity requirements. By counting the total amount of written data, the storage utilization rate and future expansion requirements can also be evaluated.

[0064] In addition, the total amount of data written can be analyzed in each unit time period to obtain the change pattern of data written. For example, by comparing the total amount of data written each day, the regular fluctuation of data flow can be found, and then the aggregation strategy can be optimized based on the change pattern.

[0065] The above are the types, acquisition methods and usage of the current traffic parameters of the database. The following introduces how to obtain data density:

[0066] In some implementations, obtaining the current data density of the database may include: obtaining the number of data points in a preset time period of the database; and determining the data density based on the preset time period and the number of data points.

[0067] It is understandable that the total number of data points in the database is a global statistical value, which cannot reflect the changing trend of the data flow in a specific time period, and is used to judge whether the data is dense or sparse. When judging whether the data is dense or sparse, more attention is paid to the fluctuation of the data flow in a short period of time, rather than the cumulative result of the entire historical data. Therefore, the embodiment of the present application obtains the number of data points in a preset time period. The number of data points in a period of time can accurately reflect the real-time characteristics of the data flow, thereby being able to directly reflect the data density of the current database, and provide an accurate basis for adjusting the aggregation window.

[0068] It should be noted that the preset time period can be set by the staff according to actual needs. For example, the preset time period can be 1 day, 1 hour, etc., and the preset time period is not specifically limited here.

[0069] After obtaining the number of data points in the preset time period, the number of data points can be divided by the time interval of the preset time period to obtain the data density. For example, if the number of data points in 1 hour is 50,000, the data density per second can be 13.8.

[0070] In this embodiment, the density or sparseness of the current data can be determined based on the data density. Specifically, the threshold can be dynamically adjusted based on the statistical results of the number of data points in the historical data stream. For example, the mean of the data stream in the past hour or day is calculated, and the threshold of the current time period is dynamically set. The threshold can be a multiple of the mean (such as 1.5 times or 0.5 times). The current number of data points is compared with the dynamic threshold in real time to determine the density or sparseness of the current data.

[0071] The above is an introduction to the method of obtaining data density. The following is a detailed introduction to load parameters:

[0072] The load parameter refers to the system load. The load parameter can be represented by the comprehensive utilization rate of the CPU, memory, and I / O (Input / Output). The range of the load parameter is 0 to 1.

[0073] Through the load parameters, the current system load status can be obtained. The system load status will affect the data aggregation process. Therefore, the embodiment of the present application obtains the system load parameters and adjusts the aggregation window size based on the load parameters to reduce the impact of the system load on the data aggregation process. For example, under high load conditions, the aggregation window can be dynamically increased to reduce computing pressure, and the window can be reduced under low load to provide more detailed data.

[0074] Step 120: Calculate the adjustment factor of the aggregation window based on the traffic parameter, data density and load parameter using a preset formula, and adjust the initial aggregation window size of the database based on the adjustment factor to obtain a target aggregation window size.

[0075] After obtaining the current traffic parameters, data density, and load parameters of the database, these three parameters can be comprehensively analyzed to determine the pressure characteristics of the data flow, and the size of the aggregation window can be dynamically adjusted according to the pressure characteristics to optimize query efficiency and resource utilization.

[0076] Specifically, a pre-built fusion formula (i.e., a preset formula) can be used to calculate the adjustment factor of the aggregation window based on the traffic parameter, data density, and load parameter. In some implementations, the preset formula is:

[0077] A=w1*R+w2*D+w3*L

[0078] In the formula, A is the adjustment factor, with a value range of [-1,1]; R is the data write rate (unit: points / second); w1 is the weight of the data write rate; D is the data density; w2 is the weight of the data density; L is the load parameter; w3 is the weight of the load parameter. It should be noted that the adjustment factor is used to determine the adjustment range of the aggregation window size; w1, w2 and w3 can be set by the staff through a large number of experiments or experience, and these three weights can also be dynamically adjusted according to the system operation scenario.

[0079] Figure 2 This is the second flow chart of the method for dynamically adjusting the aggregation window of the database in the embodiment of the present application. Figure 2 As shown, in some implementations, before calculating the adjustment factor of the aggregation window based on the traffic parameter, data density, and load parameter using a preset formula, the method for dynamically adjusting the aggregation window of the database may further include the following steps:

[0080] Step 210: Determine the current scene type of the database.

[0081] Step 220: Based on the current scenario type of the database, dynamically adjust the weight of the data write rate, the weight of the data density, and the weight of the load parameter.

[0082] In this embodiment, the weight of the data write rate, the weight of the data density, and the weight of the load parameter can be determined according to the current scenario type of the database. For example, if the current database is in a high-frequency write state, it means that the data write rate will have a greater impact on the current aggregation window size. In this case, the weight of the data write rate can be set larger.

[0083] In some embodiments, the scenario types include high-frequency writing scenarios, density change scenarios, and resource-scarce scenarios. Dynamically adjusting the weight of the data writing rate, the weight of the data density, and the weight of the load parameter may include: if the scenario type is a high-frequency writing scenario, the weight of the data writing rate is greater than the weight of the data density, and the weight of the data density is greater than the weight of the load parameter; if the scenario type is a density change scenario, the weight of the data density is greater than the weight of the data writing rate, and the weight of the data writing rate is greater than the weight of the load parameter; if the scenario type is a resource-scarce scenario, the weight of the load parameter is greater than the weight of the data writing rate, and the weight of the data writing rate is greater than the weight of the data density.

[0084] It should be noted that the high-frequency write scenario refers to: the write rate of the current database is high; the density change scenario refers to: the data density of the current database changes greatly, for example, the data density changes from dense to sparse (which can be judged by the judgment method in the aforementioned embodiment); the resource-scarce scenario refers to: the system load is close to the system load upper limit and resources are tight.

[0085] If the scenario type of the current database is a high-frequency write scenario, the weight of the data write rate is greater than the weight of the data density, and the weight of the data density is greater than the weight of the load parameter, that is, w1>w2>w3.

[0086] If the scenario type of the current database is a density change scenario, the weight of data density is greater than the weight of data write rate, and the weight of data write rate is greater than the weight of load parameters, that is, w2>w1>w3.

[0087] If the scenario type of the current database is a resource-scarce scenario, the weight of the load parameter is greater than the weight of the data write rate, and the weight of the data write rate is greater than the weight of the data density, that is, w3>w1>w2.

[0088] Therefore, the weights of various parameters can be dynamically adjusted according to the current scenario type of the database, so that the calculated adjustment factor is more in line with the current database situation and the adjustment factor is more accurate, thereby enabling the adjustment of the aggregation window size to be more accurate and reliable.

[0089] After calculating the adjustment factor of the aggregation window, the initial aggregation window size of the database can be adjusted based on the adjustment factor to calculate the target aggregation window size. The initial aggregation window size of the database is recorded as S0, and the target aggregation window size is recorded as S1. The target aggregation window size can be calculated by the following formula:

[0090] S1=S0*(1+k*A) k is the adjustment ratio coefficient, which can control the sensitivity of the adjustment. The value can be set by the staff according to actual needs and is not specifically limited here. It should be noted that if the adjustment factor A is greater than zero, k is a positive value; if the adjustment factor A is less than zero, k is a negative value.

[0091] As an example, if the initial aggregation window size is 10 seconds, the adjustment factor A calculated by the preset formula is 0.4, and the set adjustment ratio coefficient is 0.2, then the target aggregation window size is: S1=10×(1+0.2×0.4)=10.8 seconds.

[0092] Step 130: Determine the target aggregation granularity of the aggregation window based on the query demand data.

[0093] It is understandable that the query requirements of users often change dynamically. Some scenarios require more fine-grained aggregated data, while other scenarios may require more coarse-grained aggregated data. If the aggregation window of the database is fixed and cannot be quickly adjusted to respond to different query requirements, it will affect the user experience. Therefore, the embodiment of the present application dynamically adjusts the granularity of the aggregation window according to the query requirements of the user, thereby reducing unnecessary calculations and saving computing resources.

[0094] In this embodiment, for the frequent query requirements of users, aggregation windows of different granularities can be dynamically generated. Generating windows of different granularities means aggregating data within the same time range in different time units.

[0095] In some embodiments, the query demand data includes a query time range. Determining a target aggregation granularity of the aggregation window based on the query demand data may include: if the query time range meets a first preset range, determining the target aggregation window granularity to be a first granularity; if the query time range meets a second preset range, determining the target aggregation window granularity to be a second granularity; and the second granularity is smaller than the first granularity.

[0096] It should be noted that the first preset range can be: a range in months; the second preset range can be: a range in days; the first granularity can be: summary by day, week, month; the second granularity can be: summary by second, minute, hour. The first granularity and the second granularity can be set by the staff according to actual needs and are not specifically limited here.

[0097] Specifically, in a query process, multiple aggregation scales may be involved, such as a high-level aggregation scale (i.e., the first preset range) and a low-level aggregation scale (i.e., the second preset range). When the user's query time range meets the first preset range, for example, viewing a month's data by day, the target aggregation window granularity can be set to aggregate by day, week, or month. When the user's query time range meets the second preset range, for example, viewing detailed data within a day by hour, the target aggregation window granularity can be set to aggregate by second, minute, or hour.

[0098] In this way, unnecessary data retrieval can be avoided and query efficiency can be significantly improved. In addition, the diverse query needs of users can be met and the query granularity can be flexibly selected according to different business needs, thereby improving the user experience.

[0099] In some embodiments, after obtaining the current traffic parameters, data density, load parameters of the database and the user's query demand data, the method may also include: determining whether a preset adjustment trigger condition is met; if it is determined that the preset adjustment trigger condition is met, using a preset formula and calculating the adjustment factor of the aggregation window based on the traffic parameters, data density and load parameters; wherein the preset adjustment trigger condition includes a preset interval time and / or a change in the data write rate exceeding a preset change value.

[0100] Specifically, a preset adjustment trigger condition can be set, and when the system reaches the preset adjustment trigger condition, the aggregation window is adjusted. For example, the aggregation window is adjusted once every certain period of time (such as 5 minutes); or the aggregation window adjustment is triggered when the data flow changes (that is, the data write rate changes).

[0101] Step 140: Set a new aggregation window based on the target aggregation window size and the target aggregation granularity, and use the new aggregation window to perform aggregation query.

[0102] In this embodiment, when the system receives a query request input by a user, the system can first extract the required aggregation operation (such as average value AVG, total sum SUM, etc.) and the query time range. Further, a new aggregation window is set according to the target aggregation window size and target aggregation granularity obtained in the above manner; finally, the new aggregation window is used to perform data aggregation query operations. In addition, indexes and other query optimization techniques can be used to improve query performance.

[0103] After obtaining the aggregate query results, the aggregate query results can be fed back to the user. In this embodiment, the user can upload his query request and view the aggregate query results through the user terminal. Specifically, the user terminal can provide a user interaction interface, allowing the user to enter the query conditions, for example, the user selects the data source, time range and aggregation operation; the user can also view the system's monitoring data and aggregate query results, including the current data flow rate, system load, aggregation window settings, and data obtained by the aggregate query, etc. In addition, the user's query history can also be recorded to facilitate the user to repeat the query or make comparisons.

[0104] The method for dynamically adjusting the aggregation window of a database in an embodiment of the present application has the following advantages:

[0105] 1. Improve query performance. Dynamically adjust the aggregation window to optimize query performance according to real-time data traffic changes. During peak data traffic, increasing the aggregation window can reduce the computational burden and thus speed up query response time; while when data traffic is low, reducing the window can improve the refinement of data and meet users' needs for real-time and accuracy.

[0106] 2. Flexible adaptation to different scenarios. The system can flexibly adjust the size of the aggregation window according to the characteristics of the data flow and user needs. This flexibility enables the system to provide the best data processing strategy in different application scenarios and adapt to various business needs.

[0107] 3. Resource optimization. By dynamically adjusting the aggregation window, the system can effectively utilize computing and storage resources. When data traffic is high, the use of computing resources is reduced; when traffic is low, the waste of storage space is avoided. This resource optimization helps reduce operating costs, especially in cloud computing environments.

[0108] 4. Improve data accuracy. The dynamic aggregation window can be adjusted according to the actual situation of the data to avoid data redundancy or loss caused by fixed windows. This enables the system to provide more accurate and timely data analysis results under different data flow conditions.

[0109] 5. Real-time monitoring and feedback. The system monitors data traffic and system load in real time, and can adjust aggregation strategies in a timely manner through a dynamic feedback mechanism to ensure that the system is always in the best state. This real-time monitoring can help developers and operation and maintenance personnel to find problems and optimize them in a timely manner.

[0110] 6. Enhance user experience. The dynamic aggregation window mechanism can meet the diverse query needs of users. Users can flexibly select the query granularity according to different business needs, which improves the user experience.

[0111] Based on the above embodiments, the embodiments of the present application also provide a system for dynamically adjusting the aggregation window of a database. Figure 3 is a schematic diagram of a database aggregation window dynamic adjustment system according to an embodiment of the present application. Figure 3 As shown, the database aggregation window dynamic adjustment system 300 may include: an acquisition module 310 , a window size adjustment module 320 , a window granularity determination module 330 and an aggregation query module 340 .

[0112] Among them, the acquisition module 310 is used to obtain the current traffic parameters, data density, load parameters of the database and the user's query demand data; the window size adjustment module 320 is used to use a preset formula and calculate the adjustment factor of the aggregation window based on the traffic parameters, data density and load parameters, and adjust the initial aggregation window size of the database based on the adjustment factor to obtain the target aggregation window size; the window granularity determination module 330 is used to determine the target aggregation granularity of the aggregation window based on the query demand data; the aggregation query module 340 is used to set a new aggregation window based on the target aggregation window size and the target aggregation granularity, and use the new aggregation window to perform aggregation query.

[0113] Therefore, the current traffic parameters, data density, load parameters of the database and the query demand data of the user are obtained through the acquisition module 310; the window size adjustment module 320 then uses the preset formula and calculates the adjustment factor of the aggregation window based on the traffic parameters, data density and load parameters, and adjusts the initial aggregation window size of the database based on the adjustment factor, so that the aggregation window size adapts to the current state of the database and the changes in the data flow; the window granularity determination module 330 then determines the target aggregation granularity of the aggregation window based on the query demand data, so that the granularity of the aggregation window is more adapted to user needs; finally, the aggregation query module 340 sets a new aggregation window based on the target aggregation window size and the target aggregation granularity, and uses the new aggregation window to perform aggregation query, thereby making the aggregation window more adaptable to real-time changes in data and improving the flexibility of aggregation query.

[0114] In some implementations, the traffic parameter includes a data writing rate, and the preset formula is:

[0115] A=w1*R+w2*D+w3*L

[0116] Wherein, A is the adjustment factor, and its value range is [-1,1]; R is the data writing rate; w1 is the weight of the data writing rate; D is the data density; w2 is the weight of the data density; L is the load parameter; and w3 is the weight of the load parameter.

[0117] In some implementations, the window size adjustment module 320 is further used to: determine the current scenario type of the database; and dynamically adjust the weight of the data write rate, the weight of the data density, and the weight of the load parameter based on the current scenario type of the database.

[0118] In some embodiments, the scenario types include high-frequency writing scenarios, density change scenarios, and resource-scarce scenarios; the window size adjustment module 320 is specifically used for: if the scenario type is a high-frequency writing scenario, the weight of the data write rate is greater than the weight of the data density, and the weight of the data density is greater than the weight of the load parameter; if the scenario type is a density change scenario, the weight of the data density is greater than the weight of the data write rate, and the weight of the data write rate is greater than the weight of the load parameter; if the scenario type is a resource-scarce scenario, the weight of the load parameter is greater than the weight of the data write rate, and the weight of the data write rate is greater than the weight of the data density.

[0119] In some implementations, the acquisition module 310 is specifically used to: acquire the number of data points in a preset time period in the database; and determine the data density based on the preset time period and the number of data points.

[0120] In some embodiments, the query demand data includes a query time range. The window granularity determination module 330 is specifically used to: if the query time range meets the first preset range, determine the target aggregation window granularity as the first granularity; if the query time range meets the second preset range, determine the target aggregation window granularity as the second granularity; the second granularity is smaller than the first granularity.

[0121] In some embodiments, the window size adjustment module 320 is also specifically used to: determine whether a preset adjustment trigger condition is met; if it is determined that the preset adjustment trigger condition is met, use a preset formula and calculate the adjustment factor of the aggregation window based on traffic parameters, data density, and load parameters; wherein the preset adjustment trigger condition includes a preset interval time and / or a change in data write rate exceeding a preset change value.

[0122] In some embodiments, the current traffic parameters of the database also include the current data traffic, the traffic peak value within a preset time period, the traffic average value within a preset time period, and the total write volume within multiple consecutive unit time periods. The acquisition module 310 is also specifically used to: determine whether the database is in a traffic peak period based on the current data traffic and the traffic peak value within the preset time period, and if it is determined that the database is in a traffic peak period, direct the data traffic to the low-load node of the database; determine whether there is an anomaly in the data flow of the database based on the traffic peak value, the traffic average value, the first preset threshold value, and the second preset threshold value within the preset time period; wherein the first preset threshold value is dynamically determined based on the historical peak value data, and the second preset threshold value is dynamically determined based on the historical average value data; determine the change pattern of the total write volume based on the total write volume within each unit time period, and optimize the aggregation query strategy based on the change pattern.

[0123] In some embodiments, the system may also include a system management module for managing and maintaining the system, such as configuration management, performance monitoring, and log management. Specifically, configuration management: allows administrators to set system parameters (such as the initial size of the aggregation window, thresholds, etc.) and save these configurations. Performance monitoring: records key performance indicators during system operation and provides a visual interface to facilitate administrators to analyze and optimize system performance. Log management: records the system's operating logs, including data writing, query operations, and exception handling, to facilitate problem tracking and troubleshooting.

[0124] It should be noted that for details not disclosed in the system for dynamically adjusting the aggregation window of a database in this embodiment, please refer to the details disclosed in the embodiment of the method for dynamically adjusting the aggregation window of a database in the embodiments of this specification, and no further details will be given here.

[0125] Based on the above embodiments, Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the method for dynamically adjusting the aggregation window of the database, including: obtaining the current traffic parameters, data density, load parameters and query demand data of the database; calculating the adjustment factor of the aggregation window based on the traffic parameters, data density and load parameters using a preset formula, and adjusting the initial aggregation window size of the database based on the adjustment factor to obtain the target aggregation window size; determining the target aggregation granularity of the aggregation window based on the query demand data; setting a new aggregation window based on the target aggregation window size and the target aggregation granularity, and performing aggregation query using the new aggregation window.

[0126] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0127] On the basis of the above embodiments, on the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the dynamic adjustment method of the aggregation window of the database provided by the above methods, including: obtaining the current traffic parameters, data density, load parameters of the database and the user's query demand data; using a preset formula and calculating the adjustment factor of the aggregation window based on the traffic parameters, data density and load parameters, and adjusting the initial aggregation window size of the database based on the adjustment factor to obtain the target aggregation window size; determining the target aggregation granularity of the aggregation window based on the query demand data; setting a new aggregation window based on the target aggregation window size and the target aggregation granularity, and using the new aggregation window to perform aggregation query.

[0128] On the basis of the above embodiments, on another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for dynamically adjusting the aggregation window of a database provided by the above methods, including: obtaining the current traffic parameters, data density, load parameters of the database and the query demand data of the user; using a preset formula and calculating the adjustment factor of the aggregation window based on the traffic parameters, data density and load parameters, and adjusting the initial aggregation window size of the database based on the adjustment factor to obtain a target aggregation window size; determining the target aggregation granularity of the aggregation window based on the query demand data; setting a new aggregation window based on the target aggregation window size and the target aggregation granularity, and using the new aggregation window to perform aggregation query.

[0129] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. A method for dynamically adjusting the aggregation window of a database, characterized in that: include: Obtain the database's current traffic parameters, data density, load parameters, and user query demand data; Calculating an adjustment factor of the aggregation window based on the traffic parameter, the data density, and the load parameter using a preset formula, and adjusting an initial aggregation window size of the database based on the adjustment factor to obtain a target aggregation window size; Determining a target aggregation granularity of the aggregation window based on the query demand data; A new aggregation window is set based on the target aggregation window size and the target aggregation granularity, and an aggregation query is performed using the new aggregation window.

2. The method for dynamically adjusting the aggregation window of a database according to claim 1, characterized in that: The traffic parameter includes a data writing rate, and the preset formula is: A=w1*R+w2*D+w3*L In the formula, A is the adjustment factor, and its value range is [-1,1]; R is the data write rate; w1 is the weight of the data write rate; D is the data density; w2 is the weight of the data density; L is the load parameter; and w3 is the weight of the load parameter.

3. The method for dynamically adjusting the aggregation window of a database according to claim 2, characterized in that: Before calculating the adjustment factor of the aggregation window based on the traffic parameter, the data density, and the load parameter using a preset formula, the method further includes: Determining a current scenario type of the database; Based on the current scenario type of the database, the weight of the data write rate, the weight of the data density, and the weight of the load parameter are dynamically adjusted.

4. The method for dynamically adjusting the aggregation window of a database according to claim 3, characterized in that: The scenario types include high-frequency writing scenarios, density change scenarios, and resource-scarce scenarios; The dynamically adjusting the weight of the data writing rate, the weight of the data density, and the weight of the load parameter includes: If the scenario type is the high-frequency writing scenario, the weight of the data writing rate is greater than the weight of the data density, and the weight of the data density is greater than the weight of the load parameter; If the scenario type is the density change scenario, the weight of the data density is greater than the weight of the data write rate, and the weight of the data write rate is greater than the weight of the load parameter; If the scenario type is the resource-scarce scenario, the weight of the load parameter is greater than the weight of the data write rate, and the weight of the data write rate is greater than the weight of the data density.

5. The method for dynamically adjusting the aggregation window of a database according to claim 1, characterized in that: Get the current data density of the database, including: Obtain the number of data points in a preset time period of the database; The data density is determined based on the preset time period and the number of data points.

6. The method for dynamically adjusting the aggregation window of a database according to claim 1, characterized in that: The query demand data includes a query time range; The determining the target aggregation granularity of the aggregation window based on the query demand data includes: If the query time range meets the first preset range, determining the target aggregation window granularity to be the first granularity; If the query time range meets the second preset range, the target aggregation window granularity is determined to be a second granularity; the second granularity is smaller than the first granularity.

7. The method for dynamically adjusting the aggregation window of a database according to claim 2, characterized in that: After obtaining the current traffic parameters, data density, load parameters of the database and the query demand data of the user, the method further includes: Determine whether the preset adjustment trigger condition is met; If it is determined that the preset adjustment trigger condition is met, calculating the adjustment factor of the aggregation window based on the traffic parameter, the data density and the load parameter using a preset formula; The preset adjustment triggering condition includes the passage of a preset interval time and / or the change value of the data writing rate exceeds a preset change value.

8. The method for dynamically adjusting the aggregation window of a database according to any one of claims 1 to 7, characterized in that: The current traffic parameters of the database also include the current data traffic, the traffic peak value within a preset time period, the traffic average value within a preset time period, and the total write volume within multiple consecutive unit time periods; The method further comprises: Based on the current data traffic and the traffic peak value within the preset time period, determine whether the database is in a traffic peak period, and if it is determined that the database is in the traffic peak period, direct the data traffic to a low-load node of the database; Based on the flow peak value within the preset time period, the flow average value, a first preset threshold value and a second preset threshold value, determining whether there is an abnormality in the data flow of the database; wherein the first preset threshold value is dynamically determined based on historical peak value data, and the second preset threshold value is dynamically determined based on historical average value data; Based on the total write volume in each of the unit time periods, a change pattern of the total write volume is determined, and the aggregate query strategy is optimized based on the change pattern.

9. A database aggregation window dynamic adjustment system, characterized in that: include: The acquisition module is used to obtain the current traffic parameters, data density, load parameters of the database and the user's query demand data; A window size adjustment module, configured to calculate an adjustment factor of an aggregation window based on the traffic parameter, the data density, and the load parameter using a preset formula, and to adjust an initial aggregation window size of the database based on the adjustment factor to obtain a target aggregation window size; A window granularity determination module, configured to determine a target aggregation granularity of the aggregation window based on the query demand data; The aggregation query module is used to set a new aggregation window based on the target aggregation window size and the target aggregation granularity, and perform aggregation query using the new aggregation window.

10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the method for dynamically adjusting the aggregation window of a database according to any one of claims 1 to 8 is implemented.

Images