Data storage method and system, electronic equipment and storage medium

Abstract

The invention discloses a data storage method and system, electronic equipment and a storage medium, and the method comprises the steps: when at least one data writing request is obtained, determining the type of a target data table, and when the target data table is a first type of data table, writing the target data table in the first type of data table; predicting a predicted data volume written into at least one memory space in continuous N time windows; determining an opportunity of triggering a disk dropping request in each time window based on the predicted data volume, and generating at least one disk dropping request when a disk dropping request triggering condition is met; based on the load parameters, distributing at least one disk dropping request to at least one service node corresponding to each time window; and storing the data in each memory space into a hard disk corresponding to at least one service node. Through the method, the technical problem that the data processing capacity of the distributed database is low in the related technology is solved, and the technical effect of improving the data processing capacity of the distributed database is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for transferring data to a disk, an electronic device, and a storage medium. Background Art

[0002] In the HBase database, for data tables with low write frequency, the data accumulation in the Memstore (the memory space of the RS node) is relatively slow. The relevant technology usually relies on the HBase service to perform regular checks at preset intervals to determine whether the data in the Memstore needs to be written to the hard disk corresponding to each RS node. Among them, RS (RegionServer) is used to provide read and write services to the outside world.

[0003] However, the above-mentioned periodic checking method may cause a large number of Flush requests (data disk write requests) to be triggered in a short period of time, resulting in a surge in normal read and write request delays and excessive pressure on the hard disk load, thereby causing technical problems such as low data processing capabilities of the HBase database. Summary of the invention

[0004] The present application provides a method and system for storing data on a disk, an electronic device, and a storage medium, so as to at least solve the problem of low data processing capability of an HBase database in the related art.

[0005] According to one aspect of an embodiment of the present application, a method for flushing data to a disk is provided, comprising: in response to at least one data write request to write target data to a target data table, determining the type of the target data table, wherein the target data table is divided into at least one table unit, and the at least one table unit corresponds one-to-one to at least one memory space; in a case where the type indicates that the target data table is a first type data table, predicting a predicted amount of data written to the at least one memory space within N consecutive time windows, wherein N is a positive integer greater than or equal to 1; in a case where the predicted amount of data satisfies a flushing request trigger condition, generating at least one flushing request; based on a load parameter of at least one service node, dynamically allocating the at least one flushing request to the at least one service node corresponding to each time window; and flushing the data in the at least one memory space to at least one hard disk corresponding to the at least one service node.

[0006] According to another aspect of an embodiment of the present application, a data disk drop system is also provided, including: a monitoring module, used to obtain at least one data write request to write target data into a target data table, wherein the target data table is divided into at least one table unit, and the at least one table unit corresponds one-to-one to at least one memory space; a disk drop request control module, used to determine the type of the target data table and, when the type indicates that the target data table is a first type of data table, predict the predicted amount of data written to the at least one memory space within N consecutive time windows; and generate at least one disk drop request based on the predicted amount of data, wherein N is a positive integer greater than or equal to 1; a collaborative scheduling module, used to dynamically allocate the at least one disk drop request to the at least one service node corresponding to each time window based on the load parameters of the at least one service node, so as to drop the data in the at least one memory space to the at least one hard disk corresponding to the at least one service node.

[0007] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of any of the above-mentioned data storage methods through the computer program.

[0008] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data storage methods when running.

[0009] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the steps of the above-mentioned data storage method.

[0010] Through the above-mentioned embodiments provided by the present application, based on the judgment of the type of data table and the estimation of the amount of data written in at least one memory space, the triggering time of the data drop request of the first type of data table is dynamically predicted, and at least one drop request is evenly distributed to at least one service node corresponding to each time window according to the real-time load of each service node. In other words, by predicting the triggering time of the drop request in each time window and the real-time load of the service node, it is ensured that the drop request is distributed to each service node in turn according to the load level, so as to achieve global load balancing, avoid further congestion of high-load service nodes, improve the data writing performance and stability of the HBase database, and achieve the technical effect of improving the data processing capability of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 It is a schematic diagram of an application scenario of a method for transferring data to a disk according to an embodiment of the present application.

[0013] Figure 2 This is a process intention of an optional method for transferring data to a disk according to an embodiment of the present application.

[0014] Figure 3 It is an overall schematic diagram of an optional method for transferring data to a disk according to an embodiment of the present application.

[0015] Figure 4 It is a schematic diagram of an optional time window and all time periods according to an embodiment of the present application.

[0016] Figure 5 This is a schematic diagram of an optional triggering Flush request according to an embodiment of the present application.

[0017] Figure 6 It is a schematic diagram of an optional mapping relationship between virtual nodes and physical nodes according to an embodiment of the present application.

[0018] Figure 7 It is a schematic diagram of generating a flush request slot in an optional manner based on a combination of virtual node slots and time windows according to an embodiment of the present application.

[0019] Figure 8 It is a structural block diagram of an optional data disk storage system according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0022] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0023] According to one aspect of an embodiment of the present application, a method for storing data on a disk is provided. Optionally, in this embodiment, the method for storing data on a disk may be applied to, but is not limited to, Figure 1 In the hardware scenario shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0024] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data disk method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] The embodiments of the present application may be applicable to, but not limited to, various application scenarios of HBase in the field of big data processing and storage, especially environments involving a large number of low-frequency write tables. For ease of understanding, specific examples of several application scenarios are given below.

[0027] (1) Data analysis and storage: For example, in the process of IoT data storage, a large amount of data will be generated during the operation of IoT devices, including data statistics tables on multiple dimensions such as device status, device parameters, and environmental monitoring. These tables are updated infrequently but data continues to accumulate, so it is necessary to continuously optimize the write strategy to improve storage efficiency.

[0028] (2) Finance and Statistics: Although real-time transaction data is written frequently, the update frequency of statistical information such as transaction history records is also high. The technical solution of this application can optimize the writing management of this type of data and avoid a surge in delays in real-time transaction systems.

[0029] (3) E-commerce and advertising platforms: For example, by analyzing users’ historical purchasing behaviors, browsing records, etc., the data statistics tables generated have a lower writing frequency than the users’ real-time behavior tables. The technical solution of this application can optimize the writing of such data tables and improve the real-time and accuracy of data analysis.

[0030] (4) Social media and content management: For example, statistical information such as the number of posts forwarded by users and the number of likes, which is updated relatively infrequently, can use the optimized data storage strategy in the technical solution of this application to ensure the consistency and integrity of the data.

[0031] By applying the technical solution of the present application to the above-mentioned scenario, the performance bottleneck problem of the HBase cluster caused by the existence of a large number of low-frequency write tables is solved. By optimizing the triggering mechanism of data disk drop requests, the pressure on the system caused by a large number of Flush requests in a short period of time is dispersed, and the disk I / O storm and delay are reduced, thereby improving the data writing performance and system stability.

[0032] The data storage method of the embodiment of the present application can be executed by a server device, or by a server device in combination with at least one of the terminal devices (which can also be understood as the input and output devices 108). The terminal device can also execute the data storage method of the embodiment of the present application by a client installed thereon.

[0033] Take the method of storing data in the present embodiment performed by the server as an example. Figure 2 is a flow chart of an optional method for storing data on a disk according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include steps S202 to S210.

[0034] Step S202, in response to at least one data write request to write target data into a target data table, determining the type of the target data table, wherein the target data table is divided into at least one table unit, and at least one table unit corresponds to at least one memory space in a one-to-one manner.

[0035] Step S204: when the type indicates that the target data table is a first type data table, predict the amount of data written into at least one memory space within N consecutive time windows, where N is a positive integer greater than or equal to 1.

[0036] Step S206: if the predicted data volume meets the disk-push request triggering condition, generate at least one disk-push request.

[0037] Step S208: Based on the load parameter of at least one service node, dynamically allocate at least one disk placement request to at least one service node corresponding to each time window.

[0038] Step S210: write the data in at least one memory space to at least one hard disk corresponding to at least one service node.

[0039] Before explaining the technical solution of the present application, the meaning of the professional terms or professional nouns involved is briefly introduced first.

[0040] HBase: is a distributed, column-oriented open source database that can store massive amounts of data. The underlying data files are stored in HDFS (Hadoop Distributed File System). HBase data tables can be divided into multiple Regions (table units) through horizontal partitioning. Each Region is approximately 100MB to 1GB in size to achieve load balancing and efficient management of data.

[0041] Region: For HBase, a table is horizontally divided into multiple Regions. Each Region is the basic unit of data management and is evenly distributed on all RS nodes in the cluster to achieve load balancing of read and write requests. It is the smallest unit of cluster load balancing. Each Region maintains the startRowKey and endRowKey. If the RowKey (row key value) of a piece of data is within the RowKey range maintained by a Region, the data is managed and maintained by this Region.

[0042] Region Server (service node, referred to as RS node): Each Region Server in the HBase cluster is responsible for providing read and write services. They evenly distribute Regions from different tables to disperse the load of the cluster and achieve fast response to read and write requests.

[0043] Memstore: When data is written to an HBase data table, it is first written to the memory space (Memstore) corresponding to each Region. When the amount of data in the Memstore reaches a certain threshold or after a certain period of time, the data flush operation will be triggered, that is, the data is written from the memory space corresponding to each Region to the hard disk. Each Region corresponds to an RS node.

[0044] Cold tables and hot tables: In HBase, data tables are divided into cold tables and hot tables according to the write frequency. Cold tables refer to tables with low write frequency (also known as first-type data tables), while hot tables (also known as second-type data tables) are the opposite, with high write frequency.

[0045] As a commonly used distributed database in the field of big data, HBase will create business tables, index tables, statistical tables, etc. according to business needs. Among them, business tables write a large amount of data, while statistical tables may only write business statistical data at a low frequency. Therefore, according to the native writing logic of HBase, business tables can complete data writing and flushing to disk well, but for statistical tables with low frequency of data writing, the amount of data in Memstore is small and does not meet the conditions for flushing to disk (for example, the amount of data must reach 128MB to trigger the flushing mechanism). Data can only be flushed to disk by relying on the HBase service's regular check once an hour. However, when there are a large number of statistical tables with low writing frequency, a large number of flush requests are triggered during the system's regular check, causing a surge in a large number of flush requests in a short period of time, resulting in a surge in the system's read and write request time delay.

[0046] In view of the above problems, the technical solution of this application proposes a method for storing data on disk, and the overall implementation process is as follows: Figure 3 As shown, it includes a system monitoring module, a Flush request module, and a Flush collaborative scheduling module.

[0047] Among them, various monitoring indicator data from the client and the server are obtained in real time through the system monitoring module. For example, the client indicators include but are not limited to the number of write requests for the data table, and the server indicators include but are not limited to the task request form of all RS services, RS request delay, and Memstore data volume.

[0048] It should be noted that a data write request may be, but is not limited to, a request to write the target data into the Memstore memory space corresponding to a Region. The request includes the table name of the target data table to which the target data is written. When a data write request is received, the actual amount of data in at least one Memstore memory space will increase. The increased amount of data includes the total amount of data of at least one piece of data that needs to be written in the current data write request, for example, 50MB, 10MB, etc.

[0049] A flush request (also known as a flush request) may include, but is not limited to, a process of flushing data in at least one Memstore memory space to the hard disk corresponding to the service node when the flush request triggering condition is met.

[0050] Obviously, one Region of a data table corresponds to one memory space, one Region corresponds to one RS node, one Flush request corresponds to one Region, and one service node can maintain at least one Region of multiple data tables at the same time.

[0051] like Figure 3As shown, after monitoring the table write request (which can also be understood as a data write request) sent by the client, first determine whether the target data table is a first type data table or a second type data table. If it is a first type data table, according to the dynamic Flush trigger mechanism, predict the predicted data volume to be written to at least one memory space in N consecutive time windows, and generate a Flush request in each time window when the Flush request trigger condition is met. Finally, according to the RS load, distribute the generated at least one Flush request to each RS node to flush the data in each memory space on each service node to the corresponding hard disk.

[0052] On the contrary, if the target data table is the second type of data table, the original Flush request trigger mechanism will be used. When the data in a memory space reaches a preset threshold (for example, 100MB), a Flush request will be triggered and the data in the relevant memory space will be flushed to the corresponding hard disk.

[0053] Different from the traditional HBase that triggers a Flush mechanism at a preset interval, the embodiment of the present application uses a dynamic prediction mechanism to more accurately control the time point of triggering the Flush request, thereby avoiding unnecessary resource waste and performance bottlenecks.

[0054] In addition, based on the real-time load parameters of each RS node, the generated Flush requests are dynamically allocated to the service nodes corresponding to each time window. This means that the system will take into account indicators such as the current task volume, request latency, and Memstore data volume of each RS node to ensure that Flush requests can be executed first on RS with lower loads, avoiding system performance degradation caused by concentrated Flush requests. At the same time, it avoids the pressure on the same node caused by a large number of requests at the same time, and realizes the smooth implementation of data storage.

[0055] Through the above method, based on the judgment of the type of data table and the estimation of the amount of data written in at least one memory space, the triggering time of the data drop request of the first type of data table is dynamically predicted, and at least one drop request is evenly distributed to at least one service node corresponding to each time window according to the real-time load of each service node. In other words, by predicting the triggering time of the drop request in each time window and the real-time load of the service node, it is ensured that the drop request is distributed to each service node in turn according to the load level, so as to achieve global load balancing, avoid further congestion of high-load service nodes, improve the data writing performance and stability of the HBase database, and achieve the technical effect of improving the data processing capability of the database.

[0056] In an exemplary embodiment, the above-mentioned response to at least one data write request to write target data into the target data table determines the type of the target data table, including: obtaining the number of data write requests for the target data table in the current unit time; obtaining the total number of data write requests for at least one data table in the database cluster in the historical period; based on the total number, determining the average number of write requests for at least one data table in the historical unit time; when the ratio between the number of data write requests and the average number of write requests is less than the determination coefficient, determining the type of the target data table as a first type of data table.

[0057] In this embodiment, the type of the target data table may be determined by, but is not limited to, the following formula (1).

[0058] (1)

[0059] The number of write requests in the past 24 hours is only an example of a current unit time and is not limited thereto.

[0060] The number of write requests in the past 24 hours in the above formula (1) refers to the total number of requests to write to the target data table in the past 24 hours. When a data write request sent by the data end is obtained, the table name carried in the request has determined the data table in HBase to which the target data is written according to the current data write request, that is, the data table to be written is determined according to the data write request.

[0061] The historical average daily number of write requests for all tables in the above formula (1) may be, but is not limited to, the total number of data write requests for all data tables in the HBase cluster within a historical period. Then, based on the total number and the number of data tables, the historical average daily number of write requests for all tables is determined, thereby determining the judgment coefficient.

[0062] Through the above method, the system can identify which data tables belong to the first type of data tables and which data tables belong to the second type of data tables, laying a foundation for the implementation of the subsequent dynamic flush control strategy.

[0063] The method of introducing the determination coefficient to quantitatively determine the target data table type not only improves the accuracy of the table type determination result, but also provides a reliable data basis for the subsequent prediction mechanism and improves the actual prediction accuracy of the Flush request trigger.

[0064] At the same time, after determining that the data table is a cold table (first type data table), the system can adopt different management strategies for these tables, such as the dynamic Flush trigger mechanism, to avoid unnecessary consumption of resources. At the same time, by dynamically allocating Flush requests according to the load of RS nodes, resource utilization is further optimized to avoid affecting the overall performance due to excessive load on some nodes.

[0065] In an exemplary embodiment, the above-mentioned method predicts the predicted data volume written into at least one memory space within N consecutive time windows when the type indicates that the target data table is a first type data table, including: obtaining each data write request in turn from at least one data write request as a current data write request; determining a current table unit corresponding to the current data write request and a current memory space corresponding to the current table unit; and using Poisson distribution to predict the current predicted data volume written into the current memory space within N consecutive time windows, wherein the predicted data volume includes the current predicted data volume.

[0066] In this embodiment, a Memstore corresponding to a Region is taken as an example to describe the specific implementation process of predicting the data volume of a memory space.

[0067] The dynamic prediction mechanism predicts the Flush trigger mechanism based on historical writes, Memstore data volume and other indicators. When the total data volume S of a Memstore reaches 100MB (128×0.8, 80% of the HDFS file block size), the Flush request is triggered. The above S can be determined by, but is not limited to, the following formula (2).

[0068] (2)

[0069] When calculating the data volume of a Memstore according to formula (2), the priority is determined in descending order from top to bottom, that is, Poisson distribution is used first to predict the flush trigger timing, followed by exponential smoothing method, and finally the flush request is triggered according to the original flush mechanism.

[0070] The following formula (3) after Poisson distribution transformation predicts the total amount of data written to a Memstore in all time periods.

[0071] (3)

[0072] in, Represents a time window, for example, 5 minutes. Poisson distribution is often used to deal with write coefficients that are random and have no significant periodicity. According to Poisson distribution, the frequency of writing k times of data in the current time window can be predicted. The expected value of the write request in the current time window can be obtained by summing them up. The total number of write requests can be obtained by summing all time windows. The formula of Poisson distribution is as follows (4).

[0073] (4)

[0074] Where P is the probability of k in the time window, that is, the probability of writing k times in the time window; k is a non-negative integer (for example, k=0,1,2...), k represents the number of times the write request event occurs; λ is a positive real number (λ>0), which represents the average number of write request events in the unit time window, e is the base of the natural logarithm (about 2.71828), e is a constant, and k! represents the factorial of k, that is, .

[0075] It should be noted that in the predicted total amount of data written to a Memstore in all time periods mentioned in the above formula (3), all time periods can be, but are not limited to, the time when the last Flush request was triggered as the starting time, and the time included in all time windows from the starting time to the current time.

[0076] For example, suppose the current time is the third time window 3, then all the above time periods represent 1~ 3. Duration between.

[0077] In this embodiment, the predicted data write amount of a Memstore is taken as an example to describe the total data amount written to the memory space in all time periods using the Poisson distribution method. Obviously, the predicted data amount of the at least one memory space includes the sum of the predicted data write amounts of each memory space.

[0078] Poisson distribution is a statistical method that is suitable for describing the probability distribution of the number of events that occur in a fixed time, especially when the events occur infrequently and independently. Here, Poisson distribution is used to predict the current predicted amount of data written to Memstore in N consecutive time windows.

[0079] The Poisson distribution prediction model is based on historical write data. It can provide an expected value for the amount of data written, thereby helping the system predict when the flush threshold is reached, for example, 80% of 128MB. The effectiveness of this prediction model lies in its ability to predict future trends based on past write patterns, especially for data tables with relatively stable historical write frequencies.

[0080] By using Poisson distribution, the system can more accurately predict the amount of data written to Memstore in N consecutive time windows based on historical data, thereby predicting in advance when the cold table is close to the flush threshold, avoiding delayed flush problems caused by sudden increases in data volume, and enhancing the accuracy of the system's prediction of data write volume.

[0081] This prediction mechanism helps to manage the amount of data in Memstore more finely. Through dynamic monitoring and prediction, the system can adjust the Flush strategy in a timely manner to ensure that the data of the cold table is flushed to the disk at the appropriate time, avoiding excessive occupation of memory space and thus optimizing the efficiency of memory resource utilization.

[0082] In an exemplary embodiment, the above method also includes: obtaining the current actual data amount that has actually been written to the current memory space after responding to the current data write request; when the difference between the current predicted data amount and the current actual data amount is greater than or equal to a first threshold, using the exponential smoothing method to predict the current predicted data amount written to the current memory space within N consecutive time windows.

[0083] When it is detected that the business write request is written continuously and the deviation between the Poisson distribution prediction value and the actual Memstore data write amount is large (for example, when the difference is greater than 2M), switch to the exponential smoothing method to predict the total amount of data written in all time periods, which is specifically achieved through the following formula (5).

[0084] (5)

[0085] in, Indicates the amount of forecast data written into the memory space in the previous time window. is the smoothing factor (0< <1, usually 0.1~0.3) Indicates the amount of data actually written to the memory space within the current time window.

[0086] Assume that Figure 4 The three time windows shown are taken as an example for all time periods. For a Memstore corresponding to a Region, the Poisson distribution is used to predict that the total amount of data that may be written to the Memstore space is 50MB, and the space occupied by the data actually written to the Memstore space in these three time windows is 55MB. If the difference between the two is greater than the first threshold of 2MB, the exponential smoothing method is used to predict the total amount of data written to each Memstore in all time periods.

[0087] Obviously, as time goes by, all the above time periods change dynamically, for example, from the initial one time window to the first two time windows, the first three time windows, and so on.

[0088] Exponential smoothing is suitable for tables where data is written continuously and has small fluctuations, and has better prediction effects in the short term. It mainly predicts future values ​​by assigning exponentially decreasing weights to historical data. Its core assumption is that "recent data has more predictive value than long-term data."

[0089] The switching scheme between the exponential smoothing method and the Poisson distribution method provided in this embodiment makes the entire dynamic flush control framework more flexible, which ensures that the prediction model can be dynamically adjusted according to the actual situation to adapt to changes in the data writing mode. When the writing frequency of the cold table suddenly increases or the data volume fluctuates greatly, the Poisson distribution prediction method may not be able to mechanically and accurately reflect the current situation. At this time, it will switch to the exponential smoothing method. By giving higher weight to recent data, it can quickly capture the data change trend and ensure the real-time and accuracy of the prediction results.

[0090] By introducing the exponential smoothing method, the system can more accurately predict the trend of data volume changes, thereby triggering the Flush operation more timely and accurately when the data volume approaches the Flush trigger threshold, avoiding delays caused by sudden increases in data volume.

[0091] In an exemplary embodiment, the above-mentioned use of the exponential smoothing method to predict the current predicted data volume written to the current memory space within N consecutive time windows includes: predicting the i+1th data volume written to the current memory space within the i+1th time window through the following steps, where i is a positive integer greater than or equal to 2 and less than or equal to N: obtaining the i-1th data volume predicted to be written to the current memory space within the i-1th time window; obtaining the i-th actual data volume actually written to the current memory space within the i-th time window; determining the i+1th data volume based on the smoothing factor, the i-1th data volume and the i-th actual data volume, wherein the current predicted data volume includes the i+1th data volume.

[0092] In this embodiment, the calculation process of the exponential smoothing method is described by taking the calculation of the predicted data writing amount in the next time window as an example, as shown in the following formula (6).

[0093] (6)

[0094] in, is the predicted value based on the next time window, that is, the predicted data volume written into a memory space in the next time window of the current time window (which can also be understood as the predicted value), Vt is the actual observed value (also known as the actual value) at the current moment (also known as the current time window), is the smoothing factor (0< <1, usually 0.1~0.3).

[0095] The exponential smoothing method in the above formula (6) determines the amount of predicted data in the next time window based on the smoothing factor, the amount of predicted data in the previous time window, and the actual amount of data in the current time window. Among them, the selection of the smoothing factor is the key, which determines the degree of influence of historical data and current data on the prediction results. It is usually between 0.1 and 0.3 to ensure that the prediction result skill responds quickly to changes in data trends and is not overly affected by short-term fluctuations.

[0096] According to the recursive characteristics of the exponential smoothing method, its core is to use the smoothing factor to perform weighted averaging on historical data and new data. This calculation method can effectively reduce the impact of short-term fluctuations, improve the stability and accuracy of the prediction results, and make the predicted i-th data volume closer to the true value, providing a solid data foundation for the decision-making of the Flush operation.

[0097] In addition, by switching between the Poisson distribution and exponential smoothing methods, the data volume can be accurately predicted, helping the system to plan Flush operations in advance and avoiding resource scheduling confusion caused by sudden changes in data volume. This not only improves the accuracy and reliability of the prediction results, but also enhances the robustness of the dynamic adaptability of the prediction model. It has significant technical effects and practical application value for optimizing the timing of Flush operations on cold tables and improving the performance and stability of HBase clusters when processing low-frequency writes to data tables.

[0098] In an exemplary embodiment, the above-mentioned generating at least one disk drop request when the predicted data volume meets the disk drop request trigger condition includes: generating at least one disk drop request when the predicted data volume is greater than or equal to a second threshold, wherein the second threshold is determined based on the space upper limit value of one memory space in at least one memory space.

[0099] This embodiment is the execution layer in the dynamic Flush control framework. According to the predicted data volume in at least one memory space in the above embodiment and the pre-set second threshold, the triggering timing of the disk request is determined, that is, when the data volume in a Memstore reaches the second threshold, the generation of a Flush request is triggered.

[0100] For example, if Figure 5 As shown, assuming that the time window is predicted according to the above Poisson distribution or exponential smoothing method 1~ 2, the amount of data written to a memory space is 50MB and 55MB respectively. Since the sum of the two, 105, exceeds the second threshold of 100MB, then in the third time window 3. The trigger generates a Flush request.

[0101] It should be noted that after the Flush request is generated, the RS node is determined according to the Region corresponding to the Flush request, and the Flush request is sent to the Flush request queue of the RS node maintained by the Flush collaborative scheduling module.

[0102] That is to say, within a time window, multiple Flush requests are allowed to be triggered in parallel at the same time, but the multiple Flush requests are not sent to the RS node first, but are temporarily sent to the Flush request queue.

[0103] This embodiment can achieve refined management of the Memstore data volume. The prediction mechanism dynamically predicts the data volume within N consecutive time windows. When the predicted value is close to the second threshold, the system can prepare for the Flush operation in advance, avoiding the data volume exceeding the Memstore upper limit, thereby protecting memory resources and reducing system abnormalities caused by memory overflow.

[0104] The second threshold is set based on 80% of the Memstore space upper limit. This strategy ensures that Flush is triggered when the data volume reaches a reasonable threshold. It avoids the delayed Flush problem caused by a sudden increase in data volume and prevents resource waste caused by frequent Flush due to a small amount of data, thus achieving the optimal timing selection for the Flush operation.

[0105] In combination with the description in the above embodiments, it can be seen that whether it is to predict the amount of data written to each memory space by dynamically switching the Poisson distribution and the exponential smoothing method, or to dynamically predict multiple Flush requests allowed to be triggered in parallel within each time window by comparing the predicted data amount with the second threshold, it reflects the flexibility and rationality of dynamically predicting the triggering timing of Flush.

[0106] In an exemplary embodiment, the above method also includes: in the i-th time window of N consecutive time windows, predicting the predicted sub-data volume of each memory space in at least one memory space to obtain a set of predicted sub-data volumes; based on the set of predicted sub-data volumes, determining a set of cumulative predicted data volumes for each memory space in i consecutive time windows; when at least part of the cumulative predicted data volumes in a set of cumulative predicted data volumes is greater than a third threshold, simultaneously triggering a set of disk placement requests corresponding to at least part of the memory spaces in at least one memory space in the i-th time window.

[0107] For example, assuming that at least one data write request is 3 data write requests, determine Region1, Region2 and Region3 corresponding to the 3 data write requests respectively, then in the second time window, predict the predicted sub-data volume of each memory space in Region1~Region3 in parallel, and at the same time determine the total amount of data predicted to be written in each Region in the first two time windows.

[0108] Determine the size of the predicted total amount of written data for each Region in all time periods (in this embodiment, the first two time windows) and the second threshold, and determine the number of Flush requests triggered in each time window when the predicted total amount of written data is greater than or equal to the second threshold.

[0109] For example, assuming that the total amount of predicted data written to Region1 and Region2 in the first two time windows is greater than the second threshold, then the Flush requests corresponding to Region1 and Region2 will be triggered in parallel in the third time window.

[0110] By predicting the data volume of all memory spaces in the HBase cluster in this embodiment, the system can respond to the growth of data volume in each memory space in a timely manner, and can globally identify and handle situations where multiple memory spaces are approaching or exceeding their space limits at the same time, effectively avoiding performance bottlenecks and stability problems that may be caused by local overloads.

[0111] Through the sub-data volume prediction of N consecutive time windows and the subsequent cumulative prediction, this embodiment can provide a more comprehensive and in-depth analysis of the data volume growth trend, help the system accurately predict the usage of cluster resources, and provide a solid data basis for intelligent decision-making.

[0112] In other words, by introducing global trend prediction and optimized management of cluster resources, not only the prediction accuracy of Memstore data volume is improved, but also the intelligent allocation and management of the entire HBase cluster resources are realized, which has a significant technical effect on improving system stability, response speed and overall performance. This not only demonstrates the optimization potential of resource management strategies in the field of big data processing, but also reflects the system's intelligent adjustment capabilities in dealing with dynamic business scenarios and resource allocation challenges, which has far-reaching significance for promoting the application of HBase clusters in the field of real-time data processing and storage.

[0113] In an exemplary embodiment, the above-mentioned dynamically allocating at least one disk drop request to at least one service node corresponding to each time window based on the load parameters of at least one service node includes: according to the mapping relationship between at least one service node and at least one table unit, respectively sending a disk drop request to at least one disk drop request queue of at least one service node; determining the load parameters of each service node in each time window based on the performance indicators of at least one service node, wherein the performance indicators include the number of service node tasks and the service node request delay; dynamically adjusting the number of disk drop request slots of each service node in each time window based on the load parameters, wherein the disk drop request slot is used to carry the disk drop requests in at least one disk drop request queue; based on the number of disk drop request slots, dynamically allocating at least one disk drop request to at least one service node corresponding to each time window.

[0114] When the Flush request triggering condition is determined to be met through the technical solution in the above embodiment, a group of Flush requests corresponding to at least one memory space (which can also be understood as a group of disk drop requests) is triggered and generated. At this time, the group of Flush requests will not be immediately distributed to the Flush request slot of each RS node, but the Flush request will be first sent to the Flush request queue of the corresponding RS node.

[0115] The RS node to which the memory space belongs is determined according to the Region corresponding to each Flush request, and each RS node maintains a Flush request queue, wherein the number of Flush requests allowed to be carried in the request queue is preset.

[0116] Based on the performance indicators of each service node, the load parameters of the service node in each time window are generated, where the performance indicators include but are not limited to Figure 3 In addition to the number of RS request tasks and RS request delay shown, it also includes network status, disk I / O, central processing unit (CPU) and other indicators. When calculating the load parameters of the service node, at least two of the above multiple performance indicators can be randomly selected for the load calculation process, which is not limited in the embodiments of the present application.

[0117] The reason for calculating the load parameter of each service node is that the lower the load of a service node, the more Flush requests are allowed to be allocated to the service node; conversely, the higher the load of a service node, the fewer Flush requests are allowed to be allocated to the service node.

[0118] By calculating the load of each service node in each time window, the number of Flush requests allowed to be received by each service node on RS nodes with different loads in different time windows can be accurately controlled, thereby achieving global load balancing and ensuring full utilization of resources.

[0119] Secondly, within the Flush collaborative scheduling module, based on the load parameters of each service node and the Flush requests in the Flush request queue maintained by each service node, each Flush request in the Flush request queue is distributed to the Flush request slots of the corresponding service nodes in different time windows according to the first-in-first-out rule.

[0120] It should be noted that within each time window, the number of Flush request slots varies according to the load of the service node. The purpose is to ensure that different numbers of Flush requests are distributed according to the RS node load to achieve global load balancing.

[0121] By introducing the Flush request queue and Flush request slot of the RS node (which can also be understood as the drop request slot), the dynamic Flush control framework in the embodiment of the present application is further improved, and by dynamically allocating the number of Flush requests and Flush request slots, accurate management of RS node load and global resource optimization are achieved. In other words, this embodiment not only takes into account data prediction and the generation of drop requests, but also goes deep into the allocation of requests and node load monitoring, forming a complete closed loop from data prediction to resource management, which significantly improves the performance and stability of the HBase cluster when processing a large number of low-frequency write tables.

[0122] In an exemplary embodiment, the above-mentioned dynamic adjustment of the number of disk request slots of each service node in each time window based on load parameters includes: dynamically generating a virtual node adjustment coefficient for each service node in each time window based on the number of service node tasks, service node request delay and load balancing factor, wherein the larger the value of the virtual node adjustment coefficient, the lower the load of each service node; determining the number of virtual nodes of each service node in each time window based on the virtual node adjustment coefficient; generating a preset number of virtual node slots for each service node in each time window based on the number of virtual nodes; and dynamically adjusting the number of disk request slots for each service node in each time window based on the virtual node slots and N consecutive time windows.

[0123] In this embodiment, the RS load is calculated based on the statistical RS indicator data, and the number of virtual nodes is dynamically adjusted according to the calculated RS load, wherein the number of RS virtual nodes with higher load is smaller, and the number of RS virtual nodes with lower load is larger. The purpose is to enable the service nodes with lower load to receive more Flush requests within the current time window, to achieve global load balancing, and to ensure that resources can be fully utilized.

[0124] In order to describe the load size of the RS node, a load balancing factor L is designed in the embodiment of the present application. L can be determined by, but is not limited to, the following formula (7).

[0125] L= (7)

[0126] Where W represents the average number of pending tasks in the system, Y represents the average system delay, w represents the number of pending tasks in the current node, and y represents the average delay in the current node. The virtual node coefficient L is adjusted according to the current load of the service node. The number of virtual nodes corresponding to a physical RS node is usually between 100 and 1000.

[0127] When L is a positive number, it means that the number of tasks on the current node is less than the average value, that is, the load is low, and the number of virtual nodes can be increased; when L is a negative number, the number of tasks on the current node is greater than the average value, that is, the load is large, and the number of virtual nodes can be reduced. In other words, the larger L is, the lower the load of the RS node is, and the more RS virtual nodes there are.

[0128] The number of virtual nodes may be calculated by, but is not limited to, the following formula (8).

[0129] (8)

[0130] in, represents the load balancing factor of the i-th service node in the current time window, Indicates the minimum value of the load balancing factor, Indicates the maximum value of the load balancing factor.

[0131] In this embodiment, a design method for RS virtual node slots is also introduced. Specifically, based on the consideration of different loads of RS nodes and Flush request load balancing, multiple RS virtual node slots with unique identifiers are generated for each physical node (i.e., RS node). The mapping relationship between each virtual node and the physical node is as follows: Figure 6 As shown, the virtual node ID = physical node ID + separator + sequence number, for example, regionserver-5#0001, regionserver-5#0002, regionserver-5#0010.

[0132] According to the combination of RS virtual node slots and time windows, each service node is dynamically adjusted to allow the bearer to evenly distribute the Flush requests in each Flush request queue to each RS node within each time window.

[0133] By adopting the above method, the dynamic generation of virtual node adjustment coefficients and the flexible adjustment of the number of virtual node slots can achieve refined balancing of RS node load and efficient use of resources. This solution not only takes into account data prediction and the generation of disk drop requests, but also further optimizes the allocation of flush requests and node load management, improving the performance and stability of HBase cluster processing data writes.

[0134] In an exemplary embodiment, the above-mentioned method of dynamically allocating at least one disk drop request to at least one service node corresponding to each time window based on the number of disk drop request slots includes: obtaining the i-th group of virtual node slots corresponding to the i-th service node in the i-th time window, wherein i is a positive integer greater than or equal to 2 and less than or equal to N; determining the i-th group of disk drop request slots of the i-th service node in the i-th time window based on the i-th group of virtual node slots and the i-th time window; and allocating the i-th group of disk drop requests in the i-th disk drop request queue of the i-th service node to the i-th group of disk drop request slots based on the number of the i-th group of disk drop request slots, wherein the number of the i-th group of disk drop request slots is greater than or equal to the number of disk drop requests in the i-th group of disk drop requests, and the number of the i-th group of disk drop request slots is determined based on the load upper limit of the i-th service node.

[0135] like Figure 7 As shown, in the embodiment of the present application, a combination of RS virtual node slots and time partition slots (which can also be understood as time windows) is adopted to distribute the Flush requests pre-stored in the Flush request queue to the disk request slots (which can also be understood as Flush request slots) corresponding to each RS node in a first-in-first-out order.

[0136] It should be noted that an RS virtual node slot and a time window can be combined to generate a flush request slot, which is used to carry the flush requests in the flush request queue. Within the time granularity of each time partition slot, the load of all service nodes is recalculated, and different numbers of virtual nodes are generated according to the calculated load. One virtual node corresponds to one virtual node slot.

[0137] According to the different numbers of virtual nodes generated, the number of corresponding virtual node slots is determined, and then the number of corresponding virtual node slots is determined according to the virtual node slots and the number of corresponding virtual nodes. Figure 7The combination of the time partition slots shown generates the Flush request slot of the current service node in the current time window, and distributes the Flush requests to the corresponding Flush request slots in sequence according to the order of the Flush requests in the Flush request queue sent to the current service node.

[0138] In addition, when the RS node is dynamically expanded or reduced, it will trigger the migration of Region (from other nodes to the expanded or reduced RS node). When expanding, the Region deployed on other nodes will be migrated to the current RS node, and when reducing, the Region will be migrated out of the current RS node.

[0139] The above-mentioned flush request slot based on the combination of RS virtual node slot and time window can realize load balancing and overload current limiting. According to the RS load, the number of flush requests that can be carried in the time partition slot is limited, so as to ensure that the requests in the flush request queue corresponding to the RS are consumed according to its capacity.

[0140] When the Flush request slot cannot consume all the Flush requests generated by the current time partition slot, it will be postponed to the next time partition slot to continue consumption. When the current time partition slot generates fewer Flush requests, the Flush request slot generated by RS may be idle.

[0141] By adopting the above method, the Flush requests that are triggered centrally in a short time according to preset intervals in the prior art can be dispersed to different RS nodes in different time windows, avoiding the accumulation of RS queues caused by a large number of Flush requests, which leads to a surge in the time delay of normal read and write requests, and at the same time reduces the pressure on the disk load caused by concentrated Flush.

[0142] Through the dynamic scheduling mechanism in the embodiment of the present application, the system can effectively respond to fluctuations in business demand and changes in node load, ensuring the rational allocation and efficient utilization of cluster resources. It not only improves the stability of the system when processing a large number of low-frequency write tables, but also enhances the scalability of the cluster, providing a solid technical foundation for future business growth and system expansion.

[0143] In addition, by dynamically determining the number of disk request slots, not only is the monitoring and management accuracy of the service node's request processing capabilities improved, but also the reasonable allocation of requests and efficient use of resources are achieved, the performance and stability of the HBase cluster when processing low-frequency writes to tables are improved, and the system's ability to cope with complex business scenarios is improved.

[0144] In an exemplary embodiment, the method further includes: in a case where at least one disk drop request includes a first part of disk drop requests generated by a first type of data table and a second part of disk drop requests generated by a second type of data table, responding to the second part of disk drop requests has a higher priority than responding to the first part of disk drop requests; or in a case where at least one disk drop request includes only a third part of disk drop requests generated by a first type of data table, responding to each disk drop request in sequence according to the chronological order of generation time of each disk drop request in the third part of disk drop requests; or in a case where at least one disk drop request includes only a fourth part of disk drop requests generated by a second type of data table, responding to each disk drop request in sequence according to the chronological order of generation time of each disk drop request in the fourth part of disk drop requests.

[0145] In order to enrich the request response strategy of the dynamic flush control framework in the embodiment of the present application, by distinguishing the disk write requests generated by the first type of data table (hot table) and the second type of data table (cold table), and setting different response priorities based on the request type and generation time, personalized management of data writing to tables with different characteristics is achieved. The core of this strategy is to intelligently adjust the response order according to the characteristics and generation time of the request, ensure the rapid disk write of key data and the orderly processing of non-key data, and improve the overall response speed and resource utilization efficiency of the system.

[0146] Specifically, when the flush request queue contains flush requests from both hot tables and cold tables, the hot table's requests, i.e., the second part of the flush requests, are responded to first. This is because the hot table has a higher frequency of data writes, which has a more significant impact on system performance and response speed. Prioritizing processing can effectively avoid business delays and improve overall system performance. When there are only flush requests from the first type of data table (hot table) or the second type of data table (cold table), the system responds in the order in which the flush requests are generated. This strategy ensures the fairness and order of data processing and avoids data loss or processing confusion.

[0147] Through the above method, not only the request response mechanism of the dynamic flush control framework is improved, but also it is demonstrated how to optimize resource management and business response through intelligent algorithms and dynamic strategies in the field of big data processing, and improve the system's ability to cope with complex business scenarios.

[0148] By distinguishing the data table types and setting the request response priority, as well as the strategy of responding in the order of generation time, not only the processing flow of different types of data tables is optimized, but also the reasonable allocation and utilization efficiency of resources are improved. It has a significant technical effect on improving the performance and stability of the HBase cluster when processing low-frequency and high-frequency write tables, which not only ensures the rapid response of key businesses, but also ensures the orderly processing of non-key businesses, improves the overall performance of the system, and enhances the system's adaptability and stability.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0150] According to another aspect of the embodiment of the present application, a data storage system is also provided. The structural diagram of the system is shown in FIG. Figure 8 As shown, it includes a monitoring module 802 , a disk drop request control module 804 and a collaborative scheduling module 806 .

[0151] The monitoring module 802 is a monitoring module used to obtain at least one data write request to write target data into a target data table, wherein the target data table is divided into at least one table unit, and at least one table unit corresponds to at least one memory space.

[0152] The disk drop request control module 804 is used to determine the type of the target data table and, when the type indicates that the target data table is a first type data table, predict the amount of data to be written into at least one memory space within N consecutive time windows; and generate at least one disk drop request based on the predicted amount of data, where N is a positive integer greater than or equal to 1.

[0153] The collaborative scheduling module 806 is used to dynamically allocate at least one disk placement request to at least one service node corresponding to each time window based on the load parameters of at least one service node, so as to place the data in at least one memory space on at least one hard disk corresponding to at least one service node.

[0154] like Figure 3 As shown in the figure, the monitoring module (or system monitoring module) is responsible for collecting indicators such as the number of data write requests sent by the client, the number of RS request tasks sent by the server, RS request latency, and Memstore data volume to ensure that the write operation of the target data table is accurately monitored. These indicators are used for RS load calculation, target data table type determination, and prediction result evaluation.

[0155] The target data table is divided into multiple table units, each of which corresponds to a memory space (Memstore). This mechanism ensures load balancing and efficient processing of data writing.

[0156] The disk drop request control module is a key component of the technical solution of the present application. It is mainly used to determine the type of the target data table and implement a dynamic Flush judgment mechanism based on time prediction. The table type is determined by predicting the Flush trigger timing according to the Poisson distribution or exponential smoothing method, and the RS node is determined according to the region corresponding to the Flush request. The Flush request is sent to the Flush request queue of the RS node maintained by the Flush collaborative scheduling module.

[0157] Specifically, the above formula (1) is used to determine the type of the target data table and distinguish the different characteristics of hot and cold tables. For cold tables (first type data tables), the module predicts the change in the amount of data in Memstore within N consecutive time windows based on historical data and prediction models. When the predicted amount of data reaches the set threshold, a request to flush the data to disk is generated. This strategy ensures that the data in the cold table can be processed in a timely manner when it is close to the flush threshold, avoids the situation where a large number of requests are triggered in a concentrated manner, and reduces the risk of disk I / O storms.

[0158] The collaborative scheduling module dynamically generates different numbers of virtual nodes in each time interval based on RS node monitoring indicators and pre-set time intervals, and generates Flsuh request slots that carry Flush requests. The module maintains a Flush request queue for each RS node and distributes the Flush requests in the queue to the Flsuh request slots in a first-in-first-out order. In each time window, the number of request slots changes according to the load, ensuring that different numbers of Flush requests are distributed according to the node load to achieve global load balancing.

[0159] The specific execution steps involved in the various calculation processes in the above modules and the dynamic prediction of Flush timing can refer to the description in the above embodiments, which will not be repeated here.

[0160] The dynamic Flush control framework in the above data disk system provided by this application optimizes the Flush mechanism of HBase cluster MemStore. At the same time, based on the combination of dynamic RS virtual node slots and time partition slots of RS load, the number of RS virtual node slots is dynamically calculated in each time window, and the Flush request slot is generated according to its carrying capacity, and the Flush request queue corresponding to the RS is consumed to ensure the overall load balance between all RSs and on the timeline.

[0161] Obviously, the data storage system can be used to implement the data storage method provided in the above embodiments, and the description has been made no further. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0162] It should be noted that the monitoring module 802 in this embodiment can be used to execute the above step S202, the disk drop request control module 804 in this embodiment can be used to execute the above steps S204 and S206, and the collaborative scheduling module 806 in this embodiment can be used to execute the above steps S208 and S210.

[0163] Through the embodiments provided by the present application, based on the judgment of the type of data table and the estimation of the amount of data written in at least one memory space, the triggering time of the data drop request of the first type of data table is dynamically predicted, and at least one drop request is evenly distributed to at least one service node corresponding to each time window according to the real-time load of each service node. In other words, by predicting the triggering time of the drop request in each time window and the real-time load of the service node, it is ensured that the drop request is distributed to each service node in turn according to the load level, so as to achieve global load balancing, avoid further congestion of high-load service nodes, improve the data writing performance and stability of the HBase database, and achieve the technical effect of improving the data processing capability of the database.

[0164] In an exemplary embodiment, the above-mentioned disk write request control module 804 includes: a first acquisition unit, used to obtain the number of data write requests for the target data table in the current unit time; a second acquisition unit, used to obtain the total number of data write requests for at least one data table in the database cluster in the historical period; a first processing unit, used to determine the average number of write requests for at least one data table in the historical unit time based on the total number; a second processing unit, used to determine the type of the target data table as a first type of data table when the ratio between the number of data write requests and the average number of write requests is less than the determination coefficient.

[0165] In an exemplary embodiment, the above-mentioned disk write request control module 804 includes: a third acquisition unit, which is used to sequentially acquire each data write request from at least one data write request as a current data write request; a third processing unit, which is used to determine a current table unit corresponding to the current data write request and a current memory space corresponding to the current table unit; a first prediction unit, which is used to predict the current predicted data volume written to the current memory space within N consecutive time windows using Poisson distribution, wherein the predicted data volume includes the current predicted data volume.

[0166] In an exemplary embodiment, the above system also includes: a first acquisition module, used to obtain the current actual data amount that has been actually written to the current memory space after responding to the current data write request; a first writing module, used to predict the current predicted data amount written to the current memory space within N consecutive time windows using an exponential smoothing method when the difference between the current predicted data amount and the current actual data amount is greater than or equal to a first threshold.

[0167] In an exemplary embodiment, the above-mentioned first writing module includes: a fourth processing unit, used to predict the i-th data amount written to the current memory space within the i-th time window through the following steps, where i is a positive integer greater than or equal to 2 and less than or equal to N: obtain the i-1th data amount predicted to be written to the current memory space within the i-1th time window; obtain the i-th actual data amount actually written to the current memory space within the i-th time window; determine the i-th data amount based on the smoothing factor, the i-1th data amount and the i-th actual data amount, wherein the current predicted data amount includes the i-th data amount.

[0168] In an exemplary embodiment, the disk drop request control module 804 includes: a fifth processing unit, which is used to generate at least one disk drop request when the predicted data volume is greater than or equal to a second threshold, wherein the second threshold is determined based on a space upper limit value of one memory space in at least one memory space.

[0169] In an exemplary embodiment, the system further includes: a first prediction module, used to predict the predicted sub-data volume of each memory space in at least one memory space in the i-th time window of N consecutive time windows to obtain a set of predicted sub-data volumes; a first processing module, used to determine a set of cumulative predicted data volumes for each memory space in i consecutive time windows based on the set of predicted sub-data volumes; and a second processing module, used to simultaneously trigger a set of disk placement requests corresponding to at least part of the memory spaces in at least one memory space in the i-th time window when at least part of the cumulative predicted data volumes in the set of cumulative predicted data volumes is greater than a third threshold.

[0170] In an exemplary embodiment, the collaborative scheduling module 806 includes: a sending unit, which is used to send a disk drop request to at least one disk drop request queue of at least one service node according to the mapping relationship between at least one service node and at least one table unit; a sixth processing unit, which determines the load parameters of each service node in each time window based on the performance indicators of at least one service node, wherein the performance indicators include the number of service node tasks and the service node request delay; a first adjustment unit, which is used to dynamically adjust the number of disk drop request slots of each service node in each time window based on the load parameters, wherein the disk drop request slots are used to carry the disk drop requests in at least one disk drop request queue; a first allocation unit, which is used to dynamically allocate at least one disk drop request to at least one service node corresponding to each time window based on the number of disk drop request slots.

[0171] In an exemplary embodiment, the above-mentioned first adjustment unit includes: a first processing sub-unit, used to dynamically generate a virtual node adjustment coefficient for each service node in each time window based on the number of service node tasks, the service node request delay and the load balancing factor, wherein the larger the value of the virtual node adjustment coefficient, the lower the load of each service node; a second processing sub-unit, used to determine the number of virtual nodes for each service node in each time window based on the virtual node adjustment coefficient; a third processing sub-unit, used to generate a preset number of virtual node slots for each service node in each time window based on the number of virtual nodes; an adjustment sub-unit, used to dynamically adjust the number of disk request slots for each service node in each time window based on the virtual node slots and N consecutive time windows.

[0172] In an exemplary embodiment, the above-mentioned first allocation unit includes: a first acquisition subunit, used to obtain the i-th group of virtual node slots corresponding to the i-th service node in the i-th time window, wherein i is a positive integer greater than or equal to 2 and less than or equal to N; a fourth processing subunit, used to determine the i-th group of disk drop request slots of the i-th service node in the i-th time window based on the i-th group of virtual node slots and the i-th time window; the first allocation subunit, used to allocate the i-th group of disk drop requests in the i-th disk drop request queue of the i-th service node to the i-th group of disk drop request slots based on the number of the i-th group of disk drop request slots, wherein the number of the i-th group of disk drop request slots is greater than or equal to the number of disk drop requests in the i-th group of disk drop requests, and the number of the i-th group of disk drop request slots is determined based on the load upper limit of the i-th service node.

[0173] In an exemplary embodiment, the data drop system further includes: a third processing module, for responding to the second partial drop request with a higher priority than the first partial drop request when at least one drop request includes a first partial drop request generated by a first type of data table and a second partial drop request generated by a second type of data table; or for responding to each drop request in sequence according to the chronological order of generation time of each drop request in the third partial drop request when at least one drop request includes only a fourth partial drop request generated by a second type of data table; or for responding to each drop request in sequence according to the chronological order of generation time of each drop request in the fourth partial drop request when at least one drop request includes only a fourth partial drop request generated by a second type of data table.

[0174] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0175] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data disk-push method embodiments.

[0176] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data storage method embodiments when running.

[0177] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0178] According to another aspect of the embodiments of the present application, a computer program product is further provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data storage method embodiments are implemented.

[0179] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data storage method embodiments are implemented.

[0180] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0181] The above is a detailed introduction to a method for storing data on a disk provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for storing data on a disk, characterized in that: include: In response to at least one data write request to write target data into a target data table, determining a type of the target data table, wherein the target data table is divided into at least one table unit, and the at least one table unit corresponds to at least one memory space in a one-to-one manner; In the case where the type indicates that the target data table is a first type data table, predicting the amount of data written into the at least one memory space within N consecutive time windows, where N is a positive integer greater than or equal to 1; If the predicted data volume meets the disk-to-disk request triggering condition, generating at least one disk-to-disk request; Based on a load parameter of at least one service node, dynamically allocating the at least one disk placement request to the at least one service node corresponding to each time window; The data in the at least one memory space are respectively written to at least one hard disk corresponding to the at least one service node.

2. The method according to claim 1, characterized in that The step of determining the type of the target data table in response to at least one data write request to write target data into the target data table comprises: Obtain the number of data write requests for the target data table within a current unit time; Get the total number of data write requests for at least one data table in the database cluster during the historical period; Based on the total number of times, determining an average number of write requests for the at least one data table within a historical unit time; When the ratio between the number of data write requests and the average number of write requests is smaller than the determination coefficient, the type of the target data table is determined to be a first type data table.

3. The method according to claim 1, characterized in that When the type indicates that the target data table is a first type data table, predicting the predicted data amount written into the at least one memory space within N consecutive time windows includes: sequentially acquiring each data write request from the at least one data write request as a current data write request; Determine a current table unit corresponding to the current data write request and a current memory space corresponding to the current table unit; Using Poisson distribution, a current predicted data volume written to the current memory space within the N consecutive time windows is predicted, wherein the predicted data volume includes the current predicted data volume.

4. The method according to claim 3, characterized in that: The method further comprises: Acquire a current actual amount of data that has been actually written to the current memory space after responding to the current data write request; When the difference between the current predicted data volume and the current actual data volume is greater than or equal to a first threshold, the current predicted data volume written into the current memory space within the N consecutive time windows is predicted using an exponential smoothing method.

5. The method according to claim 4, characterized in that The method of using the exponential smoothing method to predict the current predicted data volume written into the current memory space within the N consecutive time windows includes: The i+1th amount of data written to the current memory space in the i+1th time window is predicted by the following steps, where i is a positive integer greater than or equal to 2 and less than or equal to N: Obtaining the i-1th amount of data predicted to be written to the current memory space within the i-1th time window; Obtaining the i-th actual amount of data actually written to the current memory space within the i-th time window; Based on the smoothing factor, the i-1th data amount and the i-th actual data amount, the i+1th data amount is determined, wherein the current predicted data amount includes the i+1th data amount.

6. The method according to claim 1, characterized in that The generating at least one disk-push request when the predicted data volume meets the disk-push request triggering condition includes: When the predicted data volume is greater than or equal to a second threshold, the at least one disk write request is generated, wherein the second threshold is determined based on a space upper limit value of one of the at least one memory space.

7. The method according to claim 1, characterized in that The method further comprises: In an i-th time window of the N consecutive time windows, predicting a predicted sub-data amount of each memory space in the at least one memory space to obtain a set of predicted sub-data amounts; Based on the set of predicted sub-data amounts, determining a set of cumulative predicted data amounts for each memory space within i consecutive time windows; When at least part of the accumulated predicted data amounts in the group is greater than a third threshold, a group of disk write requests corresponding to at least part of the memory spaces in the at least one memory space are simultaneously triggered within the i-th time window.

8. The method according to claim 1, characterized in that: The dynamically allocating the at least one disk placement request to the at least one service node corresponding to each time window based on the load parameter of the at least one service node includes: According to the mapping relationship between the at least one service node and the at least one table unit, the one disk placement request is respectively sent to at least one disk placement request queue of the at least one service node; Determining a load parameter of each service node in each time window based on a performance indicator of the at least one service node, wherein the performance indicator includes a number of service node tasks and a service node request delay; Based on the load parameter, dynamically adjust the number of disk drop request slots of each service node in each time window, wherein the disk drop request slots are used to carry the disk drop requests in the at least one disk drop request queue; Based on the number of the disk placement request slots, the at least one disk placement request is dynamically allocated to the at least one service node corresponding to each time window.

9. The method according to claim 8, characterized in that The dynamically adjusting the number of the disk request slots of each service node in each time window based on the load parameter includes: Based on the number of tasks of the service node, the request delay of the service node and the load balancing factor, dynamically generate a virtual node adjustment coefficient of each service node in each time window, wherein the larger the value of the virtual node adjustment coefficient is, the lower the load of each service node is; Determining the number of virtual nodes of each service node within each time window based on the virtual node adjustment coefficient; Based on the number of the virtual nodes, generating a preset number of virtual node slots for each service node in each time window; Based on the virtual node slot and N consecutive time windows, the number of disk drop request slots of each service node in each time window is dynamically adjusted.

10. The method according to claim 8, characterized in that The dynamically allocating the at least one disk placement request to the at least one service node corresponding to each time window based on the number of disk placement request slots includes: Get the i-th group of virtual node slots corresponding to the i-th service node in the i-th time window, where i is a positive integer greater than or equal to 2 and less than or equal to N: Based on the i-th group of virtual node slots and the i-th time window, determine the i-th group of disk placement request slots of the i-th service node within the i-th time window; Based on the number of the i-th group of disk drop request slots, the i-th group of disk drop requests in the i-th disk drop request queue of the i-th service node is allocated to the i-th group of disk drop request slots, wherein the number of the i-th group of disk drop request slots is greater than or equal to the number of disk drop requests in the i-th group of disk drop requests.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: In the case where the at least one disk placement request includes a first partial disk placement request generated by the first type of data table and a second partial disk placement request generated by the second type of data table, a priority of responding to the second partial disk placement request is higher than a priority of responding to the first partial disk placement request; or In a case where the at least one disk placement request only includes the third part of disk placement requests generated by the first type of data table, responding to each of the disk placement requests in the third part in the order of the generation time of each of the disk placement requests; or In the case that the at least one disk placement request only includes the fourth part of disk placement requests generated by the second type of data table, each of the disk placement requests is responded to in sequence according to the time sequence of generating each of the fourth part of disk placement requests.

12. A data storage system, characterized in that: include: A monitoring module, configured to obtain at least one data write request for writing target data into a target data table, wherein the target data table is divided into at least one table unit, and the at least one table unit corresponds to at least one memory space in a one-to-one manner; a disk write request control module, configured to determine a type of the target data table and, when the type indicates that the target data table is a first type data table, predict a predicted amount of data to be written into the at least one memory space within N consecutive time windows; and generate at least one disk write request based on the predicted amount of data, wherein N is a positive integer greater than or equal to 1; A collaborative scheduling module is used to dynamically allocate the at least one disk placement request to the at least one service node corresponding to each time window based on the load parameters of the at least one service node, so as to place the data in the at least one memory space on the at least one hard disk corresponding to the at least one service node.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for storing data on a disk as claimed in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for storing data on a disk as claimed in any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for storing data on a disk as claimed in any one of claims 1 to 11 are implemented.

Images