Data Disk Landing Method and System, Electronic Device, and Storage Medium

Abstract

The present application discloses a data disk dropping method and system, an electronic device, and a storage medium. The method includes: when at least one data writing request is obtained, determining the type of the target data table, and in the case where the target data table is a first type of data table, predicting the predicted data volume written into at least one memory space within N consecutive time windows; determining the timing of triggering a disk dropping request within each time window based on the predicted data volume, and generating at least one disk dropping request when the disk dropping request triggering condition is satisfied; based on the load parameters, allocating at least one disk dropping request to at least one service node corresponding to each time window; so as to respectively drop the data in each memory space to the hard disks corresponding to at least one service node. By this method, the technical problem of low data processing capacity of distributed databases in the related art is solved, and the technical effect of improving the data processing capacity of distributed databases is achieved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data disk writing method and system, an electronic device, and a storage medium. Background Art

[0002] In the HBase database, for data tables with a low writing frequency, the data accumulation in the Memstore (the memory space of the RS node) is relatively slow. In related technologies, it is usually dependent on the HBase service to perform regular checks at preset intervals to determine whether the data in the Memstore needs to be written to the hard disks corresponding to each RS node. Here, the RS (RegionServer) is used to provide read and write services externally.

[0003] However, the above-mentioned regular check method may cause a large number of Flush requests (data disk writing requests) to be triggered intensively in a short period of time, resulting in problems such as a sharp increase in the latency of normal read and write requests and excessive pressure on the hard disk load, thus causing the technical problem of low data processing capacity of the HBase database. Summary of the Invention

[0004] This application provides a data disk writing method and system, an electronic device, and a storage medium to at least solve the problem of low data processing capacity of the HBase database in related technologies.

[0005] According to one aspect of the embodiments of this application, a data disk writing method is provided, including: in response to at least one data writing request for writing target data into a target data table, determining the type of the target data table, where the target data table is divided into at least one table unit, and the at least one table unit corresponds to at least one memory space one by one; in the case where the type indicates that the target data table is a first type data table, predicting the predicted data volume written into the at least one memory space within consecutive N time windows, where N is a positive integer greater than or equal to 1; in the case where the predicted data volume meets the disk writing request triggering condition, generating at least one disk writing request; based on the load parameters of at least one service node, dynamically allocating the at least one disk writing request to the at least one service node corresponding to each time window; and writing the data in the at least one memory space to the at least one hard disk corresponding to the at least one service node respectively.

[0006] According to another aspect of the embodiments of the present application, there is also provided a data disk-down system, including: a monitoring module, configured to obtain at least one data write request for writing target data into a target data table, where the target data table is divided into at least one table unit, and the at least one table unit corresponds to at least one memory space one by one; a disk-down request control module, configured to determine the type of the target data table, and predict the predicted data volume written into the at least one memory space within N consecutive time windows when the type indicates that the target data table is a first-type data table; and generate at least one disk-down request based on the predicted data volume, where N is a positive integer greater than or equal to 1; a collaborative scheduling module, configured to dynamically allocate the at least one disk-down request to the at least one service node corresponding to each time window based on the load parameters of at least one service node, so as to respectively disk down the data in the at least one memory space to at least one hard disk corresponding to the at least one service node.

[0007] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute the steps of any of the above data disk-down methods through the computer program.

[0008] According to yet another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above data disk-down methods when running.

[0009] According to yet another aspect of the embodiments of the present application, there is provided a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of any of the above data disk-down methods.

[0010] Through the above embodiments provided by this application, based on the type judgment of the data table and the estimation of the data write volume in at least one memory space, the triggering time of the data disk drop request for the first type of data table is dynamically predicted, and at least one disk drop request is evenly distributed to at least one service node corresponding to each time window according to the real-time load of each service node. In other words, by predicting the triggering time of the disk drop request within each time window and the real-time load of the service node, it is ensured that the disk drop requests are sequentially allocated to each service node according to the load level, achieving global load balancing, avoiding further congestion of high-load service nodes, improving the data write performance and stability of the HBase database, and achieving the technical effect of improving the data processing ability of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of this application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0012] Figure 1 FIG. is a schematic diagram of an application scenario of a data disk drop method according to an embodiment of this application.

[0013] Figure 2 FIG. is a flowchart of an optional data disk drop method according to an embodiment of this application.

[0014] Figure 3 FIG. is an overall schematic diagram of an optional data disk drop method according to an embodiment of this application.

[0015] Figure 4 FIG. is a schematic diagram of an optional time window and all time periods according to an embodiment of this application.

[0016] Figure 5 FIG. is a schematic diagram of an optional triggering of a Flush request according to an embodiment of this application.

[0017] Figure 6 FIG. is a schematic diagram of an optional mapping relationship between virtual nodes and physical nodes according to an embodiment of this application.

[0018] Figure 7 FIG. is a schematic diagram of an optional generation of a Flush request slot based on a combination of a virtual node slot and a time window according to an embodiment of this application.

[0019] Figure 8 FIG. is a block diagram of the structure of an optional data disk drop system according to an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

[0021] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0022] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0023] According to one aspect of the embodiments of the present application, a data disk dropping method is provided. Optionally, in this embodiment, the above data disk dropping method can be but is not limited to being applied to a hardware scenario as Figure 1 shown, where the server device may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processors 102 may include, but are not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above server device may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above server device. For example, the server device may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0024] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data disk dropping method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above-mentioned method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the server device through a network. Examples of the above-mentioned network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the server device. In one instance, the transmission device 106 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] The embodiments of the present application can be, but are not limited to, applied to various application scenarios of HBase in the field of big data processing and storage, especially in an environment containing a large number of low-frequency write tables. For the sake of easy understanding, specific examples of several application scenarios are given below.

[0027] (1) Data analysis and storage: For example, during the process of storing Internet of Things data, a large amount of data will be generated during the operation of Internet of Things devices, including data statistical tables in multiple dimensions such as device status, device parameters, and environmental monitoring. These tables include data that is not updated frequently but accumulates continuously, so a write strategy that needs to be continuously optimized is required to improve storage efficiency.

[0028] (2) Finance and statistics: Although real-time transaction data is written frequently, the update frequency of statistical information such as transaction history records is low. The technical solution of the present application can optimize the write management of this type of data and avoid a sharp increase in the latency of the real-time transaction system.

[0029] (3) E-commerce and advertising platforms: For example, by analyzing users' historical purchase behaviors, browsing records, etc., the generated data statistical tables have a lower write frequency compared to users' real-time behavior tables. The technical solution of the present application can optimize the writing of such data tables and improve the real-time performance and accuracy of data analysis.

[0030] (4) Social media and content management: For example, statistical information such as the number of reposts and likes of user posts has a relatively low update frequency. The optimized data storage strategy in the technical solution of this application can be used to ensure data consistency and integrity.

[0031] By applying the technical solution of this application to the above scenarios, the performance bottleneck problem of the HBase cluster caused by the existence of a large number of low-frequency write tables is solved. By optimizing the trigger mechanism of data disk write requests, the pressure on the system caused by a large number of Flush requests in a short period of time is dispersed, and the disk I / O storm and latency are reduced, thereby improving data write performance and system stability.

[0032] The data disk write method in the embodiments of this application can be executed by a server device, or can be executed by a server device in combination with at least one of terminal devices (which can also be understood as input / output device 108). Among them, the execution of the data disk write method in the embodiments of this application by a terminal device can also be executed by a client installed thereon.

[0033] Taking the execution of the data disk write method in this embodiment by a server as an example, Figure 2 is a schematic flowchart of an optional data disk write method according to an embodiment of this application, as Figure 2 shown, the process of this method can include steps S202 to step S210.

[0034] Step S202, in response to at least one data write request for writing target data into a target data table, determine the type of the target data table, where the target data table is divided into at least one table unit, and at least one table unit corresponds to at least one memory space one by one.

[0035] Step S204, in the case where the type indicates that the target data table is a first type of data table, predict the amount of predicted data written into at least one memory space within N consecutive time windows, where N is a positive integer greater than or equal to 1.

[0036] Step S206, in the case where the predicted data volume meets the disk write request trigger condition, generate at least one disk write request.

[0037] Step S208, based on the load parameters of at least one service node, dynamically allocate at least one disk write request to at least one service node corresponding to each time window.

[0038] Step S210, respectively write the data in at least one memory space to at least one hard disk corresponding to at least one service node.

[0039] Before explaining the technical solution of this application, first briefly introduce the meanings of professional terms or nouns involved.

[0040] HBase: It is a distributed, column-oriented open-source database that can store massive amounts of data. The underlying data files are stored in HDFS (Hadoop Distributed File System). Through horizontal splitting, the data tables in HBase can be divided into multiple Regions (table units), with each Region sized approximately from 100MB to 1GB to achieve data load balancing and efficient management.

[0041] Region: For HBase, a table is evenly divided into multiple Regions horizontally. Each Region is the basic unit of data management, evenly distributed across all RS nodes in the cluster to achieve load balancing for read and write requests and is the smallest unit of cluster load balancing. Moreover, each Region maintains a stareRowKey and an endRowKey. If the RowKey (row key value) of a piece of data is within the range of the RowKey maintained by a certain Region, then this Region is responsible for managing and maintaining this piece of data.

[0042] Region Server (service node, abbreviated as RS node): Each Region Server in the HBase cluster is responsible for providing read and write services. They evenly distribute Regions from different tables to disperse the load of the cluster and achieve a fast response to read and write requests.

[0043] Memstore: When data is written into the data table of HBase, it is first written into the memory space (Memstore) corresponding to each Region. When the data volume in the Memstore reaches a certain threshold or after a certain period of time, it will trigger the Flush disk operation, that is, write the data from the memory space corresponding to each Region to the hard disk. Among them, each Region corresponds to an RS node.

[0044] Cold table and hot table: In HBase, data tables are divided into cold tables and hot tables according to different writing frequencies. A cold table refers to a table with a low writing frequency (which can also be understood as the first type of data table), while a hot table (which can also be understood as the second type of data table) is the opposite, with a high writing frequency.

[0045] As a commonly used distributed database in the field of big data, HBase creates business tables, index tables, statistical tables, etc. according to business needs. Among them, a large amount of data is written into the business table, while the statistical table may only write business statistical data at a low frequency. Therefore, according to the native writing logic of HBase, the business table can better complete data writing and Flush to disk. However, for the statistical table with low-frequency data writing, the amount of data in its Memstore is small and does not meet the disk-flushing condition (for example, the amount of data reaches 128MB to trigger the Flush disk-flushing mechanism). It can only rely on the regular check of the HBase service once an hour to flush the data to disk. However, when there are a large number of statistical tables with low writing frequencies, it causes a large number of Flush requests to be triggered centrally during the system's regular check, resulting in a large increase in a large number of Flush requests in a short period of time, thus causing a large increase in the time delay of the system's read and write requests.

[0046] In view of the above problems, a data disk-flushing method is proposed in the technical solution of this application. Its overall implementation process is as Figure 3 shown, including a system monitoring module, a Flush request module, a Flush collaborative scheduling module, etc.

[0047] Among them, the system monitoring module obtains various monitoring index data from the client and the server in real time. For example, client indicators include but are not limited to the number of write requests for data tables, and server indicators include but are not limited to the number of task requests of all RS services, RS request latency, and the amount of data in the Memstore and other indicators.

[0048] It should be noted that the data write request can be but is not limited to a request to write target data into the Memstore memory space corresponding to a Region. This request includes the table name of the target data table to which the target data is to be written. When a data write request is received, the actual amount of data in at least one Memstore memory space will increase. The increased amount of data includes the total amount of at least one piece of data to be written in the current data write request. For example, 50MB, 10MB, etc.

[0049] The disk-flushing request (which can also be understood as a Flush request) can be but is not limited to an operation process of flushing the data in at least one Memstore memory space to the hard disk corresponding to the service node when the Flush request trigger condition is met.

[0050] Obviously, one Region of a data table corresponds to one memory space, one Region corresponds to one RS node, one Flush request corresponds to one Region, and one service node can simultaneously maintain at least one Region of multiple data tables.

[0051] As Figure 3As shown, after monitoring the table write request (which can also be understood as a data write request) sent by the client, first determine whether the target data table is a first-type data table or a second-type data table. If it is a first-type data table, according to the dynamic Flush trigger mechanism, predict the predicted data volume to be written into at least one memory space within consecutive N time windows, and generate Flush requests within each time window when the Flush request trigger condition is met. Finally, according to the RS load, allocate the generated at least one Flush request to each RS node to flush the data in each memory space on each service node to the corresponding hard disk.

[0052] On the contrary, if the target data table is a second-type data table, then the original Flush request trigger mechanism will be followed. When the data in a memory space reaches a preset threshold (for example, 100MB), a Flush request is triggered, and at the same time, the data in the relevant memory space is flushed to the corresponding hard disk.

[0053] Different from the traditional HBase where a Flush mechanism is triggered at a preset interval, in the embodiments of this application, through a dynamic prediction mechanism, the time point for triggering a Flush request can be more accurately controlled, avoiding unnecessary resource waste and performance bottlenecks.

[0054] In addition, according to the real-time load parameters of each RS node, the generated Flush requests are dynamically allocated to the service nodes corresponding to each time window. This means that the system will consider indicators such as the current task volume, request latency, and Memstore data volume of each RS node to ensure that Flush requests can be preferentially executed on RSs with lower loads, avoiding a decline in system performance caused by concentrated Flush requests. At the same time, it avoids putting pressure on the same node due to a large number of requests at the same moment, realizing the smooth progress of data disk writing.

[0055] Through the above method, based on the type judgment of the data table and the estimation of the data write volume in at least one memory space, the trigger timing of the data disk write request for the first-type data table is dynamically predicted, and at least one disk write request is evenly allocated to at least one service node corresponding to each time window according to the real-time load of each service node. In other words, by predicting the trigger timing of the disk write request within each time window and the real-time load of the service node, it is ensured that the disk write requests are sequentially allocated to each service node according to the load level, achieving global load balancing, avoiding further congestion of high-load service nodes, improving the data write performance and stability of the HBase database, and achieving the technical effect of improving the data processing ability of the database.

[0056] In an exemplary embodiment, determining the type of the target data table in response to at least one data write request for writing target data into the target data table includes: obtaining the number of data write requests for the target data table within a current unit time; obtaining the total number of data write requests for at least one data table in the database cluster within a historical period; determining the average number of write requests for at least one data table within a historical unit time based on the total number; and determining that the type of the target data table is the first type of data table when the ratio between the number of data write requests and the average number of write requests is less than a determination coefficient.

[0057] In this embodiment, regarding the type of the target data table, it can be determined by, but not limited to, the following formula (1).

[0058] (1)

[0059] Among them, the number of write requests in the past 24 hours is only an example of the current unit time and is not limited thereto.

[0060] The number of write requests in the past 24 hours in the above formula (1) refers to the total number of requests for writing into the target data table in the just-passed 24 hours. When a data write request sent from the data end is obtained, the table name carried by the request has determined the data table in HBase to which the target data will be written according to the current data write request, that is, the data table to be written currently is determined according to the data write request.

[0061] All tables' historical average daily write request times in the above formula (1) can be, but not limited to, the total number of data write requests for all data tables in the HBase cluster within a historical period. Then, based on the total number and the number of data tables, the historical average daily write request times for all tables are determined, thereby determining the determination coefficient.

[0062] Through the above method, the system can identify which data tables belong to the first type of data tables and which data tables belong to the second type of data tables, laying a foundation for the implementation of subsequent dynamic Flush control strategies.

[0063] The method of introducing a determination coefficient to quantitatively determine the type of the target data table not only improves the accuracy of the table type determination result, but also provides a reliable data basis for subsequent prediction mechanisms, improving the prediction accuracy of Flush request triggering in practice.

[0064] Meanwhile, after determining that the data tables are cold tables (the first type of data tables), the system can adopt different management strategies for these tables, such as a dynamic Flush trigger mechanism, which avoids unnecessary consumption of resources. At the same time, by dynamically allocating Flush requests according to the load of RS nodes, resource utilization is further optimized, and the overall performance is prevented from being affected by overloading of some nodes.

[0065] In an exemplary embodiment, when the type indicates that the target data table is a first type of data table, predicting the amount of predicted data written into at least one memory space within N consecutive time windows includes: sequentially obtaining each data write request from at least one data write request as the current data write request; determining the current table cell corresponding to the current data write request and the current memory space corresponding to the current table cell; using the Poisson distribution to predict the current predicted data amount written into the current memory space within N consecutive time windows, where the predicted data amount includes the current predicted data amount.

[0066] In this embodiment, taking a Memstore corresponding to one Region as an example, the specific implementation process of predicting the data amount of one memory space is described.

[0067] This dynamic prediction mechanism is based on indicators such as historical writes and Memstore data volume to predict the Flush trigger mechanism. When the total data volume S of a Memstore reaches 100MB (128×0.8, 80% of the HDFS file block size), it is determined to trigger a Flush request, where the above S can be determined by, but not limited to, the following formula (2).

[0068] (2)

[0069] Among them, when calculating the data volume of a Memstore according to formula (2), the determination priorities decrease sequentially from top to bottom, that is, the Poisson distribution is preferentially used to predict the Flush trigger timing, followed by the exponential smoothing method to predict the Flush trigger timing, and then the Flush request is triggered according to the original Flush mechanism.

[0070] The following formula (3) after transformation using the Poisson distribution is used to predict the total amount of predicted data written into a Memstore within all time periods.

[0071] (3)

[0072] Among them, Represents a time window. For example, a 5-minute window. The Poisson distribution is often used to handle cases where the write coefficient is random and has no significant periodicity. Based on the Poisson distribution, the frequency of writing data k times within the current time window can be predicted. Summing these frequencies gives the expected value of the write requests within the current time window. Summing over all time windows gives the total number of write requests. The formula for the Poisson distribution is as shown in formula (4) below.

[0073] (4)

[0074] Where P is the probability of k within the time window, i.e., the probability of writing k times within that time window; k is a non-negative integer (e.g., k = 0, 1, 2...), and k represents the number of times the write request occurs; λ is a positive real number (λ > 0), representing the average number of write request events occurring within a unit time window, e is the base of the natural logarithm (approximately 2.71828), e is a constant, and k! represents the factorial of k, which means .

[0075] It should be noted that in the above formula (3), when predicting the total predicted data write volume of a Memstore over all time periods, all time periods can be, but are not limited to, starting from the time of the last triggered Flush request, and including the time within all time windows from the starting time to the current time.

[0076] For example, assume the current time is the 3rd time window 3. Then all the above time periods represent the duration from 1 to 3.

[0077] In this embodiment, taking the predicted data write volume of a Memstore as an example, it describes predicting the total data volume written to this memory space over all time periods using the Poisson distribution method. Obviously, the predicted data volume of the above at least one memory space includes the sum of the predicted data write volumes of each memory space.

[0078] The Poisson distribution is a statistical method applicable to describing the probability distribution of the number of events occurring within a fixed time, especially effective when the event occurrence frequency is low and independent. Here, the Poisson distribution is used to predict the current predicted data volume written to the Memstore over N consecutive time windows.

[0079] The prediction model of the Poisson distribution is based on historical write data. It can provide the expected value of the data write volume, thereby helping the system anticipate when the Flush threshold is reached, such as 80% of 128MB. The effectiveness of this prediction model lies in its ability to predict future trends based on past write patterns, especially for those data tables with relatively stable historical write frequencies.

[0080] By utilizing the Poisson distribution, the system can more accurately predict the amount of data written to the Memstore within N consecutive time windows based on historical data, thereby predicting in advance when the cold table is approaching the Flush threshold, avoiding the problem of delayed Flush caused by sudden increase in data volume, and enhancing the accuracy of the system's prediction of the data write volume.

[0081] This prediction mechanism helps to manage the data volume in the Memstore more precisely. Through dynamic monitoring and prediction, the system can adjust the Flush strategy in a timely manner to ensure that the data in the cold table is flushed to disk at an appropriate time point, avoiding excessive occupation of memory space, and thus optimizing the usage efficiency of memory resources.

[0082] In an exemplary embodiment, the above method further includes: obtaining the current actual data volume actually written to the current memory space after responding to the current data write request; when the difference between the current predicted data volume and the current actual data volume is greater than or equal to the first threshold, predicting the current predicted data volume written to the current memory space within N consecutive time windows by using the exponential smoothing method.

[0083] When it is detected that the business write requests are continuously written and the deviation between the predicted value of the Poisson distribution and the actual data write volume of the Memstore is large (for example, the difference is greater than 2M), switch to the exponential smoothing method to predict the total data write volume within all time periods, which is specifically implemented through the following formula (5).

[0084] (5)

[0085] Wherein, represents the predicted data volume written to the memory space in the previous time window, is the smoothing factor (0 < < 1, usually taking 0.1 to 0.3), represents the data volume actually written to the memory space in the current time window.

[0086] Assume that taking the 3 time windows shown in Figure 4 as all time periods as an example, for a Memstore corresponding to a Region, the data volume that may be written to this Memstore space predicted by using the Poisson distribution is a total of 50MB, while the space occupied by the data actually written to this Memstore space within these 3 time windows is 55MB. Then the difference between the two is greater than the first threshold of 2MB, and switch to using the exponential smoothing method to predict the total data write volume of each Memstore within all time periods.

[0087] Obviously, over time, all of the above time periods are dynamically changing. For example, from an initial time window to the first two time windows, the first three time windows, and so on.

[0088] The exponential smoothing method is applicable to tables where data is continuously written and has relatively small fluctuations, and it has good short-term prediction effects. It mainly predicts future values by assigning exponentially decreasing weights to historical data. Its core assumption is that "recent data is more valuable for prediction than long-term data".

[0089] Through the switching scheme between the exponential smoothing method and the Poisson distribution method provided in this embodiment, the entire dynamic Flush control framework becomes more flexible. It ensures that the prediction model can be dynamically adjusted according to the actual situation to adapt to changes in the data writing mode. When the writing frequency of a cold table suddenly increases or the data volume fluctuates greatly, the Poisson distribution prediction method may not be able to accurately reflect the current situation in a timely manner. At this time, it will switch to the exponential smoothing method. By giving higher weights to recent data, it can quickly capture the data change trend and ensure the timeliness and accuracy of the prediction results.

[0090] By introducing the exponential smoothing method, the system can more accurately predict the data volume change trend, so as to trigger the Flush operation more timely and accurately when the data volume is close to the Flush trigger threshold, avoiding delay problems caused by sudden increases in the data volume.

[0091] In an exemplary embodiment, the above prediction of the current predicted data volume written into the current memory space within N consecutive time windows using the exponential smoothing method includes: predicting the (i + 1)-th data volume written into the current memory space within the (i + 1)-th time window through the following steps, where i is a positive integer greater than or equal to 2 and less than or equal to N: obtaining the (i - 1)-th data volume predicted to be written into the current memory space within the (i - 1)-th time window; obtaining the i-th actual data volume actually written into the current memory space within the i-th time window; determining the (i + 1)-th data volume based on the smoothing factor, the (i - 1)-th data volume, and the i-th actual data volume, where the current predicted data volume includes the (i + 1)-th data volume.

[0092] In this embodiment, taking the calculation of the predicted data write volume in the next time window as an example, the calculation process of the exponential smoothing method is described, as shown in the following formula (6).

[0093] (6)

[0094] Where is the predicted value based on the next time window, that is, the predicted data volume written into a memory space within the next time window of the time window where the current moment is located (which can also be understood as the predicted value), Vt is the actual observed value (which can also be understood as the actual value) at the current moment (which can also be understood as the current time window). is the smoothing factor (0 < < 1, usually taking values from 0.1 to 0.3).

[0095] Through the exponential smoothing method in the above formula (6), based on the smoothing factor, the predicted data volume of the previous time window, and the actual data volume of the current time window, the predicted data volume of the next time window is determined. Among them, the selection of the smoothing factor is crucial, which determines the influence degree of historical data and current data on the prediction result. Usually, it is between 0.1 and 0.3 to ensure that the prediction result can quickly respond to the change of data trend and is not overly affected by short-term fluctuations.

[0096] According to the recursive characteristic of the exponential smoothing method, its core lies in using the smoothing factor to perform weighted averaging on historical data and new data. This calculation method can effectively reduce the influence of short-term fluctuations, improve the stability and accuracy of the prediction result, make the predicted i-th data volume closer to the true value, and provide a solid data basis for the decision-making of the Flush operation.

[0097] In addition, through the above switching between the Poisson distribution and the exponential smoothing method, the data volume can be accurately predicted, which helps the system plan the Flush operation in advance, avoids the resource scheduling chaos caused by the sudden change of data volume, not only improves the accuracy and reliability of the prediction result, but also enhances the dynamic adaptability and robustness of the prediction model, and has significant technical effects and practical application value for optimizing the Flush operation timing of cold tables and improving the performance and stability of the HBase cluster when processing low-frequency write data tables.

[0098] In an exemplary embodiment, the above-mentioned generating at least one disk write request when the predicted data volume meets the trigger condition for the disk write request includes: generating at least one disk write request when the predicted data volume is greater than or equal to the second threshold, where the second threshold is determined based on the upper limit value of a memory space among at least one memory space.

[0099] This embodiment is the execution layer in the dynamic Flush control framework. According to the predicted data volume in at least one memory space and the preset second threshold in the above embodiment, the trigger timing of the disk write request is determined, that is, when the data volume in a Memstore reaches the second threshold, a Flush request is triggered and generated.

[0100] For example, as Figure 5 shown, assuming that according to the above Poisson distribution or exponential smoothing method, it is predicted respectively for the time window 1 to If the amounts of data written into a memory space in two time windows are 50MB and 55MB respectively, and since their sum of 105 exceeds the second threshold of 100MB, a Flush request will be triggered in the third time window. 3 to trigger the generation of a Flush request.

[0101] It should be noted that after the Flush request is generated, the RS node is determined according to the Region corresponding to the Flush request, and the Flush request is sent to the Flush request queue of this RS node maintained by the Flush collaborative scheduling module.

[0102] That is to say, within a time window, multiple Flush requests are allowed to be triggered in parallel simultaneously. However, instead of directly sending multiple Flush requests to the RS node, they are temporarily sent to the Flush request queue.

[0103] This embodiment can achieve refined management of the Memstore data volume. The prediction mechanism dynamically predicts the data volume within consecutive N time windows. When the predicted value approaches the second threshold, the system can prepare the Flush operation in advance, avoiding the data volume exceeding the Memstore upper limit, thereby protecting the memory resources and reducing system exceptions caused by memory overflow.

[0104] The second threshold is set based on 80% of the Memstore space upper limit value. This strategy ensures that the Flush is triggered when the data volume reaches a reasonable threshold, avoiding both the problem of delayed Flush caused by sudden data volume growth and resource waste caused by frequent Flush due to small data volume, and achieving the optimal timing selection for the Flush operation.

[0105] Combined with the description in the above embodiments, whether it is to predict the data write volume of each memory space by dynamically switching between the Poisson distribution and the exponential smoothing method, or to dynamically predict multiple Flush requests allowed to be triggered in parallel within each time window by judging the comparison between the predicted data volume and the second threshold, both reflect the flexibility and rationality of dynamically predicting the trigger timing of Flush.

[0106] In an exemplary embodiment, the above method further includes: within the i-th time window among consecutive N time windows, predicting the predicted sub-data volume of each memory space in at least one memory space to obtain a set of predicted sub-data volumes; based on the set of predicted sub-data volumes, determining a set of cumulative predicted data volumes of each memory space within consecutive i time windows; in the case where at least some of the cumulative predicted data volumes in the set of cumulative predicted data volumes are greater than the third threshold, simultaneously triggering a set of disk write requests corresponding to at least some of the memory spaces in at least one memory space within the i-th time window.

[0107] For example, assume that there are at least three data write requests. Determine the Region1, Region2, and Region3 corresponding to the three data write requests respectively. Then, within the second time window, predict the predicted sub-data volume of each memory space in Region1 to Region3 in parallel, and at the same time, judge the total amount of predicted data written in each Region within the first two time windows.

[0108] Judge the magnitude relationship between the total amount of predicted data written in each Region over all time periods (in this embodiment, referring to the first two time windows), and when the total amount of predicted data written is greater than or equal to the second threshold, determine the number of Flush requests triggered within each time window.

[0109] For example, assume that the total amount of predicted data written in Region1 and Region2 within the first two time windows is greater than the second threshold. Then, within the third time window, trigger the Flush requests corresponding to Region1 and Region2 in parallel.

[0110] By predicting the data volume of all memory spaces in the HBase cluster in this embodiment, the system can timely respond to the growth of the data volume of each memory space, and can globally identify and handle the situation where multiple memory spaces simultaneously approach or exceed their space limits, effectively avoiding the performance bottlenecks and stability problems that may be caused by local overload.

[0111] Through the prediction of the sub-data volume in N consecutive time windows and subsequent cumulative prediction, this embodiment can provide a more comprehensive and in-depth analysis of the data volume growth trend, help the system accurately predict the usage of cluster resources, and provide a solid data basis for intelligent decision-making.

[0112] That is to say, by introducing global trend prediction and optimized management of cluster resources, not only the prediction accuracy of the Memstore data volume is improved, but also the intelligent allocation and management of the entire HBase cluster resources are realized, which has significant technical effects on improving system stability, response speed, and overall performance. This not only demonstrates the optimization potential of resource management strategies in the field of big data processing, but also reflects the intelligent adjustment ability of the system in dealing with dynamic business scenarios and resource allocation challenges, and has far-reaching significance for promoting the application of HBase clusters in the field of real-time data processing and storage.

[0113] In an exemplary embodiment, dynamically allocating at least one disk dropping request to at least one service node corresponding to each time window based on the load parameters of at least one service node includes: distributing a disk dropping request to at least one disk dropping request queue of at least one service node respectively according to the mapping relationship between at least one service node and at least one table unit; determining the load parameters of each service node within each time window based on the performance metrics of at least one service node, where the performance metrics include the number of service node tasks and the service node request latency; dynamically adjusting the number of disk dropping request slots of each service node within each time window based on the load parameters, where the disk dropping request slots are used to carry the disk dropping requests in at least one disk dropping request queue; and dynamically allocating at least one disk dropping request to at least one service node corresponding to each time window based on the number of disk dropping request slots.

[0114] When it is determined through the technical solution in the above embodiment that the Flush request trigger condition is met, a set of Flush requests (which can also be understood as a set of disk dropping requests) corresponding to at least one memory space is triggered to be generated. At this time, the set of Flush requests will not be immediately distributed to the Flush request slots of each RS node, but the Flush requests will be first sent to the Flush request queues of the corresponding RS nodes.

[0115] Among them, the RS node to which the memory space belongs is determined according to the Region corresponding to each Flush request, and each RS node maintains a Flush request queue, where the number of Flush requests allowed to be carried in the request queue is preset.

[0116] Generate the load parameters of each service node within each time window according to the performance metrics of each service node, where the performance metrics include but are not limited to Figure 3 the number of RS request tasks and RS request latency shown, etc. In addition, it also includes metrics such as network status, disk I / O, and central processing unit (CPU). When calculating the load parameters of the service node, at least 2 types can be randomly selected from the above multiple performance metrics for the load calculation process, and the embodiments of the present application do not limit this.

[0117] The reason for calculating the load parameters of each service node is that the lower the load of a service node, the more Flush requests are allowed to be allocated to this service node; conversely, the higher the load of a service node, the fewer Flush requests are allowed to be allocated to this service node.

[0118] By calculating the load of each service node within each time window, accurately controlling the number of Flush requests that can be received on the RS nodes with different loads of each service node in different time windows, global load balancing is achieved, ensuring the full utilization of resources.

[0119] Secondly, inside the Flush collaborative scheduling module, combining the load parameters of each service node and the Flush requests in the Flush request queue maintained by each service node, according to the first-in, first-out rule, distribute each Flush request in the Flush request queue to the Flush request slots of the corresponding service node in different time windows.

[0120] It should be noted that within each time window, the number of Flush request slots changes according to the load of the service node. The purpose is to ensure that different numbers of Flush requests are allocated according to the RS node load, achieving global load balancing.

[0121] By introducing the Flush request queue and Flush request slots (which can also be understood as disk write request slots) of the RS node, the dynamic Flush control framework in the embodiment of the present application is further improved. And by dynamically allocating Flush requests and the number of Flush request slots, precise management of the RS node load and global resource optimization are achieved. In other words, this embodiment not only considers data prediction and the generation of disk write requests, but also delves into the distribution of requests and node load monitoring, forming a complete closed-loop from data prediction to resource management, significantly improving the performance and stability of the HBase cluster when processing a large number of low-frequency write tables.

[0122] In an exemplary embodiment, the above-mentioned dynamically adjusting the number of disk write request slots of each service node in each time window based on load parameters includes: dynamically generating a virtual node adjustment coefficient for each service node in each time window based on the number of service node tasks, service node request latency, and load balancing factor, where the larger the value of the virtual node adjustment coefficient, the lower the load of each service node; determining the number of virtual nodes of each service node in each time window based on the virtual node adjustment coefficient; generating a preset number of virtual node slots for each service node in each time window based on the number of virtual nodes; and dynamically adjusting the number of disk write request slots of each service node in each time window based on the virtual node slots and consecutive N time windows.

[0123] In this embodiment, based on the statistical data of various RS indicators, the RS load is calculated, and the number of virtual nodes is dynamically adjusted according to the calculated RS load. Among them, the number of RS virtual nodes with a higher load is smaller, and the number of RS virtual nodes with a lower load is larger. The purpose is to enable the service nodes with a lower load to receive more Flush requests within the current time window, achieve global load balancing, and ensure that resources can be fully utilized.

[0124] To describe the load size of the RS node, a load balancing factor L is designed in the embodiment of the present application. L can be determined by the following formula (7) but is not limited thereto.

[0125] L = (7)

[0126] Wherein, W represents the average number of tasks to be processed in the system, Y represents the average system delay, w represents the number of tasks to be processed by the current node, and y represents the average delay of the current node. The virtual node coefficient L is adjusted according to the current load situation of the service node. The number of virtual nodes corresponding to a physical RS node is usually between 100 and 1000.

[0127] When L is a positive number, it means that the number of tasks of the current node is less than the average value, that is, the load is lower, and the number of virtual nodes can be increased; when L is a negative number, the number of tasks of the current node is greater than the average value, that is, the load is larger, and the number of virtual nodes can be reduced. In other words, the larger L is, the lower the load of the RS node and the more RS virtual nodes.

[0128] Among them, the number of virtual nodes can be calculated by the following formula (8) but is not limited thereto.

[0129] (8)

[0130] Wherein, represents the load balancing factor of the i-th service node within the current time window, represents the minimum value of the load balancing factor, represents the maximum value of the load balancing factor.

[0131] In this embodiment, a design method of RS virtual node slots is also introduced. Specifically, based on the consideration of different loads of RS nodes and Flush request load balancing, multiple RS virtual node slots with unique identifiers are generated for each physical node (i.e., RS node). The mapping relationship between each virtual node and the physical node is as Figure 6 shown, wherein the virtual node ID = physical node ID + delimiter + serial number. For example, regionserver-5#0001, regionserver-5#0002, regionserver-5#0010.

[0132] Dynamically adjust each service node to allow it to carry within each time window according to the combination of RS virtual node slots and time windows, and evenly distribute the Flush requests in each Flush request queue to each RS node.

[0133] By adopting the above method, through the dynamic generation of the virtual node adjustment coefficient and the flexible adjustment of the number of virtual node slots, the fine-grained balance of the RS node load and the efficient utilization of resources are realized. This solution not only considers data prediction and the generation of disk write requests, but further optimizes the distribution of Flush requests and the load management of nodes, improving the performance and stability of the HBase cluster in processing data writes.

[0134] In an exemplary embodiment, the dynamically allocating at least one disk write request to at least one service node corresponding to each time window based on the number of disk write request slots includes: obtaining the i-th group of virtual node slots corresponding to the i-th service node within the i-th time window, where i is a positive integer greater than or equal to 2 and less than or equal to N; determining the i-th group of disk write request slots of the i-th service node within the i-th time window based on the i-th group of virtual node slots and the i-th time window; and allocating the i-th group of disk write requests in the i-th disk write request queue of the i-th service node to the i-th group of disk write request slots, where the number of the i-th group of disk write request slots is greater than or equal to the number of disk write requests in the i-th group of disk write requests, and the number of the i-th group of disk write request slots is determined based on the load upper limit of the i-th service node.

[0135] As Figure 7 shown, in the embodiment of the present application, the Flush requests pre-stored in the Flush request queue are distributed to the disk write request slots (which can also be understood as Flush request slots) corresponding to each RS node in the order of first in first out by combining RS virtual node slots and time partition slots (which can also be understood as time windows).

[0136] It should be noted that a RS virtual node slot and a time window can be combined to generate a Flush request slot for carrying the Flush requests in the Flush request queue. Within the time granularity of each time partition slot, recalculate the loads of all service nodes, generate different numbers of virtual nodes according to the calculated loads, and one virtual node corresponds to one virtual node slot.

[0137] According to the generated different numbers of virtual nodes, determine the corresponding number of virtual node slots, and then according to the virtual node slots and as Figure 7The combination of the time partition slots shown generates the Flush request slots for the current service node within the current time window, and distributes the Flush requests to the corresponding Flush request slots in the order of the Flush requests in the Flush request queue sent to the current service node.

[0138] In addition, when the RS node performs dynamic scaling, it will trigger the migration of Regions (from other nodes to the scaled RS node). During expansion, the Regions deployed on other nodes will be migrated into the current RS node, and during contraction, the Regions will be migrated out of the current RS node.

[0139] The above Flush request slots based on the combination of RS virtual node slots and time windows can achieve load balancing and overload current limiting. According to the RS load situation, the number of Flush requests that can be carried within this time partition slot is restricted, so as to ensure that the requests in the Flush request queue corresponding to this RS are consumed according to its capacity.

[0140] When the Flush request slots cannot consume all the Flush requests generated by the current time partition slot, they will be carried forward to the next time partition slot for continued consumption. When there are fewer Flush requests generated by the current time partition slot, the Flush request slots generated by the RS may be idle.

[0141] By adopting the above method, the Flush requests that are centrally triggered at preset intervals in the prior art can be dispersed to different RS nodes in different time windows, avoiding the situation where the RS queue accumulates due to a large number of Flush requests, resulting in a sharp increase in the time delay of normal read and write requests, and at the same time reducing the pressure on the disk load caused by the centralized Flush and centralized disk writing.

[0142] Through the dynamic scheduling mechanism in the embodiments of the present application, the system can effectively cope with the fluctuations of business requirements and the changes of node loads, ensuring the reasonable allocation and high-efficiency utilization of cluster resources, not only improving the stability of the system when processing a large number of low-frequency write tables, but also enhancing the scalability of the cluster, providing a solid technical foundation for future business growth and system expansion.

[0143] In addition, by dynamically determining the number of disk write request slots, not only the monitoring and management accuracy of the request processing capacity of the service node is improved, but also the reasonable allocation of requests and the high-efficiency utilization of resources are realized, improving the performance and stability of the HBase cluster when processing low-frequency write tables, and enhancing the system's ability to cope with complex business scenarios.

[0144] In an exemplary embodiment, the above method further includes: when at least one disk dropping request includes a first partial disk dropping request generated from a first type of data table and a second partial disk dropping request generated from a second type of data table, the priority of responding to the second partial disk dropping request is higher than that of the first partial disk dropping request; or when at least one disk dropping request only includes a third partial disk dropping request generated from a first type of data table, each disk dropping request is sequentially responded to according to the chronological order of the generation time of each disk dropping request in the third partial disk dropping request; or when at least one disk dropping request only includes a fourth partial disk dropping request generated from a second type of data table, each disk dropping request is sequentially responded to according to the chronological order of the generation time of each disk dropping request in the fourth partial disk dropping request.

[0145] To enrich the request response strategy of the dynamic Flush control framework in the embodiments of the present application, by distinguishing disk dropping requests generated from a first type of data table (hot table) and a second type of data table (cold table), and setting different response priorities based on the request type and generation time, personalized management of data writing for different characteristic tables is achieved. The core of this strategy lies in intelligently adjusting the response order according to the characteristics and generation time of the requests, ensuring the rapid disk dropping of key data and the orderly processing of non-key data, and improving the overall response speed and resource utilization efficiency of the system.

[0146] Specifically, when the disk dropping request queue contains disk dropping requests generated from both hot tables and cold tables, the requests generated from hot tables, that is, the second partial disk dropping requests, are preferentially responded to. This is because the data writing frequency of hot tables is higher, and the impact on system performance and response speed is more significant. Preferential processing can effectively avoid business delays and improve the overall system performance. When there are only disk dropping requests generated from the first type of data table (hot table) or the second type of data table (cold table), the system responds sequentially according to the chronological order of the generation of Flush requests. This strategy ensures the fairness and orderliness of data processing and avoids data loss or processing chaos.

[0147] By the above method, not only the request response mechanism of the dynamic Flush control framework is improved, but also it demonstrates how to optimize resource management and business response through intelligent algorithms and dynamic strategies in the field of big data processing, and improve the system's ability to handle complex business scenarios.

[0148] By distinguishing the data table types to set the request response priority and the strategy of responding according to the generation time order, not only the processing flow of different types of data tables is optimized, but also the reasonable allocation and utilization efficiency of resources are improved. It has significant technical effects on improving the performance and stability of the HBase cluster when processing low-frequency and high-frequency write tables, not only ensuring the rapid response of key services, but also ensuring the orderly processing of non-key services, improving the overall performance of the system, and enhancing the adaptability and stability of the system.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0150] According to another aspect of the embodiments of the present application, a data disk dropping system is further provided. The structural schematic diagram of the system is as Figure 8 shown, including a monitoring module 802, a disk dropping request control module 804, and a collaborative scheduling module 806.

[0151] The monitoring module 802 is used to obtain at least one data writing request for writing target data into a target data table. Among them, the target data table is divided into at least one table unit, and at least one table unit corresponds to at least one memory space one by one.

[0152] The disk dropping request control module 804 is used to determine the type of the target data table, and when the type indicates that the target data table is a first type of data table, predict the predicted data volume written into at least one memory space within N consecutive time windows; and generate at least one disk dropping request based on the predicted data volume, where N is a positive integer greater than or equal to 1.

[0153] The collaborative scheduling module 806 is used to dynamically allocate at least one disk dropping request to at least one service node corresponding to each time window based on the load parameters of at least one service node, so as to drop the data in at least one memory space to at least one hard disk corresponding to at least one service node respectively.

[0154] As Figure 3 shown, the monitoring module (or the system monitoring module) is responsible for collecting metrics such as the number of data writing requests sent by the client, the number of RS request tasks sent by the server, the RS request latency, and the Memstore data volume, etc., to ensure that the writing operation of the target data table is accurately monitored. These metrics are used for RS load calculation, type determination of the target data table, and evaluation of prediction results, etc.

[0155] Among them, the target data table is divided into multiple table units, and each table unit corresponds to a memory space (Memstore). This mechanism ensures the load balance and efficient processing of data writing.

[0156] The disk drop request control module is a key component of the technical solution of this application. It is mainly used to determine the type of the target data table and implement a dynamic Flush judgment mechanism based on time prediction. The table type is determined based on the Poisson distribution or exponential smoothing method to predict the Flush trigger timing, and the RS node is determined according to the region corresponding to the Flush request. The Flush request is sent to the Flush request queue of the RS node maintained by the Flush collaborative scheduling module.

[0157] Specifically, using the above formula (1), the target data table type is determined, distinguishing the different characteristics of hot and cold tables. For cold tables (type 1 data tables), the module predicts the change in Memstore data volume over N consecutive time windows based on historical data and prediction models. When the predicted data volume reaches the set threshold, a disk flush request is generated. This strategy ensures that cold table data can be processed promptly when it approaches the flush threshold, avoiding the situation where a large number of requests are triggered simultaneously and reducing the risk of disk I / O storms.

[0158] The collaborative scheduling module dynamically generates varying numbers of virtual nodes within each time interval based on RS node monitoring metrics and pre-set time intervals. These virtual nodes then generate Flush request slots to carry flush requests. The module maintains a flush request queue for each RS node and distributes flush requests to the flush request slots in a first-in, first-out order. Within each time window, the number of request slots varies based on load, ensuring that varying numbers of flush requests are distributed according to node load, achieving global load balancing.

[0159] The specific execution steps involved in the various calculation processes in the above modules and the dynamic prediction of Flush timing can be referred to the description in the above embodiments and will not be repeated here.

[0160] This application proposes a dynamic flush control framework for the data storage system, optimizing the flush mechanism of the HBase cluster MemStore. Furthermore, based on the dynamic RS virtual node slots and time-partitioned slots based on RS load, the number of RS virtual node slots is dynamically calculated within each time window. Flush request slots are generated based on their carrying capacity, consuming the corresponding RS flush request queues to ensure overall load balancing across all RSs and timelines.

[0161] Obviously, the above data disk writing system can be used to implement the data disk writing method provided in the above embodiments, and the descriptions that have been made will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0162] It should be noted that the monitoring module 802 in this embodiment can be used to execute the above step S202, the disk writing request control module 804 in this embodiment can be used to execute the above step S204 and step S206, and the collaborative scheduling module 806 in this embodiment can be used to execute the above step S208 and step S210.

[0163] Through the embodiments provided in the present application, based on the type judgment of the data table and the estimation of the data writing amount in at least one memory space, the triggering time of the data disk writing request for the first type of data table is dynamically predicted, and at least one disk writing request is evenly distributed to at least one service node corresponding to each time window according to the real-time load of each service node. In other words, by predicting the triggering time of the disk writing request in each time window and the real-time load of the service node, it is ensured that the disk writing requests are sequentially allocated to each service node according to the load level, achieving global load balancing, avoiding further congestion of high-load service nodes, improving the data writing performance and stability of the HBase database, and achieving the technical effect of improving the data processing ability of the database.

[0164] In an exemplary embodiment, the above disk writing request control module 804 includes: a first acquisition unit, configured to acquire the number of data writing requests for a target data table within a current unit time; a second acquisition unit, configured to acquire the total number of data writing requests for at least one data table in the database cluster within a historical period; a first processing unit, configured to determine the average number of writing requests for at least one data table within a historical unit time based on the total number; and a second processing unit, configured to determine that the type of the target data table is the first type of data table when the ratio between the number of data writing requests and the average number of writing requests is less than a determination coefficient.

[0165] In an exemplary embodiment, the above-mentioned disk dropping request control module 804 includes: a third acquisition unit configured to sequentially acquire each data writing request from at least one data writing request as the current data writing request; a third processing unit configured to determine the current table unit corresponding to the current data writing request and the current memory space corresponding to the current table unit; a first prediction unit configured to predict, using the Poisson distribution, the current predicted data volume written into the current memory space within N consecutive time windows, where the predicted data volume includes the current predicted data volume.

[0166] In an exemplary embodiment, the above-mentioned system further includes: a first acquisition module configured to acquire the current actual data volume actually written into the current memory space after responding to the current data writing request; a first writing module configured to, when the difference between the current predicted data volume and the current actual data volume is greater than or equal to a first threshold, predict, using the exponential smoothing method, the current predicted data volume written into the current memory space within N consecutive time windows.

[0167] In an exemplary embodiment, the above-mentioned first writing module includes: a fourth processing unit configured to predict the i-th data volume written into the current memory space in the i-th time window through the following steps, where i is a positive integer greater than or equal to 2 and less than or equal to N: acquiring the (i - 1)-th data volume predicted to be written into the current memory space in the (i - 1)-th time window; acquiring the i-th actual data volume actually written into the current memory space in the i-th time window; determining the i-th data volume based on the smoothing factor, the (i - 1)-th data volume, and the i-th actual data volume, where the current predicted data volume includes the i-th data volume.

[0168] In an exemplary embodiment, the disk dropping request control module 804 includes: a fifth processing unit configured to generate at least one disk dropping request when the predicted data volume is greater than or equal to a second threshold, where the second threshold is determined based on the space upper limit value of one of at least one memory space.

[0169] In an exemplary embodiment, the above-mentioned system further includes: a first prediction module configured to predict, in the i-th time window among N consecutive time windows, the predicted sub-data volume of each memory space in at least one memory space to obtain a set of predicted sub-data volumes; a first processing module configured to determine, based on the set of predicted sub-data volumes, a set of cumulative predicted data volumes of each memory space within i consecutive time windows; a second processing module configured to, when at least some of the set of cumulative predicted data volumes are greater than a third threshold, simultaneously trigger a set of disk dropping requests corresponding to at least some of the memory spaces in at least one memory space in the i-th time window.

[0170] In an exemplary embodiment, the above-mentioned collaborative scheduling module 806 includes: a sending unit configured to send a disk writing request to at least one disk writing request queue of at least one service node respectively according to the mapping relationship between at least one service node and at least one table unit; a sixth processing unit configured to determine the load parameter of each service node in each time window based on the performance metrics of at least one service node, where the performance metrics include the number of service node tasks and the service node request latency; a first adjustment unit configured to dynamically adjust the number of disk writing request slots of each service node in each time window based on the load parameter, where the disk writing request slots are used to carry the disk writing requests in at least one disk writing request queue; a first allocation unit configured to dynamically allocate at least one disk writing request to at least one service node corresponding to each time window based on the number of disk writing request slots.

[0171] In an exemplary embodiment, the above-mentioned first adjustment unit includes: a first processing subunit configured to dynamically generate a virtual node adjustment coefficient of each service node in each time window based on the number of service node tasks, the service node request latency, and a load balancing factor, where the larger the value of the virtual node adjustment coefficient, the lower the load of each service node; a second processing subunit configured to determine the number of virtual nodes of each service node in each time window based on the virtual node adjustment coefficient; a third processing subunit configured to generate a preset number of virtual node slots for each service node in each time window based on the number of virtual nodes; an adjustment subunit configured to dynamically adjust the number of disk writing request slots of each service node in each time window based on the virtual node slots and consecutive N time windows.

[0172] In an exemplary embodiment, the above-mentioned first allocation unit includes: a first obtaining subunit configured to obtain the i-th group of virtual node slots corresponding to the i-th service node in the i-th time window, where i is a positive integer greater than or equal to 2 and less than or equal to N; a fourth processing subunit configured to determine the i-th group of disk writing request slots of the i-th service node in the i-th time window based on the i-th group of virtual node slots and the i-th time window; a first allocation subunit configured to allocate the i-th group of disk writing requests in the i-th disk writing request queue of the i-th service node to the i-th group of disk writing request slots based on the number of the i-th group of disk writing request slots, where the number of the i-th group of disk writing request slots is greater than or equal to the number of disk writing requests in the i-th group of disk writing requests, and the number of the i-th group of disk writing request slots is determined based on the load upper limit of the i-th service node.

[0173] In an exemplary embodiment, the above data disk dropping system further includes: a third processing module, configured to, when at least one disk dropping request includes a first partial disk dropping request generated from a first type of data table and a second partial disk dropping request generated from a second type of data table, respond to the second partial disk dropping request with a higher priority than the first partial disk dropping request; or when at least one disk dropping request only includes a third partial disk dropping request generated from a first type of data table, respond to each disk dropping request in sequence according to the order of generation time of each disk dropping request in the third partial disk dropping request; or when at least one disk dropping request only includes a fourth partial disk dropping request generated from a second type of data table, respond to each disk dropping request in sequence according to the order of generation time of each disk dropping request in the fourth partial disk dropping request.

[0174] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited thereto: the above modules are all located in the same processor; or, the above-mentioned various modules are respectively located in different processors in any combination form.

[0175] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data disk dropping method embodiments.

[0176] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above-mentioned data disk dropping method embodiments when running.

[0177] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs, etc., various media that can store computer programs.

[0178] According to another aspect of the embodiments of the present application, there is also provided a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-mentioned data disk dropping method embodiments.

[0179] The embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-mentioned data disk dropping method embodiments.

[0180] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0181] The above has introduced in detail a data disk writing method provided by this application. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data disk writing method, characterized in that: It includes: In response to at least one data writing request for writing target data into a target data table, determining the type of the target data table, wherein the target data table is divided into at least one table unit, and the at least one table unit corresponds to at least one memory space one by one; When the type indicates that the target data table is a first type data table, predicting the predicted data volume written into the at least one memory space within N consecutive time windows, wherein each data writing request is sequentially obtained from the at least one data writing request as the current data writing request; determining the current table unit corresponding to the current data writing request and the current memory space corresponding to the current table unit; using the Poisson distribution to predict the frequency of writing data into the current memory space in the current time window; according to the frequency, determining the total number of writing requests within the N time windows; according to the total number of writing requests, predicting the current predicted data volume written into the current memory space within the N consecutive time windows, the predicted data volume includes the current predicted data volume, the N time windows include the current time window, and N is a positive integer greater than or equal to 1; When the predicted data volume meets the disk writing request trigger condition, generating at least one disk writing request; Based on the load parameters of at least one service node, dynamically allocating the at least one disk writing request to the at least one service node corresponding to each time window; Writing the data in the at least one memory space to the at least one hard disk corresponding to the at least one service node respectively.

2. The method according to claim 1, characterized in that: The determining the type of the target data table in response to at least one data writing request for writing target data into the target data table includes: Obtaining the number of data writing requests for the target data table within the current unit time; Obtaining the total number of data writing requests for at least one data table in the database cluster within the historical period; Based on the total number, determining the average number of writing requests for the at least one data table within the historical unit time; When the ratio between the number of data writing requests and the average number of writing requests is less than the determination coefficient, determining the type of the target data table as the first type data table.

3. The method according to claim 1, characterized in that: The method further includes: Obtaining the current actual data volume actually written into the current memory space after responding to the current data writing request; When the difference between the current predicted data volume and the current actual data volume is greater than or equal to the first threshold, using the exponential smoothing method to predict the current predicted data volume written into the current memory space within the N consecutive time windows.

4. The method according to claim 3, characterized in that: The using the exponential smoothing method to predict the current predicted data volume written into the current memory space within the N consecutive time windows includes: Predict the (i + 1)-th data volume written to the current memory space within the (i + 1)-th time window through the following steps, where i is a positive integer greater than or equal to 2 and less than or equal to N: Obtain the (i - 1)-th data volume predicted to be written to the current memory space within the (i - 1)-th time window; Obtain the i-th actual data volume actually written to the current memory space within the i-th time window; Determine the (i + 1)-th data volume based on a smoothing factor, the (i - 1)-th data volume, and the i-th actual data volume, where the current predicted data volume includes the (i + 1)-th data volume.

5. The method according to claim 1, wherein when the predicted data volume satisfies the trigger condition for the disk write request, generating at least one disk write request, including: when the predicted data volume is greater than or equal to a second threshold, generating the at least one disk write request, where the second threshold is determined based on the space upper limit value of one of the at least one memory spaces.

6. The method according to claim 1, wherein The method further includes: within the i-th time window of the consecutive N time windows, predicting the predicted sub-data volume of each of the at least one memory spaces to obtain a set of predicted sub-data volumes; based on the set of predicted sub-data volumes, determining a set of cumulative predicted data volumes of each memory space within the consecutive i time windows; when at least some of the cumulative predicted data volumes in the set of cumulative predicted data volumes are greater than a third threshold, simultaneously triggering a set of disk write requests corresponding to at least some of the at least one memory spaces within the i-th time window.

7. The method according to claim 1, wherein dynamically allocating the at least one disk write request to the at least one service node corresponding to each time window based on the load parameters of the at least one service node, including: in accordance with the mapping relationship between the at least one service node and the at least one table unit, distributing the one disk write request to at least one disk write request queue of the at least one service node respectively; determining the load parameter of each service node within each time window based on the performance metrics of the at least one service node, where the performance metrics include the number of service node tasks and the service node request latency; dynamically adjusting the number of disk write request slots of each service node within each time window based on the load parameter, where the disk write request slots are used to carry the disk write requests in the at least one disk write request queue; dynamically allocating the at least one disk write request to the at least one service node corresponding to each time window based on the number of disk write request slots.

8. The method according to claim 7, wherein dynamically adjusting the number of disk write request slots of each service node within each time window based on the load parameter, including: Based on the number of tasks of the service node, the request latency of the service node, and the load balancing factor, dynamically generate the virtual node adjustment coefficient of each service node within each time window, where the larger the value of the virtual node adjustment coefficient, the lower the load of each service node; Based on the virtual node adjustment coefficient, determine the number of virtual nodes of each service node within each time window; Based on the number of virtual nodes, generate a preset number of virtual node slots for each service node within each time window; Based on the virtual node slots and consecutive N time windows, dynamically adjust the number of disk write request slots of each service node within each time window.

9. The method according to claim 7, wherein The dynamically allocating the at least one disk write request to the at least one service node corresponding to each time window based on the number of disk write request slots includes: Obtain the i-th group of virtual node slots corresponding to the i-th service node within the i-th time window, where i is a positive integer greater than or equal to 2 and less than or equal to N: Based on the i-th group of virtual node slots and the i-th time window, determine the i-th group of disk write request slots of the i-th service node within the i-th time window; Based on the number of the i-th group of disk write request slots, allocate the i-th group of disk write requests in the i-th disk write request queue of the i-th service node to the i-th group of disk write request slots, where the number of the i-th group of disk write request slots is greater than or equal to the number of disk write requests in the i-th group of disk write requests.

10. The method according to any one of claims 1 to 9, wherein The method further includes: When the at least one disk write request includes a first part of disk write requests generated by the first type of data table and a second part of disk write requests generated by the second type of data table, the priority of responding to the second part of disk write requests is higher than the priority of the first part of disk write requests; or When the at least one disk write request only includes a third part of disk write requests generated by the first type of data table, respond to each of the disk write requests in the order of the generation time of each disk write request in the third part of disk write requests; or When the at least one disk write request only includes a fourth part of disk write requests generated by the second type of data table, respond to each of the disk write requests in the order of the generation time of each disk write request in the fourth part of disk write requests.

11. A data disk write system, wherein A monitoring module, configured to obtain at least one data write request for writing target data into a target data table, where the target data table is divided into at least one table unit, and the at least one table unit corresponds to at least one memory space one by one; A disk drop request control module, configured to determine the type of the target data table, and in the case that the type indicates that the target data table is a first type of data table, predict the amount of predicted data written into the at least one memory space within N consecutive time windows, where each data write request is sequentially obtained from the at least one data write request as the current data write request; determine the current table cell corresponding to the current data write request and the current memory space corresponding to the current table cell; use the Poisson distribution to predict the frequency of writing data into the current memory space in the current time window; determine the total number of write requests within the N time windows according to the frequency; predict the current predicted data amount written into the current memory space within the consecutive N time windows according to the total number of write requests, the predicted data amount includes the current predicted data amount, the N time windows include the current time window, and N is a positive integer greater than or equal to 1; A collaborative scheduling module, configured to dynamically allocate the at least one disk drop request to the at least one service node corresponding to each time window based on the load parameters of at least one service node, so as to respectively drop the data in the at least one memory space to at least one hard disk corresponding to the at least one service node.

12. An electronic device, characterized in that, Comprising: A memory, configured to store a computer program; A processor, configured to implement the steps of the data disk drop method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program implements the steps of the data disk drop method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product, comprising a computer program, characterized in that, The computer program implements the steps of the data disk drop method according to any one of claims 1 to 10 when executed by a processor.

Images