Data merging scheduling method and device, and storage medium

Abstract

The application discloses a data merging scheduling method and device and a storage medium, relates to the technical field of data storage, and comprises the following steps: listening to scheduling information of a to-be-executed table, selecting a target to-be-executed table based on the scheduling information, analyzing a data distribution state of the target to-be-executed table, performing hierarchical segmentation based on the data distribution state, and triggering a merging task. According to the application, the automatic triggering and resource-aware scheduling of the table-level merging task are realized by means of an event-driven engine, IO amplification control and accurate cavity cleaning are performed, and the data distribution representation efficiency is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a data merging scheduling method, device and storage medium. BACKGROUND

[0002] The database with LSM (Log-Structured Merge Trees, abbreviated as LSM tree) architecture optimizes query performance through append write and periodic merge operation, and relies on data merging task to clean up storage holes and reduce file overlap to maintain data order. The existing merging scheduling technology generally adopts fixed threshold trigger strategy or manual intervention mechanism, and the file overlap situation (such as the discrete interval distribution of point depth pd) is counted through histogram.

[0003] However, this scheme faces significant defects when facing HTAP (Hybrid Transaction / Analytical Processing) scenarios:

[0004] Lack of maintenance automation: the traditional method requires the database administrator to manually identify the merging time and trigger the task frequently, especially when the transaction processing (TP) load continuously writes small files, the operation and maintenance burden increases dramatically;

[0005] Weak adaptation to HTAP scenarios: frequent random updates of TP data lead to rapid degradation of data distribution (increase of hole rate and overlap), and low-density data flow into the lower layer triggers a chain of merging (waterfall effect), causing excessive input / output (IO) amplification;

[0006] Inefficient data distribution representation: the existing histogram needs at least 16 parameters to describe the file overlap state, the parameters are redundant and cannot restore the distribution pattern characteristics (such as peak position and fluctuation trend), resulting in lack of accurate basis for operation and maintenance decision.

[0007] Therefore, the above defects seriously restrict the automatic operation and maintenance capability and resource utilization efficiency of the database in the HTAP scenario.

[0008] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0009] The main purpose of the present application is to provide a data merging scheduling method, device and storage medium, which aims to solve the technical problems of poor automatic operation and maintenance capability and low resource utilization efficiency of the database in the HTAP scenario.

[0010] To achieve the above purpose, the present application provides a data merging scheduling method, which comprises:

[0011] Listening to the scheduling information of the to-be-executed table, and selecting a target to-be-executed table based on the scheduling information;

[0012] analyzing a data distribution state of the target to-be-executed table, performing hierarchical splitting based on the data distribution state, and triggering a merging task.

[0013] In an embodiment, the scheduling information of the to-be-executed table includes a priority queue of the to-be-executed table, a newly added data file, and a fixed heartbeat; and the step of listening to the scheduling information of the to-be-executed table and selecting a target to-be-executed table based on the scheduling information includes:

[0014] selecting, based on the priority queue of the to-be-executed table, a to-be-executed table whose expiration time has arrived as the target to-be-executed table;

[0015] if a newly added data file is added, moving a to-be-executed table in the newly added data file to the head of the priority queue, setting the expiration time of the moved to-be-executed table as the current time, and obtaining the target to-be-executed table;

[0016] based on the fixed heartbeat of the first preset time interval, checking the priority queue and refreshing the system memory occupancy rate to obtain a missed target to-be-executed table.

[0017] In an embodiment, the step of analyzing the data distribution state of the target to-be-executed table, performing hierarchical splitting based on the data distribution state, and triggering a merging task includes:

[0018] obtaining the memory size of the data file of the target to-be-executed table;

[0019] based on a comparison result of the memory size and a first preset threshold, dividing the data file of the target to-be-executed table into a target layer file and a composite layer file; wherein the target layer file is a data file smaller than the first preset threshold;

[0020] if the total memory size of the data files in the target layer file is greater than the first preset threshold or the number of data files is greater than the current maximum allowed file value, a target layer merging task is generated;

[0021] controlling the upper limit of the number of the target layer files through a Bezier curve.

[0022] In an embodiment, the step of analyzing the data distribution state of the target to-be-executed table, performing hierarchical splitting based on the data distribution state, and triggering a merging task further includes:

[0023] traversing each data file of the composite layer file to identify a target cluster with a point depth greater than or equal to a target threshold;

[0024] calculating the span of the data files in the target cluster, wherein the span is equal to the ratio of the overlap degree to the point depth;

[0025] The span groups are divided based on the span calculation results, and the span groups at least include a wide-span group, a medium-span group and a narrow-span group;

[0026] If the number of span groups is greater than or equal to a target threshold, a composite layer merging task is generated.

[0027] In an embodiment, the step of generating the composite layer merging task if the number of span groups is greater than or equal to the target threshold includes:

[0028] If the number of files in the narrow-span group is greater than or equal to a target threshold, the merged data file is upgraded to the next layer.

[0029] If the number of files in the medium / wide-span group is greater than or equal to a target threshold, the merged data file is retained in the current layer.

[0030] In an embodiment, the step of generating the composite layer merging task if the number of span groups is greater than or equal to the target threshold is followed by:

[0031] The cumulative number of times that the memory amount of the target to-be-executed table is calculated to be greater than a preset memory threshold is calculated.

[0032] If the cumulative number of times reaches a target preset number of times, a tombstone analysis task is triggered.

[0033] The tombstone file tombstone rate and the tombstone score of all data files are calculated.

[0034] If the tombstone file tombstone rate is greater than a second preset threshold, a tombstone merging task is generated.

[0035] Data files with a tombstone score greater than a third preset threshold are screened, and an independent merging task is generated.

[0036] In an embodiment, the step of analyzing the data distribution state of the target to-be-executed table, performing hierarchical segmentation based on the data distribution state, and triggering a merging task is followed by:

[0037] If a merging task is generated based on the data distribution state, the next expiration time of the target to-be-executed table is reset to a second preset time, and the target to-be-executed table is returned to the priority queue.

[0038] If a merging task is not generated based on the data distribution state, the next expiration time of the target to-be-executed table is doubled, and the target to-be-executed table is returned to the priority queue.

[0039] In an embodiment, the step of analyzing the data distribution state of the target to-be-executed table, performing hierarchical segmentation based on the data distribution state, and triggering a merging task is further followed by:

[0040] Sort the data files corresponding to the to-be-executed table in ascending order based on the maximum value to generate a first sequence;

[0041] Extract the point depth of each data file to form a first sequence pair of data files and point depths;

[0042] Linearly fit the first sequence pair by a polynomial function to generate a fitting curve.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a data merging scheduling device, the data merging scheduling device comprises:

[0044] The scheduling management module is configured to listen to scheduling information of the to-be-executed table, and select a target to-be-executed table based on the scheduling information.

[0045] The merging triggering module is configured to analyze the data distribution state of the target to-be-executed table, perform hierarchical segmentation based on the data distribution state, and trigger a merging task.

[0046] In addition, to achieve the above-mentioned purpose, the present application also provides a data merging scheduling device, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the data merging scheduling method as described above.

[0047] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, which is executed by a processor to implement the steps of the data merging scheduling method as described above.

[0048] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the data merging scheduling method as described above.

[0049] The one or more technical solutions provided by the present application have at least the following technical effects:

[0050] The present application listens to the scheduling information of the to-be-executed table, selects a target to-be-executed table based on the scheduling information, analyzes the data distribution state of the target to-be-executed table, performs hierarchical segmentation based on the data distribution state, and triggers a merging task. The present application solves the problems of poor automation operation capability and low resource utilization efficiency of the database in the HTAP scenario, and simultaneously realizes automatic triggering and resource-aware scheduling of table-level merging tasks through an event-driven engine, IO amplification control and accurate hole cleaning, and effective improvement of data distribution representation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or the prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative effort.

[0053] Figure 1 A flowchart provided for the data merging scheduling method of the first embodiment of the present application;

[0054] Figure 2 A flowchart provided for the data merging scheduling method of the second embodiment of the present application;

[0055] Figure 3 A brief schematic diagram of the data merging scheduling method provided for the third embodiment of the present application;

[0056] Figure 4 A flowchart provided for the data merging scheduling method of the fourth embodiment of the present application;

[0057] Figure 5 A flowchart provided for the data merging scheduling method of the fifth embodiment of the present application;

[0058] Figure 6 A flowchart provided for the data merging scheduling method of the sixth embodiment of the present application;

[0059] Figure 7 A brief schematic diagram of the data merging scheduling method provided for the seventh embodiment of the present application;

[0060] Figure 8 A module structure schematic diagram of the data merging scheduling device of the embodiment of the present application;

[0061] Figure 9 A device structure schematic diagram of the hardware running environment involved in the data merging scheduling method of the embodiment of the present application.

[0062] The object implementation, functional features and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0063] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and not to limit the present application.

[0064] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and the specific embodiments.

[0065] The prior art faces problems such as insufficient maintenance automation, weak HTAP (Hybrid Transaction / Analytical Processing) scene adaptation, and inefficient data distribution representation when facing HTAP scenes.

[0066] The present application provides a solution that selects a target to-be-executed table by listening to the scheduling of the to-be-executed table, analyzes the data distribution state of the target to-be-executed table, performs hierarchical segmentation based on the data distribution state, and triggers a merging task. The present application solves the problems of poor automation operation and maintenance capability and low resource utilization efficiency of a database in an HTAP scene. At the same time, the present application realizes automatic triggering and resource-aware scheduling of a table-level merging task through an event-driven engine, controls IO amplification, and accurately cleans up holes, and effectively improves the efficiency of data distribution representation.

[0067] Based on this, the present application embodiment provides a data merging scheduling method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the data merging scheduling method of the present application is shown in the figure.

[0068] In this embodiment, the data merging scheduling method includes steps S10-S40:

[0069] Step S10: Listen to the scheduling information of a to-be-executed table, and select a target to-be-executed table based on the scheduling information.

[0070] It should be noted that the scheduler listens to the scheduling information of the to-be-executed table through an event-driven mechanism, which can specifically include three types of cooperative event sources: a priority queue (sorted by the expiration time of the to-be-executed table), a message pipeline (responding to external events), and a fixed heartbeat (detecting at a preset period). When there is a to-be-executed table with an expiration time less than or equal to the current time in the priority queue, the scheduler selects it as the target to-be-executed table and hands it over to the analysis module. If the message pipeline receives a to-be-executed table change notification (such as a message type indicating a new file), it immediately moves the table to the head of the queue for priority processing. The fixed heartbeat forcibly checks the missing items in the priority queue every first preset time and refreshes the system resource state (CPU / memory load) to ensure that the to-be-executed table is not delayed for a long time in abnormal scenarios. This design solves the inefficiency problem of traditional polling through the dual mechanism of immediate response and periodic guarantee, significantly improving the response speed to high-frequency TP writes in HTAP scenes.

[0071] Step S20: Analyze the data distribution state of the target to-be-executed table, perform hierarchical segmentation based on the data distribution state, and trigger a merging task.

[0072] It should be noted that when the system analyzes the data distribution state of the target to-be-executed table, the hierarchical segmentation strategy is adopted to trigger the merging task; wherein, files smaller than the first preset threshold value can be identified as target layer files, when the total size of the files exceeds the first preset threshold value or the number of files exceeds the dynamic attenuation threshold value, the files are divided into narrow distance groups, medium distance groups and wide distance groups, and the merging task is triggered only when the number of files in the group is greater than or equal to the preset value.

[0073] It can be seen that the present application provides accurate triggering opportunity for the merging task based on event driving, avoids resource waste caused by polling, and solves the HTAP pain points of IO amplification and waterfall effect based on the hierarchical segmentation strategy.

[0074] Further, the scheduling of the to-be-executed table includes a priority queue of to-be-executed tables, a newly added data file and a fixed heartbeat. The step of "listening to the scheduling information of the to-be-executed table, and selecting a target to-be-executed table based on the scheduling information" in step S10 is further refined, including:

[0075] Based on the priority queue of the to-be-executed table, a to-be-executed table whose expiration time has arrived at the current time is selected as the target to-be-executed table;

[0076] It should be noted that the priority queue of the to-be-executed table refers to a data structure sorted by the expiration time of the to-be-executed table, wherein the to-be-executed table with earlier expiration time has higher priority; the current time has arrived means that the expiration time recorded in the to-be-executed table is less than or equal to the current timestamp of the system; the scheduler selects the to-be-executed table that meets the expiration condition as the target to-be-executed table by scanning the head of the priority queue. It can be seen that the passive scheduling mode triggered by time is used to replace the traditional polling, thereby avoiding most of the invalid scanning overhead.

[0077] In a specific embodiment, the priority queue contains table T1 (expiration time 09:00:00) and table T2 (09:00:05). If the current time is 09:00:03, the scheduler only selects table T1 as the target to-be-executed table to hand over to the analysis module.

[0078] If a new data file is added, the to-be-executed table in the new data file is moved to the head of the priority queue, and the expiration time of the moved to-be-executed table is set to the current time to obtain the target to-be-executed table;

[0079] It should be noted that the new data file refers to a new data file generated by a write operation (such as a small file continuously written by TP load); moving to the head of the priority queue refers to adjusting the position of the to-be-executed table in the queue to the highest priority; in addition, the new data file also includes a manually triggered task case, and the manually triggered task refers to that a user initiates a merging operation actively through an instruction (such as an MMsgKindTrigger message type); when the message pipeline receives an MMsgKindTableChange type message (indicating a new file), the scheduler immediately moves the associated table to the head of the priority queue and sets the expiration time thereof to the current time (equivalent to shortening to 0 seconds), so as to ensure that the next scheduling cycle is processed in priority.

[0080] In a specific embodiment, a user adds 5 data files to an order table, the system generates an MMsgKindTableChange message, the scheduler moves the order table to the head of the priority queue and sets the expiration time to the current time, so that the order table is immediately selected as a target to-be-executed table in the next event loop.

[0081] Based on the fixed heartbeat of the first preset time interval, the priority queue is checked and the resource state is refreshed to obtain a missed target to-be-executed table.

[0082] It should be noted that the fixed heartbeat refers to a periodically executed system check event; the refreshed resource state refers to collecting real-time indexes such as CPU utilization and memory occupancy; and the missed target to-be-executed table refers to an expired table that is not normally scheduled due to message loss or queue exception.

[0083] In a specific embodiment, the fixed heartbeat is triggered at 09:00:10, it is detected that the table T3 (with an expiration time of 09:00:08) is not processed, and the table T3 is handed over to the analysis module as a missed target; the remaining available memory of the operating system is obtained through a file system interface at the heartbeat time, the available memory threshold is set based on the remaining available memory, for example, the available memory threshold is set to 80% of the remaining available memory; the preset consumed memory of the merging task is calculated, if the preset consumed memory exceeds the available memory threshold, the number of files of each to-be-executed merging task is halved to meet the condition requirement of the available memory threshold; if the total size of the merged files still exceeds the available memory threshold after the number of files of each to-be-executed merging task is halved, halving is continued until the number of files is reduced to 1 and the available memory threshold cannot still be met, the merging task is cancelled, a log is recorded, and a manager is notified.

[0084] In a possible embodiment, the number of concurrent merging tasks is also related to resource states such as CPU utilization, network bandwidth, and disk IO load.

[0085] Further, with reference to Figure 2The second embodiment of the data merging scheduling method of the present application provides a flowchart. Based on the above-mentioned embodiments, the step of "analyzing the data distribution state of the target to-be-executed table, performing hierarchical segmentation based on the data distribution state, and triggering a merging task" in step S20 is further refined, including steps A201-A204: Figure 2

[0086] Step A201: Obtain the memory size of the target to-be-executed table data file.

[0087] It should be noted that the purpose of obtaining the memory size is to perform hierarchical division to ensure accurate separation of target layer files (small files) and composite layer files (regular files), and to avoid the problem of write amplification caused by mixed merging.

[0088] Step A202: Based on the comparison result of the memory size and the first preset threshold, divide the target to-be-executed table data file into target layer files and composite layer files; wherein the target layer file is a data file smaller than the first preset threshold.

[0089] It should be noted that the first preset threshold refers to the size threshold (such as 128 MB by default) for distinguishing target layer files and composite layer files; the target layer file refers to a data file with a memory size smaller than the first preset threshold, i.e., a 0-layer file; and the composite layer file refers to a data file with a memory size greater than or equal to the first preset threshold, such as a 1-layer and above.

[0090] In a possible implementation, the first preset threshold can be configured as 64 MB or 256 MB to adapt to the storage block size of different hardware environments.

[0091] Step A203: If the total memory size of the data files in the target layer file is greater than the first preset threshold or the number of data files is greater than the current maximum file value allowed, generate a target layer merging task.

[0092] It should be noted that the total memory size refers to the sum of the memory sizes of all data files in the target layer; the current maximum file value allowed refers to the maximum number of files temporarily stored in the target layer, which is dynamically attenuated according to the Bezier curve; and the target layer merging task refers to the merging operation task for the target layer file. The scheduler generates a target merging task when any of the following conditions is met: the total size of the target layer file is greater than the first preset threshold, and the number of target layer files is greater than the current maximum file number (calculated according to time decay, wherein the decay calculation of the current maximum file value can be realized through a pre-generated Bezier curve lookup table to avoid real-time calculation overhead); and through the dynamic attenuation mechanism, the dual goals of allowing accumulation of small files in the early stage to reduce the merging frequency (to reduce write amplification) and tightening the threshold in the later stage to avoid the impact of metadata expansion on query efficiency are balanced. ​

[0093] In one possible implementation, the target layer of the order table currently has 20 files with a total size of 120MB. The system time is 30 minutes since the last merge. The table shows that the maximum number of files allowed at this time is 8. Therefore, the scheduler generates a target layer merge task to merge the 20 files into a new file of 150MB and move it into the first layer.

[0094] Step A204: Control the upper limit of the number of target layer files using a Bézier curve.

[0095] In one specific embodiment, see Figure 3 The initial value of the Bézier curve is 32, the final value is 1, the decay process takes 1 hour, and the specific control points of the decay process include (0,0), (0.70,0), (0.0,1.0), and (1,1).

[0096] Furthermore, referring to Figure 4 The fourth embodiment of the data merging and scheduling method in this application provides a flowchart, based on the above. Figure 4 The embodiment shown further refines step S20, "analyzing the data distribution status of the target table to be executed, performing hierarchical segmentation based on the data distribution status, and triggering the merging task," and also includes steps A401 to A404:

[0097] Step A401: Traverse each layer of the composite layer file and identify target clusters with a point depth greater than or equal to the target threshold;

[0098] It should be noted that point depth (pd) refers to the number of files in the entire data file set through which a certain data value is transmitted. For a single file, the maximum point depth containing all values ​​is taken. The target threshold refers to the lower limit of the point depth that triggers cluster identification. By focusing on highly overlapping areas, the unnecessary computational overhead caused by processing low-overlapping files is avoided.

[0099] Step A402: Calculate the span of the data files in the target cluster, wherein the span is equal to the ratio of overlap to point depth;

[0100] It should be noted that overlap count (oc) refers to the number of other files that overlap with the target data file; span is defined as the ratio of overlap count to point depth (i.e., Span = oc / pd), used to quantify the density of data distribution; the scheduler calculates the span value for each data file in the target cluster (e.g., calculating separately for the 15 files in cluster C1), the smaller the value, the denser the file data (narrow span), and the larger the value, the more dispersed (wide span); the span calculation eliminates the bias caused by the difference in the number of files by using a ratio, accurately representing the data density distribution.

[0101] Step A403: dividing span groups based on span calculation results, the span groups at least including a wide-span group, a medium-span group and a narrow-span group;

[0102] It should be noted that the span group refers to dividing files into three types according to span values: a narrow-span group, a medium-span group and a wide-span group; the grouping strategy solves the waterfall effect problem by isolating files of different tightness.

[0103] In a possible implementation, the narrow-span group refers to a file set with a span value less than 2 (i.e. Span<2), the medium-span group refers to a file set with a span value satisfying 2≤Span≤10, and the wide-span group refers to a file set with a span value greater than 10 (i.e. Span>10).

[0104] Step A404: generating a composite layer merging task if the number of span group files is greater than or equal to a target threshold.

[0105] It should be noted that the composite layer merging task refers to a merging operation task for non-target layer files. By grouping target threshold control, it is ensured that only file clusters with merging value generate tasks to avoid invalid IO consumption.

[0106] Further, the step of "generating a composite layer merging task if the number of span group files is greater than or equal to a target threshold" is further refined, including:

[0107] If the number of narrow-span group files is greater than or equal to the target threshold, the merged data file is upgraded to the next layer;

[0108] It should be noted that the target threshold refers to the minimum number of files triggering the merging task; the merged data file is upgraded to the next layer, which refers to migrating the new file generated by merging to a higher level, such as from the first layer to the second layer. The merging execution module generates a narrow-span group merging task when the number of narrow-span group files is greater than or equal to the target threshold, and the output file is promoted to the next level. This design avoids mixing low-density data to trigger a chain merge, such as the waterfall effect, while reducing write amplification by moving high-density data to a higher level.

[0109] In a possible implementation, the layer upgrade operation is implemented by updating the level field of the file metadata and synchronously updating the level index.

[0110] For example, in a specific implementation, the L1 layer narrow-span group of the order table contains 5 files (the average span value is 1.2), and because the number of files is ≥3, the merging execution module generates a task: merging the 5 files into 1 new file and moving to the L2 layer.

[0111] If the number of medium / wide distance group files is greater than or equal to the target threshold, the merged data file is retained in the current layer.

[0112] It should be noted that by isolating low-density data (wide distance group) and medium-density data (medium distance group), it is prevented from flowing into the lower layer to dilute the data density, avoiding invalid cascading merging triggered by density reduction, such as waterfall effect, and reducing IO consumption. File layer retention is achieved by maintaining the level field of the metadata unchanged.

[0113] In this embodiment, the narrow distance group is used to promote the high-level data density, and the medium / wide distance group is used to block the downward infiltration of low-density data, both of which work together to solve the waterfall effect and achieve the technical effect of IO amplification.

[0114] Further, referring to Figure 5 , the fifth embodiment of the data merging scheduling method provides a flowchart, which is based on the above Figure 5 embodiment, and further includes steps A501-A504 after the step of "if the number of cross-distance group files is greater than or equal to the target threshold, generating a composite layer merging task":

[0115] Step A501: Calculate the cumulative number of times that the memory amount of the merging task of the target to-be-executed table is greater than the preset memory threshold;

[0116] It should be noted that the merging task includes a target layer merging task and a composite layer merging task; in one possible implementation, the preset memory threshold is 120MB, and the number of merging tasks exceeding 120MB of memory amount is counted.

[0117] Step A502: If the cumulative number of times reaches a target preset number of times, trigger a hole analysis task;

[0118] It should be noted that the target preset number of times refers to the cumulative merging task threshold for triggering hole analysis; the hole analysis task refers to an operation instruction specially detecting storage space (tombstone file missing row, data file deleted). By limiting the frequency of hole analysis through the task number threshold, system load fluctuation caused by high-frequency IO operation is avoided, while ensuring that the storage space is cleaned up within a reasonable period.

[0119] Step A503: Calculate the tombstone file hole rate and the hole score of all data files;

[0120] It should be noted that the tombstone file hole rate refers to the proportion of the number of missed rows in the tombstone file to the total number of tombstone rows, and the missed row refers to the row whose corresponding data file has been deleted; the hole score refers to the priority score calculated by comprehensively considering the size of the data file and the level it is in, and the formula is:

[0121]

[0122] wherein, the score is a hollow score of the data file, the DelRowCount is the number of missed rows in the tombstone file, the RowCount is the total tombstone row number, the Size is the actual memory size of the data file, the Level is the composite layer level where the data file is located, wherein the target layer level is set to 0 layer, the MaxSize is a first preset threshold, and the MaxLevel is the highest level of the data file; by default, MaxSize = 128 MB, and MaxLevel = 8; the smaller the file size, the lower the level, and the easier the merging operation is performed; the weights of the file size and the level in the above formula are both set to 50%.

[0123] Step A504: generating a tombstone merging task if the tombstone file hollow rate is greater than a second preset threshold.

[0124] It should be noted that the tombstone merging task refers to an operation task of cleaning the tombstone file hollow; by merging all tombstone files to release the storage space occupied by the missed rows, storage waste caused by the high tombstone file hollow rate is avoided, invalid space is recovered in time, and storage utilization is improved.

[0125] Step A505: screening data files with a hollow score greater than a third preset threshold to generate an independent merging task.

[0126] It should be noted that the third preset threshold linearly decays with the longest survival time of the tombstone file, for example, the initial value (survival time 0 minutes) of the third preset threshold is 60%, and the final value (survival time 40 minutes) is 10%, and the decay formula is: current threshold = max(10%, 60% - (survival time / 40 minutes) × 50%); the independent merging task refers to a dedicated merging operation of a single high hollow score data file, which is used to clean invalid data (corresponding to the record of the tombstone file) in the data file, to recover invalid space, reduce memory pressure, and improve query performance. For example, based on the linear time decay of the third preset threshold, after triggering the independent merging task, the invalid data is cleaned, the hollow rate of the tombstone file is improved, and exceeds the second preset threshold, so that a tombstone merging task is generated to recover more storage space. The present application realizes efficient use of storage resources and improvement of query performance by dynamically adjusting the threshold and the independent merging task.

[0127] The embodiment of the present application solves the hollow gap IO amplification problem based on the counting triggered mode, improves the storage utilization rate by threshold cleaning, and preferentially processes high threshold targets by using linear decay threshold, to avoid invalid IO consumption.

[0128] Further, the step of "analyzing the data distribution state of the target to-be-executed table, performing hierarchical segmentation based on the data distribution state, and triggering a merging task" in step S20 includes the following steps:

[0129] If a merging task is generated based on the data distribution state, the next expiration time of the target to-be-executed table is reset to the second preset time, and the process returns to the priority queue.

[0130] It should be noted that generating a merging task based on the data distribution state means that at least one merging task is successfully created by analyzing the data distribution of the target to-be-executed table (such as the target layer file exceeding the limit, the span grouping satisfying the condition, etc.). After generating the merging task, the scheduler resets the next expiration time of the table to the second preset time, ensuring that the newly generated changes are checked again in the short term. This immediate feedback mechanism solves the inefficiency problem of traditional polling, enabling the system to quickly respond to the continuous writing of TP data in the HTAP scenario, reducing the merging delay from minutes to seconds.

[0131] If no merging task is generated based on the data distribution state, the next expiration time of the target to-be-executed table is doubled, and the process returns to the priority queue.

[0132] It should be noted that no merging task is generated based on the data distribution state, which means that no merging task is triggered after analysis (such as the target layer file not exceeding the limit, the number of span grouping files being insufficient, etc.). When no task is generated, the scheduler can extend the check interval by an exponential backoff algorithm (such as 5 seconds→10 seconds→20 seconds), significantly reducing the frequency of invalid analysis. In one possible implementation, the interval is doubled each time no task is generated; when an external event is accessed, the initial interval time is reset, which forms a redundant guarantee with the fixed heartbeat, ensuring that long-idle tables can still be checked periodically.

[0133] In a specific implementation, no task is generated after analyzing the order table (such as the number of target layer files = 10 and the current decay threshold = 12), and the scheduler extends the next expiration time of the table from 5 seconds to 10 seconds (such as the original expiration time 09:00:10→new expiration time 09:00:20). If there is still no task next time, it will be extended to 20 seconds.

[0134] This embodiment solves the problems of TP data rapid degradation and IO amplification through the scenario conditions of resetting the next expiration time and doubling the next expiration time, which form an adaptive closed loop to replace manual intervention.

[0135] Further, referring to Figure 6 , the sixth embodiment of the data merging scheduling method of the present application provides a flowchart, which is based on the above Figure 6In the embodiment shown, the step of "analyzing the data distribution state of the target to-be-executed table, performing hierarchical segmentation based on the data distribution state, and triggering a merging task" in step S20 further includes steps A601-A603:

[0136] Step A601: Sort all data files corresponding to the to-be-executed table in ascending order based on the maximum value of the sorting key, to generate a first sequence.

[0137] It should be noted that the maximum value of the sorting key refers to the maximum value of the sorting key recorded in the metadata of the data file, which represents the upper limit of the data range of the file; the ascending order refers to sorting the data files in ascending order of the maximum value of the sorting key; and the first sequence refers to the ordered data file list generated after sorting. The sorting provides ordered input for point-depth sequence fitting, ensuring that the fitting curve accurately reflects the continuity of data distribution, such as overlapping trends.

[0138] In one possible implementation, the sorting operation can be implemented through a quicksort algorithm to meet the processing needs of large-scale files.

[0139] Step A602: Extract the point depth of each data file to form a first sequence pair of data files and point depths.

[0140] It should be noted that the first sequence pair refers to a sequence of binary tuples composed of ordered data files and their corresponding point depth values, i.e., [(file location, pd value)]. The scheduler traverses the ordered file sequence generated in step A801 to extract the pd value of each file (precomputed and cached in the metadata), forming a location-depth mapping relationship.

[0141] In one possible implementation, the pd value is precomputed through an interval tree algorithm to ensure read efficiency in high-concurrency scenarios.

[0142] Step A603: Linearly fit the first sequence pair through a polynomial function to generate a fitting curve.

[0143] It should be noted that the linear fitting of the polynomial function can use the least squares method to fit the discrete sequence pair into a continuous polynomial curve.

[0144] In one possible implementation, with reference to Figure 7The pd fitting graph corresponding to 2800 files is drawn, wherein the abscissa represents the order position of the file in ascending order of the maximum value of the sorting key (the smaller the maximum value of the sorting key, the earlier the position), and the ordinate represents the point depth of the corresponding file (the larger the ordinate value, the higher the degree of overlap of the data of the file with other files); the sequence pair is normalized, and a fourth order polynomial function (such as the form: pd(x) = a0+a1x+a2x 2 +a3x 3 +a4x 4 ) can be used to fit the data points, and five coefficients (a0 to a4) are output. The traditional histogram is replaced by function fitting, and the "position-pd" sequence of 2800 files is converted into a continuous curve, which intuitively shows the distribution form of data overlap (such as the peak value at both ends of the curve indicating that the overlap is concentrated in the first and last intervals of the maximum value of the sorting key, i.e. the high-frequency write area in the TP scenario), and only five parameters are used to compress the data overlap distribution form, while the key distribution characteristics (such as peak position and fluctuation trend) can be retained.

[0145] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data merging and scheduling method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0146] The present application also provides a data merging and scheduling device, please refer to Figure 8 , the data merging and scheduling device comprises:

[0147] 10 scheduling management module, for listening to the scheduling information of the to-be-executed table, selecting the target to-be-executed table based on the scheduling information;

[0148] 20 merging trigger module, analyzing the data distribution state of the target to-be-executed table, performing hierarchical segmentation based on the data distribution state, and triggering the merging task.

[0149] The data merging and scheduling device provided by the present application adopts the data merging and scheduling method in the above embodiment, which can solve the technical problems of poor automation operation capability and low resource utilization efficiency of the database in the HTAP scenario. Compared with the prior art, the data merging and scheduling device provided by the present application has the same beneficial effects as the data merging and scheduling method provided by the above embodiment, and the other technical features in the data merging and scheduling device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0150] The application provides a data merging scheduling device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data merging scheduling method in the above embodiment one.

[0151] Reference will now be made to the following description Figure 9 which shows a structural diagram of a data merging scheduling device suitable for implementing embodiments of the application. The data merging scheduling device in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 9 The data merging scheduling device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the application.

[0152] As shown in Figure 9 , the data merging scheduling device can include a processing apparatus 1001 (for example, a central processor, a graphic processor, or the like) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage apparatus 1003 into a random access memory 1004. In the random access memory 1004, various programs and data required for operation of the data merging scheduling device are also stored. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication apparatus 1009. The communication apparatus 1009 can allow the data merging scheduling device to perform wireless or wired communication with other devices to exchange data. Although the data merging scheduling device having various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or provided. More or fewer systems can be alternatively implemented or provided.

[0153] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0154] The data merging and scheduling device provided by the present application adopts the data merging and scheduling method in the above-mentioned embodiments, and can solve the technical problems of poor automatic operation and maintenance capability and low resource utilization efficiency of the database in the HTAP scenario. Compared with the prior art, the data merging and scheduling device provided by the present application has the same beneficial effects as the data merging and scheduling method provided by the above-mentioned embodiments, and other technical features in the data merging and scheduling device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0155] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0156] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0157] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the data merging and scheduling method in the above-mentioned embodiments.

[0158] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0159] The computer readable storage medium described above may be contained in the data merging and scheduling device, or may exist separately without being assembled into the data merging and scheduling device.

[0160] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the data merging and scheduling device, the data merging and scheduling device listens to scheduling information of a to-be-executed table, selects a target to-be-executed table based on the scheduling information, analyzes a data distribution state of the target to-be-executed table, performs hierarchical segmentation based on the data distribution state, and triggers a merging task.

[0161] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0162] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0163] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0164] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned data merging scheduling method, and can solve the technical problems of poor automatic operation ability and low resource utilization efficiency of a database in an HTAP scenario. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the data merging scheduling method provided by the above-mentioned embodiments, and will not be described here.

[0165] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the data merging scheduling method as described above.

[0166] The computer program product provided by the application can solve the technical problems of poor automatic operation and maintenance capability and low resource utilization efficiency of a database in an HTAP scenario. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the data merging scheduling method provided by the above-described embodiments, and are not described here.

[0167] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like made by using the content of the application specification and drawings is included in the patent protection scope of the application.

Claims

1. A data merging and scheduling method, characterized in that, The data merging and scheduling method includes: Monitor the scheduling information of the tables to be executed, and select a target table to be executed based on the scheduling information; Analyze the data distribution of the target table to be executed, perform hierarchical segmentation based on the data distribution, and trigger the merging task. The steps of analyzing the data distribution status of the target table to be executed, performing hierarchical partitioning based on the data distribution status, and triggering the merging task include: Get the memory size of the target table data file to be executed; Based on the comparison between the memory size and the first preset threshold, the target table data file to be executed is divided into a target layer file and a composite layer file; wherein, the target layer file is a data file smaller than the first preset threshold; If the total memory size of the data files in the target layer file is greater than the first preset threshold or the number of data files is greater than the current maximum allowed file value, then a target layer merging task is generated; The maximum number of files in the target layer is controlled by a Bézier curve. The steps of analyzing the data distribution status of the target table to be executed, performing hierarchical partitioning based on the data distribution status, and triggering the merging task further include: Traverse each layer of the composite layer file to identify target clusters whose point depth is greater than or equal to the target threshold. Calculate the span of the data files in the target cluster, where the span is equal to the ratio of overlap to point depth; Based on the span calculation results, the span groups are divided, and the span groups include at least a wide span group, a medium span group, and a narrow span group; If the number of cross-grouped files is greater than or equal to the target threshold, a composite layer merging task is generated.

2. The data merging and scheduling method as described in claim 1, characterized in that, The scheduling information of the table to be executed includes: the priority queue of the table to be executed, newly added data files, and fixed heartbeats; the step of monitoring the scheduling information of the table to be executed and selecting a target table to be executed based on the scheduling information includes: Based on the priority queue of the pending execution table, the pending execution table that has expired at the current time is selected as the target pending execution table; If a new data file is added, the table to be executed in the new data file is moved to the head of the priority queue, and the expiration time of the moved table to be executed is set to the current time to obtain the target table to be executed. Based on a fixed heartbeat at a first preset time interval, check the priority queue and refresh the system memory usage to obtain any missing target execution tables.

3. The data merging and scheduling method as described in claim 1, characterized in that, The step of generating a composite layer merging task if the number of cross-span grouped files is greater than or equal to the target threshold includes: If the number of files in the narrow-group is greater than or equal to the target threshold, the merged data files will be moved to the next level. If the number of files in the medium / wide spacing group is greater than or equal to the target threshold, the merged data files will remain in the current layer.

4. The data merging and scheduling method as described in claim 3, characterized in that, The step of generating a composite layer merging task if the number of cross-span grouped files is greater than or equal to the target threshold includes: Calculate the cumulative number of times the memory usage of the merge task for the target table to be executed exceeds a preset memory threshold; If the cumulative number of times reaches the target preset number, a void analysis task is triggered; Calculate the hole rate of the tombstone file and the hole score of all data files; If the void ratio of the tombstone file is greater than the second preset threshold, a tombstone merging task is generated. Data files with a hole score greater than a third preset threshold are selected and an independent merging task is generated.

5. The data merging and scheduling method as described in claim 1, characterized in that, After analyzing the data distribution status of the target table to be executed, performing hierarchical segmentation based on the data distribution status, and triggering the merging task, the following steps are included: If a merge task is generated based on the data distribution status, the next expiration time of the target table to be executed will be reset to the second preset time, and it will be returned to the priority queue. If no merge task is generated based on the data distribution status, the next expiration time of the target pending table is doubled, and it is returned to the priority queue.

6. The data merging and scheduling method as described in claim 1, characterized in that, After analyzing the data distribution status of the target table to be executed, performing hierarchical partitioning based on the data distribution status, and triggering the merging task, the process further includes: Sort all data files corresponding to the table to be executed in ascending order based on the maximum value to generate the first sequence; Extract the point depth of each data file to form the first sequence pair of data file and point depth; The first sequence pair is linearly fitted using a polynomial function to generate a fitted curve.

7. A data merging and scheduling device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data merging and scheduling method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data merging and scheduling method as described in any one of claims 1 to 6.

Images