Data management

Abstract

A data management method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring the current state of managed data; on the basis of the current state of the managed data and at least one indicator of the managed data, determining a target merging type of the managed data; and according to the target merging type, merging the managed data, such that the managed data is in a target state. At least one indicator of managed data may be used for representing an organization form suitable for the managed data at this moment, and a target state is the organization form suitable for the managed data at this moment. Therefore, by means of the method, when the managed data is managed, the organization form thereof can be dynamically adjusted, such that the managed data provides a service to the outside in the most suitable organization form all the time, and thus the service can achieve the optimal performance all the time.

Description

Data Management Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and more particularly, to methods and devices, electronic devices, and storage media for data management. Background Art

[0002] With the rapid development of the internet and information technology, data generation is exploding, placing increasing demands on databases and their management. Based on the different ways data is organized within persistent storage media, storage can be categorized into row-based storage and column-based storage. Row-based storage organizes data by row, storing contiguous data across rows together; column-based storage organizes data by column, storing contiguous data across columns together. Different data organization formats are suitable for different data processing scenarios.

[0003] In related technologies, data is organized in a specific form, such as row storage, column storage, redundant row storage, etc., which makes the data perform better in data processing scenarios suitable for its organizational form, but performs poorly in other scenarios. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data management method and apparatus, an electronic device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions.

[0006] According to a first aspect of one or more embodiments of the present specification, a data management method is proposed, comprising: obtaining a current state of managed data; determining a target merge type of the managed data based on the current state of the managed data and at least one indicator of the managed data; and merging the managed data according to the target merge type so that the managed data is in a target state.

[0007] In one embodiment of the present specification, determining the target merge type of the managed data based on the current state of the managed data and at least one indicator of the managed data includes: determining the target merge type from the candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data.

[0008] In one embodiment of the present specification, obtaining the current state of the managed data includes: obtaining the definition and mode of the managed data, wherein the definition of the managed data includes a redundant row storage and column storage table and a pure column storage and column storage table. In a case where the definition of the managed data is the redundant row storage and column storage table, the mode of the managed data includes a redundant row storage mode and a pure row storage mode under redundant row storage; in a case where the definition of the managed data is the pure column storage and column storage table, the mode of the managed data includes a pure column storage mode and a pure row storage mode under pure column storage.

[0009] In one embodiment of the present specification, the target merge type is determined from the candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data, including: when the definition of the managed data is the same as the mode of the managed data, the target merge type is determined in the basic column storage merge and the row storage oriented merge based on at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row storage oriented merge; when the definition of the managed data is different from the mode of the managed data, the target merge type is determined in the row storage oriented merge and the reconstructed column storage merge based on at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

[0010] In one embodiment of the present specification, obtaining the current state of the managed data includes: obtaining the mode of the managed data, wherein the mode of the managed data includes a redundant row storage mode, a pure row storage mode under redundant row storage, a pure column storage mode, and a pure row storage mode under pure column storage.

[0011] In one embodiment of the present specification, the target merge type is determined from the candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data, including: when the mode of the managed data is a redundant row storage mode or a pure column storage mode, determining the target merge type in a basic column storage merge and a row-oriented merge based on at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row-oriented merge; when the mode of the managed data is a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage, determining the target merge type in a row-oriented merge and a reconstructed column storage merge based on at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

[0012] In one embodiment of the present specification, determining the target merge type in the basic column storage merge and the row storage merge based on at least one indicator of the managed data includes: in response to at least one indicator of the managed data satisfying the de-column storage condition, determining the row storage merge as the target merge type; in response to at least one indicator of the managed data not satisfying the de-column storage condition, determining the basic column storage merge as the target merge type.

[0013] In one embodiment of the present specification, determining the target merge type in the row-oriented merge and the reconstructed column-oriented merge based on at least one indicator of the managed data includes: in response to at least one indicator of the managed data satisfying the reconstructed column-oriented merge condition, determining the reconstructed column-oriented merge as the target merge type; in response to at least one indicator of the managed data not satisfying the reconstructed column-oriented merge condition, determining the row-oriented merge as the target merge type.

[0014] In one embodiment of the present specification, the de-columning condition includes at least a first number of indicators being greater than the de-columning threshold corresponding to the indicator; the re-columning condition includes at least a second number of indicators being less than the re-columning threshold corresponding to the indicator; wherein the de-columning threshold of each indicator is greater than the re-columning threshold of the indicator.

[0015] In one embodiment of the present specification, the de-columning condition includes that at least a third number of indicators is greater than the de-columning threshold corresponding to the indicator, and at least a first proportion of indicators among all indicators is not less than the reconstruction column storage threshold corresponding to the indicator; the reconstruction column storage condition includes that at least a fourth number of indicators is less than the reconstruction column storage threshold corresponding to the indicator, and at least a second proportion of indicators among all indicators is not greater than the de-columning threshold corresponding to the indicator; wherein the de-columning threshold of each indicator is greater than the reconstruction column storage threshold of the indicator.

[0016] In one embodiment of the present specification, the managed data is stored in different nodes in a distributed database; the method further comprises: synchronizing the target merge type to each node where the managed data is located.

[0017] In one embodiment of the present specification, the method further includes: in response to receiving a data query request for the managed data, determining a data query strategy according to a current state of the managed data, and executing a data query for the managed data according to the data query strategy.

[0018] In one embodiment of the present specification, the at least one indicator of the managed data includes at least one of the following: number of rows, number of columns, amount of data, and applicable query method.

[0019] According to the second aspect of one or more embodiments of the present specification, a data management device is proposed, comprising: an acquisition module for acquiring the current status of managed data; a determination module for determining a target merge type of the managed data based on the current status of the managed data and at least one indicator of the managed data; and a merging module for merging the managed data according to the target merging type.

[0020] In one embodiment of the present specification, the determining module is configured to determine a target merging type from candidate merging types corresponding to a current state of the managed data according to at least one indicator of the managed data.

[0021] In one embodiment of the present specification, the acquisition module is used to: acquire the definition and mode of the managed data, wherein the definition of the managed data includes a redundant row storage and column storage table and a pure column storage and column storage table. In the case where the definition of the managed data is the redundant row storage and column storage table, the mode of the managed data includes a redundant row storage mode and a pure row storage mode under redundant row storage; in the case where the definition of the managed data is the pure column storage and column storage table, the mode of the managed data includes a pure column storage mode and a pure row storage mode under pure column storage.

[0022] In one embodiment of the present specification, the determination module is used to determine the target merge type from the candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data, and is used to: when the definition of the managed data is the same as the mode of the managed data, determine the target merge type in the basic column storage merge and the row storage oriented merge based on at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row storage oriented merge; when the definition of the managed data is different from the mode of the managed data, determine the target merge type in the row storage oriented merge and the reconstructed column storage merge based on at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

[0023] In one embodiment of the present specification, the acquisition module is used to: obtain the mode of the managed data, wherein the mode of the managed data includes a redundant row storage mode, a pure row storage mode under redundant row storage, a pure column storage mode, and a pure row storage mode under pure column storage.

[0024] In one embodiment of the present specification, the determination module is used to determine the target merge type from the candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data, and is used to: when the mode of the managed data is a redundant row storage mode or a pure column storage mode, determine the target merge type in the basic column storage merge and the row-oriented merge based on at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row-oriented merge; when the mode of the managed data is a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage, determine the target merge type in the row-oriented merge and the reconstructed column storage merge based on at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

[0025] In one embodiment of the present specification, the determination module is used to determine the target merge type in the basic column storage merge and the row storage merge based on at least one indicator of the managed data, and is used to: in response to at least one indicator of the managed data satisfying the de-column storage condition, determine the row-oriented merge as the target merge type; in response to at least one indicator of the managed data not satisfying the de-column storage condition, determine the basic column storage merge as the target merge type.

[0026] In one embodiment of the present specification, the determination module is used to determine the target merge type in the row-oriented merge and the reconstructed column-oriented merge based on at least one indicator of the managed data, and is used to: in response to at least one indicator of the managed data satisfying the reconstructed column-oriented merge condition, determine the reconstructed column-oriented merge as the target merge type; in response to at least one indicator of the managed data not satisfying the reconstructed column-oriented merge condition, determine the row-oriented merge as the target merge type.

[0027] In one embodiment of the present specification, the de-columning condition includes at least a first number of indicators being greater than the de-columning threshold corresponding to the indicator; the re-columning condition includes at least a second number of indicators being less than the re-columning threshold corresponding to the indicator; wherein the de-columning threshold of each indicator is greater than the re-columning threshold of the indicator.

[0028] In one embodiment of the present specification, the de-columning condition includes that at least a third number of indicators is greater than the de-columning threshold corresponding to the indicator, and at least a first proportion of indicators among all indicators is not less than the reconstruction column storage threshold corresponding to the indicator; the reconstruction column storage condition includes that at least a fourth number of indicators is less than the reconstruction column storage threshold corresponding to the indicator, and at least a second proportion of indicators among all indicators is not greater than the de-columning threshold corresponding to the indicator; wherein the de-columning threshold of each indicator is greater than the reconstruction column storage threshold of the indicator.

[0029] In one embodiment of the present specification, the managed data is stored in different nodes in a distributed database; the device further includes a synchronization module for synchronizing the target merge type to each node where the managed data is located.

[0030] In one embodiment of the present specification, the device further includes a query module for: in response to receiving a data query request for the managed data, determining a data query strategy according to the current state of the managed data, and executing a data query for the managed data according to the data query strategy.

[0031] In one embodiment of the present specification, the at least one indicator of the managed data includes at least one of the following: number of rows, number of columns, amount of data, and applicable query method.

[0032] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0033] According to a fourth aspect of one or more embodiments of this specification, an electronic device is proposed, comprising a processor and a memory for storing processor-executable instructions; wherein the processor implements the method described in the first aspect by running the executable instructions.

[0034] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0035] The technical solutions provided by the embodiments of this specification may include the following beneficial effects.

[0036] The data management method provided in the embodiments of this specification can determine a target merge type for the managed data based on the current state of the managed data and at least one indicator of the managed data, and then merge the managed data using the target merge type to bring the managed data into the target state. The at least one indicator of the managed data can be used to characterize the organizational form suitable for the managed data at that moment, i.e., the organizational form that maximizes performance. The target state is the organizational form suitable for the managed data at that moment. Therefore, this method can dynamically adjust the organizational form of the managed data when managing it, so that it always provides external services in the most suitable organizational form, thereby ensuring that the services always achieve their highest performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a flow chart of a data management method provided by an exemplary embodiment.

[0038] FIG2 is a scheduling diagram of a data management method provided by an exemplary embodiment.

[0039] FIG3A and FIG3B are schematic diagrams showing the relationship between the merge type and the mode of the managed data provided by an exemplary embodiment.

[0040] FIG4 is a schematic structural diagram of a device provided by an exemplary embodiment.

[0041] FIG5 is a block diagram of a data management device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0043] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0044] With the rapid development of the internet and information technology, data generation is exploding, placing increasing demands on databases and their management. Based on the different ways data is organized within persistent storage media, storage can be categorized into row-based storage and column-based storage. Row-based storage organizes data by row, storing contiguous data across rows together; column-based storage organizes data by column, storing contiguous data across columns together. Different data organization formats are suitable for different data processing scenarios.

[0045] In related technologies, data is organized in a specific form, such as row storage, column storage, redundant row storage, etc., which makes the data perform better in data processing scenarios suitable for its organizational form, but performs poorly in other scenarios.

[0046] Based on this, on the first aspect, at least one embodiment of this specification provides a data management method, which can dynamically adjust the organizational form of data (i.e., storage form) when managing data to adapt to the data storage needs of various scenarios, so that it can always provide services to the outside world in the most suitable organizational form, so that the service can always achieve its highest performance.

[0047] Please refer to FIG1 , which exemplarily shows the process of the method, including steps S101 to S103 .

[0048] In step S101, the current state of the managed data is obtained.

[0049] Optionally, the managed data can be an SSTable (Sorted String Table, representing static read-only data in the LSM-Tree storage engine) in the database; preferably, the managed data can be the highest-level SSTable in the database. SSTables based on the LSM-Tree structure database system can also have three modes: pure row storage, pure column storage, and redundant row storage: in the pure row storage mode, there is only a single SSTable at the highest level, and each row stores the entire row data; in the pure column storage mode, the highest level maintains an SSTable for each column group, and only stores data on the corresponding column; in the redundant row storage mode, an additional pure row storage SSTable is maintained on the basis of pure column storage. During query, the optimizer chooses to read data from the row storage SSTable or the partial column group SSTable based on the cost model. The three storage modes also have their own advantages and disadvantages: Pure row storage SSTable is suitable for transactional tasks in OLTP scenarios, can efficiently execute read and write transactions on multiple columns, and is simpler to optimize and maintain, but it also has problems such as low query efficiency and poor aggregation operation performance; Pure column storage SSTable is suitable for analytical tasks in OLAP scenarios, with more efficient aggregation query performance and better compression rate, but it also brings higher write overhead and complex maintenance costs; Redundant row storage SSTable can meet the needs of HTAP scenarios to select better execution plans, but it will also bring higher maintenance costs and storage overhead; At the same time, the amount of data, the type and frequency of database tasks will bring challenges to the storage mode of the database system.

[0050] The state of the managed data can be used to characterize its organizational form (or storage form), such as pure row storage form, pure column storage form, redundant row storage form, etc. Since the row storage table can be regarded as a column group containing all columns, it can be regarded as a special type of column storage table. Based on this, the current state of the managed data can be obtained according to the following two examples. Optional Example 1

[0051] In this example, the managed data is divided into at least two definitions (Schema), namely, redundant row storage and column storage table and pure column storage and column storage table; in the case where the definition of the managed data is the redundant row storage and column storage table, the managed data can be divided into redundant row storage mode and pure row storage mode under redundant row storage; in the case where the definition of the managed data is the pure column storage and column storage table, the mode of the managed data can be divided into pure column storage mode and pure row storage mode under pure column storage. Please refer to the following Table 1, which takes the highest-level SSTable as an example to exemplify the status of the above-mentioned managed data. Table 1: Status table of the highest-level SSTable

[0052] In this example, this step can sequentially obtain the definition and schema of the managed data to obtain the current state of the managed data. Optional Example 2

[0053] In this example, the managed data is divided into at least one of the following modes: a redundant row storage mode, a pure row storage mode under redundant row storage, a pure column storage mode, and a pure row storage mode under pure column storage. The managed data mode in this example may be the same as that in Optional Example 1, i.e., this example may be the result of Optional Example 1 omitting the definition of this data division level.

[0054] In this example, this step may sequentially obtain the modes of the managed data to obtain the current state of the managed data.

[0055] In step S102, a target merging type of the managed data is determined according to the current state of the managed data and at least one indicator of the managed data.

[0056] Optionally, based on at least one indicator of the managed data, a target merge type is determined from candidate merge types corresponding to the current state of the managed data. The candidate merge type refers to the type of merge operation, and the merge operation can adjust the state of the managed data. It should be understood that the merge operation can convert SStable modes under the same definition to each other, and the conversion of SSTable mode does not change the definition of SSTabel, but only dynamically adjusts the storage type of the highest-level SSTable, shielding the storage details from the upper layer, and realizing imperceptible conversion.

[0057] For example, candidate merge types may include basic column storage merge, row storage oriented merge and reconstruction column storage merge. The state of the managed data remains unchanged before and after the basic column storage merge; the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row storage oriented merge; the managed data is in the redundant row storage mode or the pure column storage mode after the reconstruction column storage merge. Please refer to the following Table 2, which takes the highest level SSTable as an example to exemplify the merge types of the above-mentioned managed data. Table 2: Merge types of the highest level SSTable

[0058] In one embodiment, based on the optional example 1 in the above step S101, this step can be performed in the following manner.

[0059] When the definition of the managed data is the same as the schema of the managed data, a target merge type is determined in the basic column-based merge and the row-based merge based on at least one indicator of the managed data. For example, in response to at least one indicator of the managed data satisfying a de-columnization condition, the row-based merge is determined as the target merge type; in response to at least one indicator of the managed data not satisfying the de-columnization condition, the basic column-based merge is determined as the target merge type.

[0060] Among them, when the definition and mode of the managed data are the same, part or all of the managed data is a regular column group table (i.e., full-column row storage SSTable and regular column group SSTable in redundant row storage mode, regular column group SSTable in pure column storage mode); therefore, it is necessary to use the de-column storage condition to determine whether the regular column group table is to be retained. If it is not to be retained, the row-oriented storage merge is determined as the target merge type, or if it is to be retained, the basic column storage merge is determined as the target merge type.

[0061] When the definition of the managed data differs from the schema of the managed data, a target merge type is determined in the row-oriented merge and the column-reconstructed merge based on at least one indicator of the managed data. For example, in response to at least one indicator of the managed data satisfying a column-reconstructed merge condition, the column-reconstructed merge is determined as the target merge type; in response to at least one indicator of the managed data not satisfying the column-reconstructed merge condition, the row-oriented merge is determined as the target merge type.

[0062] Among them, when the definition and mode of the managed data are different, the managed data is a full-column row storage table (that is, a full-column row storage SSTable in a pure row storage mode under redundant row storage, and a full-column row storage SSTable in a pure row storage mode under pure column storage); therefore, it is necessary to use the column reconstruction condition to determine whether the regular column group table needs to be rebuilt. If reconstruction is required, the rebuilt column storage merge will be determined as the target merge type. If reconstruction is not required, the row-oriented merge will be determined as the target merge type.

[0063] The above-mentioned column removal and column reconstruction conditions are used to measure at least one metric of the managed data to determine the target merge type. The at least one metric of the managed data includes at least one of the following: number of rows, number of columns, data volume, and applicable query method. The following describes the column removal and column reconstruction conditions in detail.

[0064] Each indicator of the managed data has a reconstruction threshold and a delisting threshold, wherein the delisting threshold of each indicator is greater than the reconstruction threshold of the indicator. For example, at least one indicator of the managed data includes: a1, a2, ..., a n ; then With reconstruction column storage threshold aicol and the decolumning threshold a irow , where 0 < a icol <a irow <1, when a i <a icol When the performance of the highest level SSTable in column storage is better than that of the highest level SSTable in row storage, i >a irow When , the performance of the top-level SSTable in row storage is better than that of the top-level SSTable in column storage.

[0065] Exemplarily, the de-listing condition includes: a first number of indicators is greater than a de-listing threshold corresponding to the indicator; that is, there are at least P (P is an integer greater than 0) that meet a i >a irow a i ; Wherein, the first number P of indicators can be all or part of the indicators, such as 1 indicator, or each indicator. The reconstruction column storage condition includes: the second number of indicators is less than the reconstruction column storage threshold corresponding to the indicator; that is, there are at least Q (Q is an integer greater than 0) indicators that meet a i <a icol a i ; Among them, the second number Q of indicators can be all or part of the indicators, such as 1 indicator, or each indicator.

[0066] In another exemplary embodiment, the de-columning condition includes: at least a third number of indicators is greater than the de-columning threshold corresponding to the indicator, and at least a first proportion of indicators among all indicators is not less than the re-columning threshold corresponding to the indicator; that is, there are at least X (X is an integer greater than 0) indicators that meet a i >a irow a i , and among all the indicators, at least α ratio of indicators satisfy a i ≥a icol ; The third number of indicators X can be all or part of the indicators, such as 1 indicator, or each indicator; the first ratio α is a ratio greater than 0 and less than 1. The re-listing condition includes: at least the fourth number of indicators is less than the corresponding re-listing threshold of the indicator, and at least the second ratio of indicators among all indicators is not greater than the corresponding de-listing threshold of the indicator; that is, there are at least Y (Y is an integer greater than 0) indicators that meet a i <a icol a i , and among all the indicators, at least one indicator with a ratio of β satisfies a i ≤a irow; Among them, the indicators of the fourth quantity Y can be all or part of the indicators, such as 1 indicator, or each indicator, and the second ratio β is a ratio greater than 0 and less than 1.

[0067] In the above two examples, each indicator of the managed data uses two thresholds, the decolumn storage threshold and the recolumn storage threshold, to determine the target merge type. This can more accurately determine the target merge type. In addition, the dual-threshold setting can avoid frequent changes in the merge type when the indicator fluctuates around the threshold, thereby improving the stability of the target merge type.

[0068] In another embodiment, based on the optional example 2 in the above step S101, this step can be performed in the following manner.

[0069] When the managed data is in a redundant row storage mode or a pure column storage mode, a target merge type is determined in the basic column storage merge and the row storage-oriented merge based on at least one indicator of the managed data. For example, in response to at least one indicator of the managed data satisfying a de-column storage condition, the row storage-oriented merge is determined as the target merge type; in response to at least one indicator of the managed data not satisfying the de-column storage condition, the basic column storage merge is determined as the target merge type.

[0070] Among them, when the mode of the managed data is redundant row storage mode or pure column storage mode, part or all of the managed data is a regular column group table (that is, full-column row storage SSTable and regular column group SSTable in redundant row storage mode, regular column group SSTable in pure column storage mode); therefore, it is necessary to use the de-column storage condition to determine whether the regular column group table is to be retained. If it is not to be retained, the row-oriented storage merge is determined as the target merge type, or if it is to be retained, the basic column storage merge is determined as the target merge type.

[0071] When the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage, a target merge type is determined in the row storage-oriented merge and the reconstructed column storage merge based on at least one indicator of the managed data. For example, in response to at least one indicator of the managed data satisfying a reconstructed column storage condition, the reconstructed column storage merge is determined as the target merge type; in response to at least one indicator of the managed data not satisfying the reconstructed column storage condition, the row storage-oriented merge is determined as the target merge type.

[0072] Among them, when the mode of the managed data is a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage, the managed data is a full-column row storage table (i.e., a full-column row storage SSTable in a pure row storage mode under redundant row storage, a full-column row storage SSTable in a pure row storage mode under pure column storage); therefore, it is necessary to use the column reconstruction condition to determine whether the conventional column group table needs to be rebuilt. If reconstruction is required, the rebuilt column storage merge will be determined as the target merge type. If reconstruction is not required, the row-oriented merge will be determined as the target merge type.

[0073] The above-mentioned column removal and column reconstruction conditions are used to measure at least one metric of the managed data to determine the target merge type. The at least one metric of the managed data includes at least one of the following: number of rows, number of columns, data volume, and applicable query method. The following describes the column removal and column reconstruction conditions in detail.

[0074] Each indicator of the managed data has a reconstruction threshold and a delisting threshold, wherein the delisting threshold of each indicator is greater than the reconstruction threshold of the indicator. For example, at least one indicator of the managed data includes: a1, a2, ..., a n ; then With reconstruction column storage threshold a icol and the decolumning threshold a irow , where 0 < a icol <a irow <1, when a i <a icol When the performance of the highest level SSTable in column storage is better than that of the highest level SSTable in row storage, i >a irow When , the performance of the top-level SSTable in row storage is better than that of the top-level SSTable in column storage.

[0075] Exemplarily, the de-listing condition includes: a first number of indicators is greater than a de-listing threshold corresponding to the indicator; that is, there are at least P (P is an integer greater than 0) that meet a i >a irow a i ; Wherein, the first number P of indicators can be all or part of the indicators, such as 1 indicator, or each indicator. The reconstruction column storage condition includes: the second number of indicators is less than the reconstruction column storage threshold corresponding to the indicator; that is, there are at least Q (Q is an integer greater than 0) indicators that meet a i <a icol a i ; Among them, the second number Q of indicators can be all or part of the indicators, such as 1 indicator, or each indicator.

[0076] In another exemplary embodiment, the de-columning condition includes: at least a third number of indicators is greater than the de-columning threshold corresponding to the indicator, and at least a first proportion of indicators among all indicators is not less than the re-columning threshold corresponding to the indicator; that is, there are at least X (X is an integer greater than 0) indicators that meet a i >a irow a i , and among all the indicators, at least α ratio of indicators satisfy a i ≥a icol ; The third number of indicators X can be all or part of the indicators, such as 1 indicator, or each indicator; the first ratio α is a ratio greater than 0 and less than 1. The re-listing condition includes: at least the fourth number of indicators is less than the corresponding re-listing threshold of the indicator, and at least the second ratio of indicators among all indicators is not greater than the corresponding de-listing threshold of the indicator; that is, there are at least Y (Y is an integer greater than 0) indicators that meet a i <a icol a i , and among all the indicators, at least one indicator with a ratio of β satisfies a i ≤a irow ; Among them, the indicators of the fourth quantity Y can be all or part of the indicators, such as 1 indicator, or each indicator, and the second ratio β is a ratio greater than 0 and less than 1.

[0077] In the above two examples, each indicator of the managed data uses two thresholds, the decolumn storage threshold and the recolumn storage threshold, to determine the target merge type. This can more accurately determine the target merge type. In addition, the dual-threshold setting can avoid frequent changes in the merge type when the indicator fluctuates around the threshold, thereby improving the stability of the target merge type.

[0078] It should be understood that if the managed data is stored in different nodes within a distributed database, the method can also synchronize the target merge type to each node where the managed data is located, so that each node where the managed data is located merges the managed data in step S103.

[0079] In step S103, the managed data are merged according to the target merging type, so that the managed data is in the target state.

[0080] Optionally, when the target merge type is a column-based merge, a separate merge operation can be performed on each top-level SSTable without affecting each other. Specifically, the regular column group SSTable can be left unchanged, or both the regular column group SSTable and the full column row-based SSTable can be left unchanged.

[0081] Optionally, when the target merge type is a row-oriented merge, only a single SSTable is selected for merge writing, and the remaining SSTables are directly recycled. The full-column row data required for the merge can be read from the redundant row-oriented SSTable or the combination of each column group SSTable. Specifically, if the current state contains a regular column group SSTabel, it will be merged into a full-column row-oriented SSTable; if the current state contains a regular column group SSTabel and a full-column row-oriented SSTable, only the full-column row-oriented SSTable will be retained.

[0082] Optionally, when the target merge type is a column-reconstructed merge, multiple parallel merge tasks are started to project the row-stored data separately and write them into each column group SSTable. Specifically, if the current state is full-column row-store mode under redundant row-store, the full-column row-store SSTable can be retained, and each column group SSTable can be rebuilt based on the full-column row-store SSTable; if the current state is full-column row-store mode under pure column-store, each column group SSTable can be rebuilt based on the full-column row-store SSTable, and the full-column row-store SSTable can be deleted.

[0083] It should be understood that if the managed data is stored in different nodes in a distributed database, the method further includes: synchronizing the target merge type to each node where the managed data is located.

[0084] It should be understood that this method can be initiated by the user or automatically scheduled by the database management system. For example, the method can be scheduled periodically by the database management system or in response to preset conditions. This method is often initiated or scheduled when the managed data needs to be sorted due to incremental data. In this case, the current state obtained in step S101 is obtained based on the baseline data of the managed data, and the at least one indicator of the managed data based on step S102 is an indicator of the sum of the baseline data and the incremental data.

[0085] It should also be understood that the method may also determine a data query strategy based on the current state of the managed data in response to receiving a data query request for the managed data, and execute a data query for the managed data according to the data query strategy.

[0086] Next, with reference to FIG2 , the process of the method obtained by combining the above-mentioned multiple embodiments will be described in detail.

[0087] First, when a user initiates a merge or a merge is scheduled, the schema (the definition of the managed data) is determined. If the schema is a redundant row- and column-level table, the current baseline SSTable mode is determined. If the schema is a pure column-level table, the current baseline SSTable mode is also determined.

[0088] Next, if the schema is a redundant row-store and column-store table, and the baseline SSTable mode is a pure row-store mode under redundant row storage, determine whether to schedule a column-store merge reconstruction. If so, schedule a column-store merge reconstruction to merge the managed data. Otherwise, schedule a row-store merge to merge the managed data.

[0089] Next, if the schema is a redundant row-store and column-store table, and the baseline SSTabel mode is a redundant row-store mode (that is, consistent with the schema), determine whether to schedule a row-oriented merge. If so, schedule a row-oriented merge to merge the managed data. Otherwise, schedule a basic column-store merge to merge the managed data.

[0090] Next, if the schema is a pure column-based table, and the baseline SSTable mode is a pure row-based mode under pure column-based mode, determine whether to schedule a column-based merge reconstruction. If so, schedule a column-based merge reconstruction to merge the managed data. Otherwise, schedule a row-based merge to merge the managed data.

[0091] Next, if the schema is a pure column-based table and the baseline SSTable mode is a pure column-based mode (that is, consistent with the schema), determine whether to schedule a row-based merge. If so, schedule a row-based merge to merge the managed data. Otherwise, schedule a basic column-based merge to merge the managed data.

[0092] Please refer to FIG. 3A and FIG. 3B , which illustrate the relationship between managed data and merge types in the method.

[0093] FIG4 is a schematic structural diagram of a device provided by an exemplary embodiment. Referring to FIG4 , at the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0094] Referring to Figure 5 , a data management device can be applied to the device shown in Figure 4 to implement the technical solution of this specification. The data management device may include: an acquisition module 501 for acquiring the current state of managed data; a determination module 502 for determining a target merge type for the managed data based on the current state of the managed data and at least one indicator of the managed data; and a merge module 503 for merging the managed data according to the target merge type.

[0095] In one embodiment of the present specification, the determining module is configured to determine a target merging type from candidate merging types corresponding to a current state of the managed data according to at least one indicator of the managed data.

[0096] In one embodiment of the present specification, the acquisition module is used to: acquire the definition and mode of the managed data, wherein the definition of the managed data includes a redundant row storage and column storage table and a pure column storage and column storage table. In the case where the definition of the managed data is the redundant row storage and column storage table, the mode of the managed data includes a redundant row storage mode and a pure row storage mode under redundant row storage; in the case where the definition of the managed data is the pure column storage and column storage table, the mode of the managed data includes a pure column storage mode and a pure row storage mode under pure column storage.

[0097] In one embodiment of the present specification, the determination module is used to determine the target merge type from the candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data, and is used to: when the definition of the managed data is the same as the mode of the managed data, determine the target merge type in the basic column storage merge and the row storage oriented merge based on at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row storage oriented merge; when the definition of the managed data is different from the mode of the managed data, determine the target merge type in the row storage oriented merge and the reconstructed column storage merge based on at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

[0098] In one embodiment of the present specification, the acquisition module is used to: obtain the mode of the managed data, wherein the mode of the managed data includes a redundant row storage mode, a pure row storage mode under redundant row storage, a pure column storage mode, and a pure row storage mode under pure column storage.

[0099] In one embodiment of the present specification, the determination module is used to determine the target merge type from the candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data, and is used to: when the mode of the managed data is a redundant row storage mode or a pure column storage mode, determine the target merge type in the basic column storage merge and the row-oriented merge based on at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row-oriented merge; when the mode of the managed data is a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage, determine the target merge type in the row-oriented merge and the reconstructed column storage merge based on at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

[0100] In one embodiment of the present specification, the determination module is used to determine the target merge type in the basic column storage merge and the row storage merge based on at least one indicator of the managed data, and is used to: in response to at least one indicator of the managed data satisfying the de-column storage condition, determine the row-oriented merge as the target merge type; in response to at least one indicator of the managed data not satisfying the de-column storage condition, determine the basic column storage merge as the target merge type.

[0101] In one embodiment of the present specification, the determination module is used to determine the target merge type in the row-oriented merge and the reconstructed column-oriented merge based on at least one indicator of the managed data, and is used to: in response to at least one indicator of the managed data satisfying the reconstructed column-oriented merge condition, determine the reconstructed column-oriented merge as the target merge type; in response to at least one indicator of the managed data not satisfying the reconstructed column-oriented merge condition, determine the row-oriented merge as the target merge type.

[0102] In one embodiment of the present specification, the de-columning condition includes at least a first number of indicators being greater than the de-columning threshold corresponding to the indicator; the re-columning condition includes at least a second number of indicators being less than the re-columning threshold corresponding to the indicator; wherein the de-columning threshold of each indicator is greater than the re-columning threshold of the indicator.

[0103] In one embodiment of the present specification, the de-columning condition includes that at least a third number of indicators is greater than the de-columning threshold corresponding to the indicator, and at least a first proportion of indicators among all indicators is not less than the reconstruction column storage threshold corresponding to the indicator; the reconstruction column storage condition includes that at least a fourth number of indicators is less than the reconstruction column storage threshold corresponding to the indicator, and at least a second proportion of indicators among all indicators is not greater than the de-columning threshold corresponding to the indicator; wherein the de-columning threshold of each indicator is greater than the reconstruction column storage threshold of the indicator.

[0104] In one embodiment of the present specification, the managed data is stored in different nodes in a distributed database; the device further includes a synchronization module for synchronizing the target merge type to each node where the managed data is located.

[0105] In one embodiment of the present specification, the device further includes a query module for: in response to receiving a data query request for the managed data, determining a data query strategy according to the current state of the managed data, and executing a data query for the managed data according to the data query strategy.

[0106] In one embodiment of the present specification, the at least one indicator of the managed data includes at least one of the following: number of rows, number of columns, amount of data, and applicable query method.

[0107] One or more embodiments of this specification also propose a computer program product, including a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0108] One or more embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0109] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0110] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0112] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0114] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0115] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0116] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0117] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0118] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A data management method, comprising: Get the current status of the managed data; determining a target merging type of the managed data according to a current state of the managed data and at least one indicator of the managed data; The managed data are merged according to the target merging type so that the managed data is in a target state.

2. The data management method according to claim 1, wherein determining the target merging type of the managed data based on the current state of the managed data and at least one indicator of the managed data comprises: According to at least one indicator of the managed data, a target merging type is determined from candidate merging types corresponding to a current state of the managed data.

3. The data management method according to claim 2, wherein obtaining the current status of the managed data comprises: Obtain the definition and mode of the managed data, wherein the definition of the managed data includes a redundant row storage and column storage table and a pure column storage and column storage table. When the definition of the managed data is the redundant row storage and column storage table, the mode of the managed data includes a redundant row storage mode and a pure row storage mode under redundant row storage; when the definition of the managed data is the pure column storage and column storage table, the mode of the managed data includes a pure column storage mode and a pure row storage mode under pure column storage.

4. The data management method according to claim 3, wherein determining the target merge type from candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data comprises: In a case where the definition of the managed data is the same as the mode of the managed data, determining a target merge type in a basic column storage merge and a row storage-oriented merge according to at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data, after the row storage-oriented merge, is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage; In a case where the definition of the managed data is different from the mode of the managed data, the target merge type is determined in the row-oriented storage merge and the reconstructed column storage merge according to at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

5. The data management method according to claim 2, wherein obtaining the current status of the managed data comprises: A mode of the managed data is obtained, wherein the modes of the managed data include a redundant row storage mode, a pure row storage mode under redundant row storage, a pure column storage mode, and a pure row storage mode under pure column storage.

6. The data management method according to claim 5, wherein determining the target merge type from candidate merge types corresponding to the current state of the managed data based on at least one indicator of the managed data comprises: In a case where the mode of the managed data is a redundant row storage mode or a pure column storage mode, determining a target merge type in a basic column storage merge and a row storage-oriented merge according to at least one indicator of the managed data, wherein the state of the managed data remains unchanged before and after the basic column storage merge, and the managed data is in a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage after the row storage-oriented merge; In the case where the mode of the managed data is a pure row storage mode under redundant row storage or a pure row storage mode under pure column storage, the target merge type is determined in the row storage oriented merge and the reconstructed column storage merge according to at least one indicator of the managed data, wherein the managed data is in the redundant row storage mode or the pure column storage mode after the reconstructed column storage merge.

7. The data management method according to claim 4 or 6, wherein determining a target merge type in basic column-based merge and row-oriented merge based on at least one indicator of the managed data comprises: In response to at least one indicator of the managed data satisfying a column-free storage condition, determining the row-oriented storage merge as the target merge type; In response to at least one indicator of the managed data not satisfying a de-columnization condition, the basic column storage merger is determined as the target merger type.

8. The data management method according to claim 7, wherein determining a target merge type in row-oriented merge and column-oriented merge based on at least one indicator of the managed data comprises: In response to at least one indicator of the managed data satisfying a column store reconstruction condition, determining the column store reconstruction merge as the target merge type; In response to at least one indicator of the managed data not satisfying a column storage reconstruction condition, the row storage oriented merge is determined as the target merge type.

9. The data management method according to claim 8, wherein the de-columning condition comprises: At least a first number of indicators is greater than a de-columning threshold corresponding to the indicators; The column storage reconstruction condition includes: at least a second number of indicators is less than a column storage reconstruction threshold corresponding to the indicator; The de-column storage threshold of each indicator is greater than the re-column storage threshold of the indicator.

10. The data management method according to claim 8, wherein the de-columning condition comprises: At least a third number of indicators is greater than a de-column storage threshold corresponding to the indicator, and at least a first proportion of indicators among all indicators is not less than a re-column storage threshold corresponding to the indicator; The column storage reconstruction condition includes: at least a fourth number of indicators is less than a column storage reconstruction threshold corresponding to the indicator, and at least a second proportion of indicators among all indicators is not greater than a column storage removal threshold corresponding to the indicator; The de-column storage threshold of each indicator is greater than the re-column storage threshold of the indicator.

11. The data management method according to claim 1, wherein the managed data is stored in different nodes in a distributed database; the method further comprises: The target merge type is synchronized to each node where the managed data is located.

12. The data management method according to claim 1, further comprising: In response to receiving a data query request for the managed data, a data query strategy is determined according to a current state of the managed data, and a data query is performed on the managed data according to the data query strategy.

13. The data management method according to claim 1, wherein the at least one indicator of the managed data includes at least one of the following: number of rows, number of columns, data volume, and applicable query method.

14. A data management device comprising: The acquisition module is used to obtain the current status of the managed data; a determination module, configured to determine a target merging type of the managed data according to a current state of the managed data and at least one indicator of the managed data; The merging module is used to merge the managed data according to the target merging type.

15. A computer program product comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

16. An electronic device comprising: processor; as well as a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 13 by running the executable instructions.

17. A computer-readable storage medium having computer instructions stored thereon, which implement the steps of the method according to any one of claims 1 to 13 when executed by a processor.

Images