Data completion method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of databases, and specifically provides a data completion method and device, electronic equipment and a storage medium. The data completion method comprises the following steps: dividing a target table into regions according to a primary key column in the target table to be processed, and obtaining at least one target region; determining the number of rows in each target region; determining a starting row offset value of each target region according to the number of rows corresponding to the at least one target region; and writing, according to the starting row offset value of each target region, at least one target column combination in the target table into a corresponding baseline ordering string table according to the data corresponding to the target column combination in each target region. In this way, corresponding column indexes do not need to be created for all columns, and the data in each column does not need to be sorted through the column indexes, so that the cumbersome steps of data completion are simplified, and the system resources and time cost are reduced.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a method, apparatus, electronic device, and storage medium for data completion. Background Technology

[0002] In practical applications, to support different types of business simultaneously, several columns in a table are typically combined to obtain a column group (CG) based on business requirements. A single table can have one or more CGs. When executing Data Definition Language (DDL) on a table, it is usually necessary to update multiple CGs. Specifically, the updated data for each CG is written to the corresponding Sorted String Table (SSTable); this process is called data completion.

[0003] In current technology, column indexes are typically created for each column in the table so that data completion for CG can be performed based on the column indexes.

[0004] However, this approach requires creating column indexes for all columns, which consumes a significant amount of system resources. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for data completion, which can reduce the system resources consumed when performing data completion.

[0006] On the one hand, this application provides a data completion method, including:

[0007] Based on the primary key column of the target table to be processed, the target table is divided into regions to obtain at least one target region;

[0008] Determine the number of rows for each target region;

[0009] Based on the number of rows corresponding to at least one target region, determine the starting row offset value for each target region; the starting row offset value is used to locate the target region.

[0010] Based on the starting row offset value of each target region, the data corresponding to at least one target column combination in the target table in each target region are written into the baseline sorting string table corresponding to the respective target column combination.

[0011] In one implementation, the target table is divided into regions based on the primary key column to obtain at least one target region, including:

[0012] The data in the primary key column is sampled, and the data in the primary key column is sorted sequentially;

[0013] Compare adjacent sample values ​​in each sample value to obtain the corresponding comparison results;

[0014] Based on the comparison results, at least one data partition row is determined;

[0015] The target table is divided into at least one target region by dividing at least one data row.

[0016] In one implementation, determining the starting row offset value for each target region based on the row number corresponding to at least one target region includes:

[0017] For each target region, in descending order of its position in the target table, perform the following steps:

[0018] If the target area is the first area, then the starting row offset of the target area is determined to be the specified starting value;

[0019] If the target area is not the first area, the starting row offset of the target area is determined based on the row numbers of all the target areas preceding the target area and the specified starting value.

[0020] In one implementation, based on the starting row offset value of each target region, the data corresponding to at least one target column combination in the target table in each target region are written into the baseline sorting string table corresponding to the respective target column combination, including:

[0021] For each target region, execute the following steps using the thread corresponding to that target region:

[0022] The number of combinations of at least one target column is determined as the initial batch size;

[0023] Based on the batch size and the starting row offset of the target region, at least one target column is combined with the corresponding data in the target region and written into the corresponding baseline sort string table.

[0024] If it is determined that the data write failed, then obtain the latest number of successful writes for the target column combination.

[0025] Determine the new batch size based on the batch size and the latest successful quantity;

[0026] Based on the new batch size, perform data write operations on the target column combination for which data writing was unsuccessful.

[0027] In one implementation, based on a new batch size, a data write operation is performed on the target column combination for which data writing failed, including:

[0028] Repeat the following steps until it is confirmed that the data was written successfully and there are no target column combinations that failed to write data:

[0029] Based on the new batch size and the starting row offset of the target region, the remaining target columns are combined with the data in the target region and written to the corresponding baseline sort string table; the remaining target column combination is the target column combination of the data that was not successfully written at the moment.

[0030] If it is determined that the data write failed, obtain the latest number of successful writes for the target column combination, and determine a new batch size based on the batch size and the latest number of successful writes.

[0031] If it is determined that the data was written successfully, and there are target column combinations where data was not written successfully, then update the new batch size based on the set ratio;

[0032] If it is determined that the data was written successfully and there is no target column combination where data was not written successfully, then the data completion process is considered complete.

[0033] In one implementation, determining a new batch size based on the batch size and the latest successful quantity includes:

[0034] Determine the product of the batch size and the set weight;

[0035] The maximum value between the product and the latest successful quantity is used to determine the new batch size.

[0036] In one embodiment, the method further includes:

[0037] If it is determined that the data writing failed, then obtain the sum of the batch sizes corresponding to all threads;

[0038] If the total number of threads is not higher than the number of threads, then it is determined that memory resources are insufficient, and the data completion process ends.

[0039] On one hand, embodiments of this application provide a data completion apparatus, including:

[0040] The partitioning unit is used to divide the target table into regions based on the primary key column of the target table to obtain at least one target region.

[0041] A determination unit is used to determine the number of rows in each target region;

[0042] The positioning unit is used to determine the starting row offset value of each target area based on the number of rows corresponding to at least one target area; the starting row offset value is used to locate the target area.

[0043] The write unit is used to write the data corresponding to at least one target column combination in the target table in each target area into the baseline sorting string table corresponding to the respective target column combination, based on the starting row offset value of each target area.

[0044] In one implementation, the partitioning unit is used for:

[0045] The data in the primary key column is sampled, and the data in the primary key column is sorted sequentially;

[0046] Compare adjacent sample values ​​in each sample value to obtain the corresponding comparison results;

[0047] Based on the comparison results, at least one data partition row is determined;

[0048] The target table is divided into at least one target region by dividing at least one data row.

[0049] In one embodiment, the positioning unit is used for:

[0050] For each target region, in descending order of its position in the target table, perform the following steps:

[0051] If the target area is the first area, then the starting row offset of the target area is determined to be the specified starting value;

[0052] If the target area is not the first area, the starting row offset of the target area is determined based on the row numbers of all the target areas preceding the target area and the specified starting value.

[0053] In one implementation, the writing unit is used for:

[0054] For each target region, execute the following steps using the thread corresponding to that target region:

[0055] The number of combinations of at least one target column is determined as the initial batch size;

[0056] Based on the batch size and the starting row offset of the target region, at least one target column is combined with the corresponding data in the target region and written into the corresponding baseline sort string table.

[0057] If it is determined that the data write failed, then obtain the latest number of successful writes for the target column combination.

[0058] Determine the new batch size based on the batch size and the latest successful quantity;

[0059] Based on the new batch size, perform data write operations on the target column combination for which data writing was unsuccessful.

[0060] In one implementation, the writing unit is configured to: repeatedly execute the following steps until it is determined that the data writing was successful and there are no target column combinations where data writing was unsuccessful:

[0061] Based on the new batch size and the starting row offset of the target region, the remaining target columns are combined with the data in the target region and written to the corresponding baseline sort string table; the remaining target column combination is the target column combination of the data that was not successfully written at the moment.

[0062] If it is determined that the data write failed, obtain the latest number of successful writes for the target column combination, and determine a new batch size based on the batch size and the latest number of successful writes.

[0063] If it is determined that the data was written successfully, and there are target column combinations where data was not written successfully, then update the new batch size based on the set ratio;

[0064] If it is determined that the data was written successfully and there is no target column combination where data was not written successfully, then the data completion process is considered complete.

[0065] In one implementation, the writing unit is used to: determine the product of the batch size and the set weight;

[0066] The maximum value between the product and the latest successful quantity is used to determine the new batch size.

[0067] In one embodiment, the writing unit is further configured to:

[0068] If it is determined that the data writing failed, then obtain the sum of the batch sizes corresponding to all threads;

[0069] If the total number of threads is not higher than the number of threads, then it is determined that memory resources are insufficient, and the data completion process ends.

[0070] On one hand, embodiments of this application provide an electronic device, including:

[0071] Processor; and

[0072] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in the various alternative implementations of any of the above-described data completion methods.

[0073] On one hand, embodiments of this application provide a storage medium storing computer instructions for causing a computer to perform the steps of the method provided in any of the optional implementations of the data completion described above.

[0074] The data completion method, apparatus, electronic device, and storage medium provided in this application embodiment divide the target table into regions based on the primary key column to obtain at least one target region; determine the number of rows in each target region; determine the starting row offset value for each target region based on the number of rows corresponding to the at least one target region; the starting row offset value is used to locate the target region; and based on the starting row offset value of each target region, write the data corresponding to at least one target column combination in the target table in each target region into the baseline sorting string table corresponding to the respective target column combination. In this way, by dividing the table into regions using the primary key column and locating the data in each region using the starting row offset, it is not necessary to create corresponding column indexes for all columns or sort the data in each column using column indexes, simplifying the cumbersome steps of data completion and reducing the consumption of system resources and time costs. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0076] Figure 1 This is a flowchart of a data completion method according to an embodiment of this application.

[0077] Figure 2 This is an example diagram illustrating how to determine the starting row offset value in an embodiment of this application.

[0078] Figure 3 This is an example diagram of a method for adaptive adjustment of batch number in an embodiment of this application.

[0079] Figure 4 This is a structural block diagram of the data completion device in the embodiments of this application.

[0080] Figure 5 This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0081] The technical solution of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0082] First, some of the terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0083] Terminal devices can be mobile terminals, fixed terminals, or portable terminals, such as mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system devices, personal navigation devices, personal digital assistants, audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that terminal devices can support any type of user-facing interface (e.g., wearable devices).

[0084] Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0085] OceanBase (OB) is a distributed relational database that is continuously available and features scalability and high performance.

[0086] Online Transaction Processing (OLTP): Generally refers to transaction operations with a small number of rows and short processing time, such as bank transactions.

[0087] Online Analytical Processing (OLAP) generally refers to complex data analysis with a large number of rows and long processing time, such as report statistics.

[0088] Log Structured Merge Tree (LSM Tree): A data storage structure consisting of multiple independent data structures.

[0089] Baseline SStable: is a data structure in an LSM Tree that is persisted on disk and consists of multiple macroblocks.

[0090] Macroblock: Contains multiple rows of sorted data, persisted on disk. The macroblock size can be fixed at 2M and consists of multiple microblocks.

[0091] Micro-blocks are the basic building blocks of macroblocks, containing multiple rows of sorted data. They are persisted on disk and their size is not fixed, usually a few KB. They are the smallest unit for reading SSTable data from disk.

[0092] The technical concept of this application will be explained below.

[0093] In some database scenarios, in order to support different types of business simultaneously (such as OLTP business and OLAP business), some databases (such as OB database) have developed columnar storage table functionality, which allows users to specify several CGs in the same table.

[0094] For example, the following statement can be used to set a column group (CG): `create table t1(c1 int primary key, c2 varchar(256), c3 int)with column group(all columns, each column)`. This creates table 1 with CGs. In terms of data organization at the storage layer, table 1 has four CGs: (c1, c2, c3), (c1), (c2), and (c3). Each CG's data is organized into an SSTable. That is, the data in each CG is actually stored in the corresponding SSTable.

[0095] When performing DDL operations on a table, it is often necessary to complete the data for multiple column classes (CGs). For example, when executing `alter table t1 modify column c2 int` on table 1 above, data completion is required for the CGs (c1, c2, c3) and c2.

[0096] In traditional techniques, column indexes are typically created for all columns when a table is created. If no column indexes are created when a table is created, they are created through a build process when needed.

[0097] However, creating column indexes for each column separately and sorting them by column data would consume a lot of system resources. Furthermore, when the BUILD process creates column indexes one by one, it needs to scan the main table data multiple times, which will cause a lot of input output (IO) overhead and the build time will be relatively long.

[0098] In view of the deficiencies of the above-mentioned related technologies, the present application provides a method, apparatus, electronic device and storage medium for data completion, which aims to reduce the system resources consumed when performing data completion.

[0099] This application provides a data completion method applicable to electronic devices. This application does not limit the type of electronic device; it can be any suitable device type, such as a terminal device or a server, which will not be elaborated further. In one application scenario, this embodiment can be applied to an OB database. When performing DDL on a target table, the target column combinations that need data reorganization are determined. Regions are divided based on the primary key columns in the target table. Data within each target region is located based on the offset of the starting row of each target region. Then, based on the offset corresponding to each target region, the data corresponding to each target column combination within the target region is written into the SSTable corresponding to each target column combination, thus achieving data completion.

[0100] See Figure 1 The diagram shown is a flowchart of a data completion method according to an embodiment of this application, applied to an electronic device. The following is a description of the method in conjunction with... Figure 1 The method is described below, and the specific implementation process is as follows:

[0101] Step 100: Divide the target table into regions based on the primary key column of the target table to be processed, and obtain at least one target region.

[0102] In one implementation, when performing step 100, the following steps may be adopted:

[0103] S1001: Sample the data in the primary key column to obtain multiple sample values.

[0104] In one implementation, a primary key is set for the target table, the primary key column in the target table is determined, the data in the primary key column is sorted, and when it is determined that data completion is required (DDL is performed on the target table), the data in the primary key column is sampled according to a set sampling interval to obtain multiple sampled values.

[0105] In this application, the primary key is used to uniquely identify a record (i.e., a row of data). Primary key values ​​cannot be duplicated or null. In this embodiment, the data in the primary key column is ordered sequentially. The sampling interval can be a fixed value or a non-fixed value (e.g., it can be continuously increasing). In practical applications, the sampling interval can be set according to the actual application scenario; for example, a sampling interval of 5 can be set, but this is not a limitation.

[0106] DDL (Data Definition Language) is used to define relation schemas, delete relations, modify relation schemas, and create various objects in the database. Database objects can include tables, clusters, indexes, views, functions, stored procedures, and triggers, among others.

[0107] S1002: Compare adjacent sample values ​​in each sample value to obtain the corresponding comparison results.

[0108] In one implementation, all sampled values ​​are numerical values. According to the order of the sampled values ​​in the primary key column, for each sampled value, the following steps are performed: determine the difference between the sampled value and the previous sampled value as the comparison result.

[0109] S1003: Based on the comparison results, determine at least one data partition row.

[0110] As an example, if the difference between a sampled value and the previous sampled value is higher than a set difference (e.g., 5), then the row containing that sampled value is determined as the data partition row.

[0111] In practical applications, the difference can be set according to the actual application scenario, and there are no restrictions here.

[0112] S1004: Divide the target table based on at least one data row to obtain at least one target region.

[0113] In this way, the table can be divided into multiple target areas.

[0114] Step 101: Determine the number of rows for each target region.

[0115] In one implementation, a thread is allocated to each target region, and the data from each target region is shuffled to the corresponding thread. Each thread determines the number of rows to receive in the target region. It should be noted that the number of rows in different target regions can be the same or different. For example, one target region may have 10 rows, and another target region may have 20 rows.

[0116] Optionally, each thread can correspond to one or more target regions. Threads can reside on the same device or on different devices; there is no restriction on this.

[0117] In this way, multiple threads can be used to process each target area in parallel, thereby improving data processing efficiency.

[0118] Step 102: Determine the starting row offset value for each target region based on the number of rows corresponding to at least one target region.

[0119] The starting row offset value is used to locate the target region. In one implementation, the following steps are performed for each target region, following the top-to-bottom order of at least one target region in the target table:

[0120] If the target region is the first region, then the starting row offset of the target region is determined to be the specified starting value. As an example, the specified starting value can be 0.

[0121] If the target area is not the first area, the starting row offset of the target area is determined based on the row numbers of all the target areas preceding the target area and the specified starting value.

[0122] As an example, if the starting value is 0, the number of rows in the first region of the target table is 10, the number of rows in the second target region is 15, then the starting row offset of the third target region is 0+10+15=25.

[0123] As another example, each thread counts the number of rows in its corresponding target region. A designated thread retrieves the statistical results from the other threads, determines the starting row offset value for each target region based on the statistical results of each thread, and sends the starting row offset value of each target region to the thread corresponding to the target region.

[0124] Optionally, the specified thread can be a pre-defined thread, any randomly selected thread, or the thread that finishes counting rows last. In practical applications, the specified thread can be determined according to the actual application scenario, and there are no restrictions here.

[0125] See Figure 2 The image shown is an example diagram for determining the starting row offset value. Figure 2 In the process, threads 1, 2, and 3 count the number of rows in their respective target regions (i.e., the first region, the second region, and the third region). If the number of rows in each target region is 100, then the starting row offset value of the first region is determined to be 0, the starting row offset value of the second region is 100, and the starting row offset value of the third region is 200.

[0126] Since the CG of non-primary key columns needs to identify each data row, in this embodiment of the application, the starting row offset value of each target area is determined by accumulating the number of rows in each target area. This process can be called row counting, so that each data row in each target area can be identified according to the starting row offset value of each target area, which facilitates subsequent data completion without having to create column indexes for each column separately. This simplifies tedious operations, reduces system overhead, and improves data processing efficiency.

[0127] Step 103: Based on the starting row offset value of each target region, write the data corresponding to at least one target column combination in the target table in each target region into the baseline sorting string table corresponding to the respective target column combination.

[0128] In one implementation, for each target region, the following steps are executed through the thread corresponding to that target region:

[0129] S1031: Determine the number of combinations of at least one target column as the initial batch size.

[0130] In one implementation, the number of target CGs to be processed in the target table, i.e., the number of combinations, is obtained and determined as the initial batch size, batch_size.

[0131] As an example, if the target table contains 4 CGs, then the initial batch_size is determined to be 4.

[0132] Here, CG is a combination of multiple columns in the table, and the target CG is the CG that needs to be re-integrated from among the CGs.

[0133] S1032: Based on the batch size and the starting row offset value of the target area, combine at least one target column with the corresponding data in the target area and write them into the corresponding baseline sort string table.

[0134] As an example, the target CG includes a first CG and a second CG. The first CG is column C2, and the second CG is columns C3 and C4. Therefore, the number of target CGs is 2, and the batch size is 2. The thread writes the data in column C1 of the target area to the first baseline SSTable corresponding to the first CG, and writes the data in columns C3 and C4 of the target area to the second baseline SSTable corresponding to the second CG.

[0135] S1033: If it is determined that the data write failed, then obtain the latest number of successful writes for the target column combination.

[0136] It should be noted that due to memory resource limitations, the data of each target CG may not be completely successfully written to the corresponding baseline SSTable. However, the data of some target CGs may be successfully written to the corresponding baseline SSTable. Therefore, when it is determined that the data writing has failed, the number of those target CGs (i.e., the target CGs that have successfully written data at present) is the latest successful number.

[0137] As an example, if the batch size is 4, and 3 target CGs are successfully written while one target CG fails to be written, then the latest successful count is determined to be 3.

[0138] Furthermore, if it is confirmed that the data writing was successful and the data in each target CG has been successfully written to the corresponding baseline SSTable, then it is confirmed that the data has been completed and the data completion process ends.

[0139] S1034: Determine the new batch size based on the batch size and the latest successful quantity.

[0140] In one implementation, the product of batch size and a set weight is determined; the product of batch size and set weight is determined; and the maximum value between the product and the latest successful quantity is determined as the new batch size.

[0141] The weight is set to a value range of (0,1).

[0142] As an example, the weight is set to 0.5. In practical applications, the weight can be set according to the actual application scenario, and there are no restrictions here.

[0143] In this way, if data writing fails, the batch size can be reduced by setting weights.

[0144] S1035: Based on the new batch size, perform a data write operation for the target column combination for which data writing was unsuccessful.

[0145] In one implementation, during S1035, the following steps can be executed repeatedly until it is determined that the data was successfully written and there are no target column combinations where data writing was unsuccessful:

[0146] S1035-1: Based on the new batch size and the starting row offset of the target area, combine the data of the remaining target columns in the target area and write them into the corresponding baseline sort string table. Then, execute any one of the steps S1035-2, S1035-3, and S1035-4.

[0147] The remaining target column combination is the target column combination for which data has not yet been successfully written.

[0148] As an example, the target CG includes a first CG, a second CG, and a third CG. If the second and third CGs are successfully written, but the first CG fails to write, then the first CG becomes the remaining target CG. Since the first CG only contains column C2, the data in column C2 within the target area is written to the first baseline SSTable corresponding to the first CG.

[0149] S1035-2: If it is determined that the data write failed, obtain the latest number of successful writes for the target column combination, determine the new batch size based on the batch size and the latest number of successful writes, and execute S1035-1.

[0150] Specifically, when executing S1035-3, please refer to S1033 and S1034 for the specific steps, which will not be repeated here.

[0151] S1035-3: If it is determined that the data writing was successful, and there are target column combinations where the data writing was unsuccessful, then based on the set ratio, update the new batch size and execute S1035-1.

[0152] The set ratio can be no less than 1, for example, the set ratio is 2.

[0153] As an example, if it is determined that the data write was successful, and there are target column combinations where data writes were unsuccessful, then the product of the current batch size (i.e., the new batch size) and the set ratio will be used as the new batch size. In practical applications, the set ratio can be set according to the actual application scenario, and there are no restrictions here.

[0154] Among them, the target column combination for which data writing was unsuccessful refers to the target column combination for which the corresponding data in the target area was not successfully written to the corresponding baseline SStable.

[0155] S1035-4: If it is determined that the data was written successfully and there is no target column combination where the data was not written successfully, then the data completion process is considered to have ended.

[0156] Furthermore, during the data completion process, if a data write failure is detected, the sum of the batch sizes corresponding to all threads can be obtained. If the sum is not higher than the number of threads, it is determined that memory resources are insufficient, and the data completion process is terminated.

[0157] The following example illustrates the above implementation using a single thread performing data completion on a target region. (See also...) Figure 3 The diagram shown is an example of a method for adaptively adjusting the batch size. The specific implementation process of this method may include:

[0158] Step 300: Determine the number of combinations of at least one target column as the initial batch size.

[0159] Step 301: According to the batch size, combine at least one target column into the corresponding data in the target area and write them into the corresponding baseline SStable.

[0160] Step 302: Determine if the data was written successfully. If yes, proceed to step 303; otherwise, proceed to step 306.

[0161] Step 303: Multiply the batch size by the set ratio to obtain the new batch size.

[0162] Step 304: Determine if there is a target CG for which data is to be written. If yes, proceed to step 301; otherwise, proceed to step 305.

[0163] Here, the target CG for the data to be written refers to the CG for which the corresponding data in the target area is to be written to the corresponding baseline SStable.

[0164] Step 305: End the data completion process.

[0165] Step 306: Obtain the sum of the batch sizes for all threads.

[0166] Step 307: Determine if the total number of threads is not higher than the number of threads. If yes, proceed to step 305; otherwise, proceed to step 308.

[0167] Step 308: Determine the product of batch size and set weight.

[0168] Step 309: Determine the latest number of successful writes to the target column combination.

[0169] Step 310: Determine the maximum value between the product and the latest successful quantity as the new batch size, and proceed to step 301.

[0170] Specifically, the steps 300-310 are detailed in steps 100-103 above and will not be repeated here.

[0171] It's important to note that the process of writing the data of each row of the target CG in the corresponding target region to the corresponding baseline SStable via a thread can be called rescan. Since a buffer of at least the macroblock size (e.g., 2MB) needs to be maintained, in scenarios with limited memory, multiple scans of the sorted data are required to avoid memory exhaustion. In scenarios with larger memory, the goal is to minimize the number of scans to save disk I / O overhead. The maximum number of columns in the table can be 4096. Because the memory consumption of the data completion link is difficult to accurately reserve, strategies (e.g., greedy algorithms) can be used for data completion to reduce overhead. Specifically, the number N (where N is a positive integer) of all target CGs can be used as the batch size, and rescan can be performed based on this batch size. If the data writing is successful, only one rescan is needed. If data writing fails, retrieve the latest successful count of CGs (code snippets) and the latest batch count (last_batch_size), and determine batch_size = max(last_batch_size / 2, last_succ_cg_count, 1). Further, if the sum of the batch_sizes of all threads is less than or equal to the number of threads, it indicates a severe shortage of memory resources; therefore, no further retries are made, and the process returns failure and exits.

[0172] In this embodiment, it is not necessary to create column indexes for each column separately; only a primary key is required, saving significant computational overhead. Furthermore, since the SSTable corresponding to the CG follows the row order of the primary key, rowoffset is used to locate the main table row, eliminating the need for redundant storage of the primary key value and saving storage overhead. Moreover, data completion for multiple CGs can be performed in a single data scan using a batching method, significantly reducing the construction overhead of columnar storage. The batch size can also be adaptively adjusted, further reducing the number of rescan operations, greatly reducing system overhead and time costs, and improving data processing efficiency and data completion performance.

[0173] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0174] Based on the same inventive concept, this application also provides a data completion device. Since the principle of the above-described device and apparatus in solving the problem is similar to that of a data completion method, the implementation of the above-described device can refer to the implementation of the method, and repeated details will not be described again. This device can be applied to electronic devices. This application does not limit the type of electronic device; it can be any suitable type of device, such as smartphones, tablet computers, etc., which will not be described again in this application.

[0175] See Figure 4 The diagram shown is a structural block diagram of a data completion apparatus according to an embodiment of this application. In some embodiments, the data completion apparatus exemplified in this application includes:

[0176] The partitioning unit 401 is used to partition the target table into regions based on the primary key column of the target table to be processed, so as to obtain at least one target region.

[0177] Determining unit 402 is used to determine the number of rows in each target region;

[0178] The positioning unit 403 is used to determine the starting row offset value of each target area according to the number of rows corresponding to at least one target area; the starting row offset value is used to locate the target area.

[0179] The writing unit 404 is used to write the data corresponding to at least one target column combination in the target table in each target area into the baseline sorting string table corresponding to the corresponding target column combination, according to the starting row offset value of each target area.

[0180] In one embodiment, the dividing unit 401 is used for:

[0181] The data in the primary key column is sampled, and the data in the primary key column is sorted sequentially;

[0182] Compare adjacent sample values ​​in each sample value to obtain the corresponding comparison results;

[0183] Based on the comparison results, at least one data partition row is determined;

[0184] The target table is divided into at least one target region by dividing at least one data row.

[0185] In one embodiment, the positioning unit 403 is used for:

[0186] For each target region, in descending order of its position in the target table, perform the following steps:

[0187] If the target area is the first area, then the starting row offset of the target area is determined to be the specified starting value;

[0188] If the target area is not the first area, the starting row offset of the target area is determined based on the row numbers of all the target areas preceding the target area and the specified starting value.

[0189] In one embodiment, the writing unit 404 is used for:

[0190] For each target region, execute the following steps using the thread corresponding to that target region:

[0191] The number of combinations of at least one target column is determined as the initial batch size;

[0192] Based on the batch size and the starting row offset of the target region, at least one target column is combined with the corresponding data in the target region and written into the corresponding baseline sort string table.

[0193] If it is determined that the data write failed, then obtain the latest number of successful writes for the target column combination.

[0194] Determine the new batch size based on the batch size and the latest successful quantity;

[0195] Based on the new batch size, perform data write operations on the target column combination for which data writing was unsuccessful.

[0196] In one implementation, the writing unit 404 is configured to: repeatedly execute the following steps until it is determined that the data writing was successful and there are no target column combinations where data writing was unsuccessful:

[0197] Based on the new batch size and the starting row offset of the target region, the remaining target columns are combined with the data in the target region and written to the corresponding baseline sort string table; the remaining target column combination is the target column combination of the data that was not successfully written at the moment.

[0198] If it is determined that the data write failed, obtain the latest number of successful writes for the target column combination, and determine a new batch size based on the batch size and the latest number of successful writes.

[0199] If it is determined that the data was written successfully, and there are target column combinations where data was not written successfully, then update the new batch size based on the set ratio;

[0200] If it is determined that the data was written successfully and there is no target column combination where data was not written successfully, then the data completion process is considered complete.

[0201] In one embodiment, the writing unit 404 is used to: determine the product of the batch size and the set weight;

[0202] The maximum value between the product and the latest successful quantity is used to determine the new batch size.

[0203] In one embodiment, the writing unit 404 is further configured to:

[0204] If it is determined that the data writing failed, then obtain the sum of the batch sizes corresponding to all threads;

[0205] If the total number of threads is not higher than the number of threads, then it is determined that memory resources are insufficient, and the data completion process ends.

[0206] On one hand, embodiments of this application provide an electronic device, including:

[0207] Processor; and

[0208] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in the various alternative implementations of any of the above-described data completion methods.

[0209] On one hand, embodiments of this application provide a storage medium storing computer instructions for causing a computer to perform the steps of the method provided in any of the optional implementations of the data completion described above.

[0210] The data completion method, apparatus, electronic device, and storage medium provided in this application embodiment divide the target table into regions based on the primary key column to obtain at least one target region; determine the number of rows in each target region; determine the starting row offset value for each target region based on the number of rows corresponding to the at least one target region; the starting row offset value is used to locate the target region; and based on the starting row offset value of each target region, write the data corresponding to at least one target column combination in the target table in each target region into the baseline sorting string table corresponding to the respective target column combination. In this way, by dividing the table into regions using the primary key column and locating the data in each region using the starting row offset, it is not necessary to create corresponding column indexes for all columns or sort the data in each column using column indexes, simplifying the cumbersome steps of data completion and reducing the consumption of system resources and time costs.

[0211] This application provides an electronic device, including:

[0212] Processor; and

[0213] The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.

[0214] This application provides a storage medium storing computer instructions for causing a computer to execute the methods of any of the above embodiments.

[0215] Figure 5 A schematic diagram of the structure of an electronic device 5000 is shown. (See also...) Figure 5 As shown, the electronic device 5000 includes a processor 5010 and a memory 5020, and optionally may also include a power supply 5030, a display unit 5040, and an input unit 5050.

[0216] The processor 5010 is the control center of the electronic device 5000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 5000 by running or executing software programs and / or data stored in the memory 5020.

[0217] In this embodiment, when the processor 5010 calls the computer program stored in the memory 5020, it executes the steps in the above embodiments.

[0218] Optionally, processor 5010 may include one or more processing units; preferably, processor 5010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 5010. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.

[0219] The memory 5020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 5000, etc. In addition, the memory 5020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0220] Electronic device 5000 also includes a power supply 5030 (such as a battery) that supplies power to various components. The power supply can be logically connected to processor 5010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0221] The display unit 5040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 5000. In this embodiment, it is mainly used to display the display interface of various applications in the electronic device 5000, as well as text, images, and other objects displayed on the display interface. The display unit 5040 may include a display panel 5041. The display panel 5041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0222] The input unit 5050 can be used to receive information such as numbers or characters input by the user. The input unit 5050 may include a touch panel 5051 and other input devices 5052. The touch panel 5051, also known as a touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 5051).

[0223] Specifically, the touch panel 5051 can detect user touch operations and the signals generated by these operations, convert them into touch point coordinates, send them to the processor 5010, and receive and execute commands from the processor 5010. Furthermore, the touch panel 5051 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 5052 can include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0224] Of course, the touch panel 5051 can cover the display panel 5041. When the touch panel 5051 detects a touch operation on or near it, it transmits the information to the processor 5010 to determine the type of touch event. Subsequently, the processor 5010 provides corresponding visual output on the display panel 5041 according to the type of touch event. Although in Figure 5 In this embodiment, the touch panel 5051 and the display panel 5041 are two separate components to realize the input and output functions of the electronic device 5000. However, in some embodiments, the touch panel 5051 and the display panel 5041 can be integrated to realize the input and output functions of the electronic device 5000.

[0225] The electronic device 5000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity sensor, etc. Of course, depending on the specific application, the electronic device 5000 may also include other components such as a camera. Since these components are not the focus of this application's embodiments, therefore... Figure 5It is not shown in the text and will not be described in detail here.

[0226] Those skilled in the art will understand that Figure 5 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0227] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0228] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this application.

Claims

1. A method for data completion, the method comprising: Based on the primary key column of the target table to be processed, the target table is divided into regions to obtain at least one target region; The data in the primary key column is sorted sequentially; Determine the number of rows for each target region; Based on the row number corresponding to the at least one target region and the top-to-bottom order of the at least one target region in the target table, the starting row offset value of each target region is determined; the starting row offset value is used to locate the target region. Based on the starting row offset value of each target region, the data corresponding to at least one target column combination in the target table in each target region are written into the baseline sorting string table corresponding to the respective target column combination.

2. The method according to claim 1, wherein dividing the target table into regions based on the primary key column of the target table to obtain at least one target region includes: The data in the primary key column is sampled; Compare adjacent sample values ​​in each sample value to obtain the corresponding comparison results; Based on the comparison results, at least one data partition row is determined; The target table is divided according to the at least one data row to obtain the at least one target region.

3. The method according to claim 1, wherein determining the starting row offset value of each target region based on the row number corresponding to the at least one target region and the top-to-bottom order of the at least one target region in the target table comprises: According to the top-to-bottom order of the at least one target region in the target table, perform the following steps for each target region: If the target region is the first region, then the starting row offset value of the target region is determined to be the specified starting value; If the target region is not the first region, the starting row offset value of the target region is determined based on the row numbers corresponding to all target regions preceding the target region and the specified starting value.

4. The method according to any one of claims 1-3, wherein the step of writing the data corresponding to at least one target column combination in the target table in each target region according to the starting row offset value of each target region into the baseline sorting string table corresponding to the corresponding target column combination includes: For each target region, the following steps are executed through the thread corresponding to that target region: The number of combinations of the at least one target column is determined as the initial batch size; According to the batch size, based on the starting row offset value of the target region, the at least one target column is combined with the corresponding data in the target region and written into the corresponding baseline sort string table; If it is determined that the data write failed, then obtain the latest number of successful writes for the target column combination. Determine a new batch size based on the batch size and the latest successful quantity; Based on the new batch size, perform a data write operation for the target column combination for which data writing was unsuccessful.

5. The method according to claim 4, wherein performing a data write operation on the target column combination for which data writing was unsuccessful, based on the new batch size, comprises: Repeat the following steps until it is confirmed that the data was written successfully and there are no target column combinations that failed to write data: Based on the new batch size, and using the starting row offset value of the target region, the remaining target columns are combined with the data in the target region and written into the corresponding baseline sorting string table; the remaining target column combination is the target column combination for which data was not successfully written at the moment. If it is determined that the data writing has failed, then obtain the latest number of successful writes of the target column combination, and determine a new batch size based on the batch size and the latest number of successful writes; If it is determined that the data was written successfully, and there are target column combinations where the data was not written successfully, then the new batch size is updated based on the set ratio; If it is determined that the data was written successfully and there is no target column combination where data was not written successfully, then the data completion process is considered complete.

6. The method according to claim 5, wherein determining the new batch size based on the batch size and the latest successful quantity includes: Determine the product of the batch size and the set weight; The new batch size is determined by the maximum value between the product and the latest successful quantity.

7. The method according to claim 4, further comprising: If it is determined that the data writing failed, then obtain the sum of the batch sizes corresponding to all threads; If the sum is not higher than the number of threads, then it is determined that memory resources are insufficient, and the data completion process ends.

8. A data completion apparatus, the apparatus comprising: The partitioning unit is used to partition the target table into regions based on the primary key column of the target table to be processed, so as to obtain at least one target region. The data in the primary key column is sorted sequentially; A determination unit is used to determine the number of rows in each target region; The positioning unit is used to determine the starting row offset value of each target region according to the number of rows corresponding to the at least one target region and the order of the at least one target region in the target table from top to bottom. The starting row offset value is used to locate the target area; The writing unit is used to write the data corresponding to at least one target column combination in the target table in each target area into the baseline sorting string table corresponding to the respective target column combination, based on the starting row offset value of each target area.

9. An electronic device, comprising: processor; as well as A memory storing computer instructions for causing the processor to perform the method according to any one of claims 1-7.

10. A storage medium storing computer instructions for causing a computer to perform the method according to any one of claims 1-7.

Images