A data processing method, device, terminal and storage medium

Abstract

The present application provides a data processing method, device, terminal and storage medium, the method comprising: determining a sharding column of a database to be processed; sharding the database table based on at least one number in the sharding column that meets a preset threshold value to obtain at least one database shard; determining the distribution of data records in the database that meet preset extraction conditions in the at least one database shard; based on the distribution of the data records, adjusting at least one of the database shards to obtain a target database shard, so that the efficiency of sharding the database can be improved, and database shards with uniform data distribution can be obtained.

Description

Technical Field

[0001] The present application relates to data processing technology, and relates to but is not limited to a data processing method, device, terminal and storage medium. Background Art

[0002] In the related art, there are many tools that support importing data from relational databases into other systems, such as offline data synchronization tools / platforms (DataX), data conversion tools (Sqoop), open source data exchange tools (Kettle) and data integration platforms (TurboDX), among which DataX and Sqoop open source tools are widely used. DataX is an open source data import and export tool that runs on a single node and does not shard the database table. Instead, it treats a table as a concurrency. For tables with large amounts of data, it performs single concurrency processing, so there is a bottleneck in efficiency. Sqoop is also an open source tool that is mainly used for importing and exporting between Hadoop components of distributed system infrastructure and relational databases. It uses ordinary column-based sharding when extracting the database, queries the maximum and minimum values ​​of the specified column, and then performs simple sharding according to the number of concurrencies set by the user. Some data transmission tools also use row-based sharding when extracting the database, that is, first query the total number of records, and then evenly split according to the number of concurrencies set by the user. When using data transfer tools to query the total number of records, it takes a long time to read tables with large amounts of data, and row-by-row sharding sometimes involves nesting of Structured Query Language (SQL), which is also time-consuming. The most important thing is that there will be duplication and missing of read data. When using these tools to extract and shard data, problems such as data skew, low efficiency, duplication and missing data may occur. Summary of the invention

[0003] To solve the above technical problems, an embodiment of the present invention provides a data processing method, device, terminal and storage medium, which at least solve the problems of data skew and low efficiency in the process of sharding a database.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a data processing method, the method comprising:

[0006] Determine the sharding columns of the database to be processed;

[0007] Based on at least one number in the sharding column that satisfies a preset threshold, sharding the database table to obtain at least one database shard;

[0008] Determine the distribution of data records in the database that meet a preset extraction condition in the at least one database shard;

[0009] Based on the distribution of the data records, at least one of the database shards is adjusted to obtain a target database shard.

[0010] The present application provides a data processing device, the device comprising:

[0011] A first determining unit, used to determine a shard column of a database to be processed;

[0012] A sharding unit, configured to shard the database table based on at least one number in the sharding column that meets a preset threshold, to obtain at least one database shard;

[0013] A second determining unit, configured to determine the distribution of data records in the database that meet a preset condition in the at least one database shard;

[0014] An adjustment unit is used to adjust at least one of the database shards based on the distribution of the data records to obtain a target database shard.

[0015] An embodiment of the present application provides a terminal, the terminal comprising at least: a controller and a storage medium configured to store executable instructions, wherein:

[0016] The controller is configured to execute stored executable instructions, and the executable instructions are configured to execute the data processing method provided above.

[0017] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are configured to execute the data processing method provided above.

[0018] The embodiments of the present application provide a data processing method, device, terminal and storage medium. By determining the sharding columns of a database to be processed and performing preliminary sharding on the database to be processed according to the sharding columns, database shards can be obtained, thereby improving the efficiency of sharding the database. Random sampling is performed in the database to determine the data distribution of the extracted data records in the above-mentioned database shards, and the database shards are adjusted according to the distribution to obtain target database shards. In this way, database shards with uniform data distribution are obtained, thereby improving the efficiency of database sharding. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a data processing method according to an embodiment of the present application;

[0020] Figure 2 This is another flowchart of the data processing method according to the embodiment of the present application;

[0021] Figure 3 This is an overall flow chart of the data processing method of the embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of the structure of the terminal described in the embodiment of the present application. DETAILED DESCRIPTION

[0024] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below.

[0025] In the related technology, when performing database sharding, there are the following problems:

[0026] (1) The DataX tool only runs on a single node and has an input / output (I / O) bottleneck. It is not suitable for importing and exporting big data and cannot perform sharding at the database table level.

[0027] (2) Although tools such as Sqoop use column-based sharding, it is easy for data skew to occur and records with empty shard column values ​​cannot be read.

[0028] (3) Other tools use row-based sharding. The disadvantage of this method is that it takes a long time to calculate and query the total number of records; sharding involves nested structured query language, which takes a long time to extract data, and is very slow when extracting large amounts of data; and data duplication and missing data are prone to occur.

[0029] To this end, the following technical solutions of the embodiments of the present application are proposed. In order to have a more detailed understanding of the characteristics and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0030] The present application embodiment provides a data processing method. Figure 1 This is a flow chart of the data processing method according to the embodiment of the present application, combined with Figure 1 The steps shown are explained:

[0031] Step S101: Determine the shard columns of the database to be processed.

[0032] Here, the database to be processed is a database whose data is to be exported to other systems or databases, and the shard column is a column of the database table to be processed, which is used to determine the shard of the database to be processed and query the sharded data. In some achievable implementations, a database to be processed may have at least one shard column.

[0033] Step S102: based on at least one number in the sharding column that meets a preset threshold, sharding the database table to obtain at least one database shard.

[0034] Here, a number that satisfies a preset threshold is determined in the shard column, such as the average value, maximum value, or minimum value of all data in the shard column. In a specific example, the number of data records included in the shard column can be determined, and the demarcation point of the database shard is determined based on the average value, maximum value, and maximum value of the data in the shard column, as well as the number of concurrent connections set by the user, and then the database sharding algorithm is used to perform preliminary sharding on the database table to obtain at least one database shard.

[0035] Step S103: Determine the distribution of data records in the database that meet preset extraction conditions in the at least one database shard.

[0036] Here, the data records that meet the preset extraction conditions can be: randomly extract a specific number of data records from the database, and then determine the data distribution of the sampled data records in each of the above database shards. For example, randomly extract 0.1% of the total data volume from the database, the total number of sampled data records is represented by Count, the number of shards is represented by m, and the average distribution number of sampled data records in the database shards is C. v , and C v It can be expressed by the following formula: C v =Count / m. The number of records in each shard of sampled data is represented by C i Indicates that 1<=i<=m, and then the number of records C of the sampled data of each shard is i , and the average number of sample data records C v The absolute value of the relative deviation |C i -C v | / C v , determine the data distribution of the sampled data records in each shard.

[0037] In some practicable implementations, a certain number of data records may be extracted according to a set data interval, or data records of certain columns may be extracted to determine data records in the database that meet the preset extraction conditions, and then the number of data records in each database shard in which the sampled data records are distributed may be determined.

[0038] Step S104: Based on the distribution of the data records, at least one of the database shards is adjusted to obtain a target database shard.

[0039] Here, according to the number of data records distributed in each database shard, the database shards with uneven distribution of data records are adjusted. In a specific example, if the sum of the squares of the difference between the number of data records distributed in a certain database shard of the sampled data and the average number of data records distributed in each database shard of the sampled data is relatively large, it means that the number of data records of the sampled data in this database shard is uneven, and the demarcation point of the database shard corresponding to the data record distribution is adjusted to obtain the target database shard with uniform data distribution.

[0040] In an embodiment of the present application, by determining the sharding columns of the database to be processed and performing preliminary sharding on the database to be processed based on the sharding columns, database shards can be obtained, thereby improving the efficiency of database sharding. Random sampling is performed in the database to determine the distribution of the extracted data records in the above-mentioned database shards, and the database shards are adjusted according to the distribution to obtain target database shards. In this way, database shards with uniform data distribution are obtained, thereby improving the efficiency of database sharding.

[0041] In some embodiments, in order to improve the efficiency of sharding the database to be processed and avoid duplication and missing problems in data extraction, any column of the database table to be processed is determined as a sharding column of the database to be processed.

[0042] Here, when there is a primary key in the database table to be processed, the primary key of the database table is used as the sharding column of the database to be processed; when there is no primary key in the database table to be processed, the column set by the user is used as the sharding column of the database to be processed; when the user does not set the sharding column, any column in the database table to be processed is selected as the sharding column of the database to be processed, which is used to split the database to be processed. In a specific example, when there is a primary key in the database table, and the primary key is a user ID, the user ID is used as the sharding column; when there is no primary key in the database table, the column set by the user, such as the timestamp column, is used as the sharding column of the database to be processed. In this way, according to any column in the database table to be processed as the sharding column of the database, the database to be processed is sharded, which can improve the efficiency of data scanning, thereby improving the efficiency of database sharding, and obtaining database shards with uniform data distribution.

[0043] In some embodiments, in order to obtain a database shard with a relatively uniform data quantity distribution, step S102 may also be implemented by the following steps:

[0044] S1021: Determine the first step length of the sharding column based at least on the maximum value, minimum value of the sharding column, and a predetermined concurrency number.

[0045] Here, query the maximum value and minimum value of the sharding column in the database table to be processed through an SQL statement. In a specific example, set the sharding column to splitCol, and query the maximum and minimum values of the splitCol column in the sharding column through the statement SELECT MAX(splitCol), MIN(splitCol) FROM table. Determine the first step length of the sharding column according to the concurrency number set by the user, that is, the number of database shards. In a specific example, assume that the concurrency number set by the user is m; the sharding column is splitCol, and the maximum and minimum values of the sharding column are represented by splitColmin and splitColmax respectively; then the first step length of the sharding column can be obtained by subtracting splitColmin from splitColmax and then performing m equal divisions.

[0046] S1022: Shard the database table based on the first step length to obtain at least one such database shard.

[0047] Here, according to the first step length and the concurrency number set by the user, determine the size of each shard in the sharding column, and then the upper and lower bounds of each shard in the sharding column can be determined, and the database table is sharded to obtain at least one database shard. In a specific example, assume that the concurrency number set by the user is m; the sharding column is splitCol, and the maximum and minimum values of the sharding column are represented by splitColmin and splitColmax respectively; the shard set is represented by splits, and the sharding is obtained by subtracting splitColmin from splitColmax and then performing m equal divisions to obtain m database shards: S1, S2, S3, …, S m , then the shard set splits = {S1, S2, S3, …, S m}, where S1 = {splitColmin <= splitCol < X1}, S2 = {X1 <= splitCol < X2}, and by analogy, S m = {X m-1 <= splitCol <= splitColmax}, where X1, X2, …, X m-1 are the upper and lower bounds corresponding to each shard in the shard set.

[0048] In this way, the database to be processed can be initially sharded according to the step length of the sharding column of the database to be processed, and database shards with a relatively uniform data quantity distribution can be obtained.

[0049] In some embodiments, in order to obtain a database shard with a more uniform data distribution, after the database is initially sharded, it is necessary to randomly sample data from the database to determine the distribution of the sampled data in the initially obtained database shards, and adjust the obtained database shards according to the distribution. The step S104 can also be implemented in the following manner:

[0050] The database shards whose distribution of the data records does not satisfy a specific condition are merged or split to obtain the target database shards.

[0051] Here, the data records are data records in a data set randomly extracted from a database, and the distribution of the data records is the distribution of the sampled data in each database shard, such as the value of the amount of the sampled data in each database shard, such as the difference between the amount of the sampled data in each database shard and the average distribution of the sampled data set in each database shard. When the above distribution of the data records in a certain database shard does not meet the conditions set by the user, the database shard is adjusted, the database shard is split, or it is merged and split with other database shards, until the distribution of the data records of the sampled data in each database shard meets the specific conditions, the adjustment of the database shard is stopped, and the target database shard is obtained.

[0052] In some achievable implementations, merging or splitting the database shards whose distribution of the data records does not satisfy a specific condition to obtain the target database shards may also be implemented by the following steps:

[0053] Step 1: Determine the absolute value of the distribution quantity difference of the data records in each of the database shards.

[0054] Here, the data distribution quantity of the data records of the sampled data in each database shard is queried through SQL statements, and the average distribution quantity of the sampled data in the database shard is determined, and then the data distribution quantity in each database shard is subtracted from the average distribution quantity to obtain the absolute value of the difference between the two, and finally the ratio between the absolute value of the difference and the average distribution quantity is calculated to obtain the absolute value of the distribution quantity difference of the data records in each of the database shards. In a specific example, 0.1% of the data records are extracted from the database, and the data distribution of the data records in the sampled data set in each database shard is counted. Assume that the number of records in the sampled data set is represented by Count, C v is the average distribution quantity: C v = Count / m, the number of each fragment of the statistical sampling data set in splits is expressed as Counts, Counts = {C1, C2, ..., C m}. The number of records in each shard of sampled data is recorded in C i Indicates that 1<=i<=m, where m represents the number of shards, then the absolute value of the distribution quantity difference in each database shard is |C i -C v | / C v .

[0055] Step 2: When the absolute value does not satisfy the specific condition, the database shards to which the data records corresponding to the absolute value belong are merged or split to obtain the target database shards.

[0056] Here, in a specific example, the specific condition is that the absolute value is less than or equal to a specific value, such as the specific condition is that the absolute value |C i -C v | / C v <=0.1. When the absolute value is greater than or equal to 0.1, the specific condition is not met, and the database shard to which the data record corresponding to the absolute value belongs needs to be merged or split. In some achievable implementations, according to the position corresponding to the database shard, the database shards adjacent to the database shard to which the data record corresponding to the absolute value belongs also need to be split or merged to obtain the target database shard.

[0057] In this way, database shards that do not meet specific conditions can be merged and split according to the absolute value of the data record distribution of the sampled data in each database shard, and database shards with uneven data distribution can be adjusted to obtain database shards with uniform data quantity distribution.

[0058] In some feasible implementations, step 2 may also be implemented by the following steps:

[0059] First, when the absolute value does not satisfy the specific condition, the step size of the database shard to which the data record corresponding to the absolute value belongs is adjusted to obtain a second step size.

[0060] Here, when the absolute value does not meet the specific conditions, the step size of the database shard to which the data record corresponding to the absolute value belongs is adjusted according to the positive or negative value of the quantity difference. i -C v | / C v The corresponding quantity difference C i -C v When it is a positive value, it means that the amount of data recorded in the database shard is relatively large, and the step length of the database shard to which it belongs is shortened to obtain the second step length. i -C v | / C v The corresponding quantity difference Ci -C v When it is a negative value, it means that the amount of data in the database shard to which the data record belongs is relatively small. At this time, the step size of the database shard to which the data record belongs is expanded to obtain the second step size.

[0061] Secondly, based on the second step length, the database shards to which the data records corresponding to the absolute value belong are merged or split to obtain the target database shard.

[0062] Here, according to the second step length, the upper and lower bounds corresponding to the database shard to which the data record belongs are adjusted to merge or split the database shard to which the data record corresponding to the absolute value belongs. When the second step length is the step length shortened by the first step length, the upper or lower bound or upper and lower bounds of the database shard to which the data record belongs are reduced according to the distribution of data records of other database shards, and the splitting of the database shard to which the data record corresponding to the absolute value belongs is completed. When the second step length is the step length enlarged by the first step length, the upper or lower bound or upper and lower bounds of the database shard to which the data record belongs are enlarged according to the distribution of data records of other database shards, and the merging of the database shard to which the data record corresponding to the absolute value belongs is completed. In a specific example: after preliminary sharding, 5 database shards are obtained, and the absolute value of the difference in the distribution quantity of the sampled data in the first and second database shards does not meet the specific conditions, wherein the difference in the distribution quantity of the data records in the first database shard to which the data records belong is positive, indicating that the data distribution in the first database shard is relatively small; the difference in the distribution quantity of the data records in the two database shards to which the data records belong is negative, indicating that there is more data in the second database shard; according to the step sizes of the first database shard and the second database shard, the step size of the first database shard is expanded, and the step size of the second database shard is shortened to obtain the second step size, and according to the second step size, the upper and lower bounds of the corresponding database shards are adjusted to complete the merging of the first database and the splitting of the second database, until the distribution of the data records of the sampled data in each database shard meets the specific conditions, then the splitting or merging of the database shards is stopped to obtain the target database shards.

[0063] Thus, according to the second step length of the adjusted database shard, the database shard to which the distribution of data records that do not meet specific conditions belongs is adjusted to obtain a database shard with uniform data record distribution. In this way, a database shard with uniform data distribution can be obtained.

[0064] In some implementations, after obtaining database shards with uniform data distribution of the database to be processed, it is necessary to perform operations such as reading and querying the data in the database shards. Therefore, after step S104, the method further includes the following process:

[0065] Based on the target database shards, a corresponding structured query language statement is generated to read the data in each of the database shards.

[0066] Here, after obtaining the target database shards, a corresponding structured query language statement is generated according to the values ​​of the start field and the end field of a column in the data area corresponding to each target database shard, so as to distribute and read the data in each database shard.

[0067] In this way, a structured query language statement corresponding to each database shard is generated according to the start and end fields of the column of the data area corresponding to the target database shard. The structured query language statement obtained in this way does not perform nesting of structured query language statements, is efficient when executing the statement, facilitates data distribution and reading, and improves the efficiency of reading data.

[0068] The present application embodiment provides a data processing method. Figure 2 Another flow chart of the data processing method of the present application embodiment is shown in FIG. Figure 2 The steps shown are explained:

[0069] Step S201: Determine the shard columns of the database to be processed.

[0070] Here, the table in the database is queried to see if there is a primary key pk. If so, the database's shard column is: splitCol=pk. Otherwise, it is checked to see if the user has set a shard column userSplitCol. If the user has set a shard column, the database's shard column is: splitCol=userSplitCol. If the table does not have a primary key and the user has not set a shard column, a column is selected from the table as the shard column.

[0071] Step S202: Determine the maximum value and the minimum value of the slice column.

[0072] Here, the maximum and minimum values ​​of the splitCol column are queried through SELECT MAX(splitCol), MIN(splitCol) FROM table.

[0073] Step S203: based on the maximum value and the minimum value, the database to be processed is fragmented to obtain database fragments.

[0074] Here, the first data sharding is performed based on the maximum and minimum values ​​of the sharding column and the number of concurrency set by the user to obtain preliminary database sharding.

[0075] Step S204: sampling from the database to determine the distribution of the sampled data in the database shards.

[0076] Here, a certain number of data records are randomly selected from the database. In a specific example, 0.1% of the total data volume is randomly selected from the database. The total number of sampled data records is represented by Count, the number of shards is represented by m, and the average number of sampled data records in the database shards is C. v , and C v It can be expressed by the following formula: C v =Count / m. The number of records in each shard of sampled data is represented by C i Indicates that 1<=i<=m, and then the number of records C of the sampled data of each shard is i , and the average number of sample data records C v The absolute value of the relative deviation |C i -C v | / C v , determine the data distribution of the sampled data records in each shard.

[0077] Step S205: Based on the distribution situation, the database shards are adjusted to obtain target database shards.

[0078] Here, if the distribution of data records |C i -C v | / C v >0.1, then repeat step S203 to adjust the database shard to which the data record belongs until |C i -C v | / C v When it is less than or equal to 0.1, the sharding process is stopped and the target database shard is obtained.

[0079] Step S206: Generate a structured query language statement based on the target database shard.

[0080] In an embodiment of the present application, by determining the sharding columns of a database to be processed and performing preliminary sharding on the database to be processed based on the sharding columns, database shards are obtained, thereby improving the efficiency of sharding the database, randomly sampling in the database, determining the distribution of the sampled data records in the above-mentioned database shards, and adjusting the database shards based on the distribution to obtain target database shards, and generating corresponding structured query language statements, thereby improving the efficiency of sharding the database, and obtaining database shards with uniform distribution of the number of data records, and further, distributing and reading the data in the database shards based on the generated structured query language statements, thereby improving the work efficiency of reading the database.

[0081] The present application embodiment provides a data processing method. Figure 3 The overall flow chart of the data processing method of the embodiment of the present application is as follows: Figure 3 As shown, the data processing method comprises the following steps:

[0082] Step S301: Start the process.

[0083] Step S302: Determine whether the primary key exists.

[0084] Here, if the primary key exists in the table of the database, step S303 is executed. If the primary key does not exist, step S304 is executed.

[0085] Step S303: determine the primary key as the sharding column.

[0086] Here, the primary key of the table is determined as the shard column of the database.

[0087] Step S304: Determine whether the column set by the user exists.

[0088] Here, if the primary key does not exist in the database table, it is determined whether the user sets a column in the table as a sharding column. If the user sets the sharding column, step S305 is executed. If the user does not set the sharding column, step S306 is executed.

[0089] Step S305: Determine the column set by the user as the sharding column.

[0090] Here, the sharding column set by the user is used as the sharding column of the database to shard the database.

[0091] Step S306: Select a column of the database table as a sharding column.

[0092] Here, when there is no primary key in the database table and the user has not set a sharding column, a column in the database table is selected as the database sharding column.

[0093] Step S307: Query the maximum and minimum values ​​of the shard column.

[0094] Here, the maximum and minimum values ​​of the splitCol column are queried through the statement SELECT MAX(splitCol), MIN(splitCol) FROM table.

[0095] Step S308: Determine database sharding.

[0096] Here, at least one database shard is determined based on the maximum value, minimum value, and concurrency number of the shard column.

[0097] Step S309: Randomly sample from the database.

[0098] Here, a certain number of data records are randomly extracted from the database, and the extraction methods may be at least the following: randomly extracting a specific number of data records from the database; or extracting a certain number of data records according to a set data interval; or extracting data records from a specific column.

[0099] Step S310: Determine the data distribution of data records in each database shard.

[0100] Here, the sample data set is traversed to determine the distribution of data records in each shard in the database shard to generate a data record Counts set, and the total number of records in the sample data set is calculated.

[0101] Step S311: Determine whether the distribution of data records meets a specific condition.

[0102] Here, the specific condition is that the absolute value of the distribution quantity difference of the data records in each of the database shards is less than or equal to a specific value, for example, the specific condition is the absolute value |C i -C v | / C v <=0.1, where C i The number of records in each shard for the sampled data, C v The average distribution of sampled data among all shards.

[0103] If the specific condition is met, step S312 is executed. If the specific condition is not met, step S308 is repeated.

[0104] Step S312: Generate a structured query language statement and execute the extraction task.

[0105] Here, after obtaining the target database shards, a corresponding structured query language statement is generated according to the values ​​of the start field and the end field of a column in the data area corresponding to each target database shard, so as to distribute and read the data in each database shard.

[0106] Step S313: End the process.

[0107] In this way, the sharding columns of the database to be processed can be determined according to whether the database has the primary key and the columns set by the user, and the database to be processed can be sharded to obtain multiple database shards. Then, random sampling is performed in the database to determine the distribution of the extracted data records in the above database shards, and the database shards are adjusted according to the distribution to obtain the target database shards, and the corresponding structured query language statements are generated. In this way, the efficiency of sharding the database is improved, and database shards with uniform distribution of the number of data records can be obtained. Moreover, the data in the database shards can be distributed and read according to the generated structured query language statements, which improves the work efficiency of reading the database.

[0108] The present application provides a data processing device. Figure 4 Schematic diagram of the structure of the data processing device of the embodiment of the present application, such as Figure 4 As shown, the data processing device 400 includes: a first determination module 401, a slicing module 402, a second determination module 403 and an adjustment module 404, wherein:

[0109] A first determination module 401 is used to determine a shard column of a database to be processed;

[0110] A sharding module 402, configured to shard the database table based on at least one number in the sharding column that meets a preset threshold, to obtain at least one database shard;

[0111] A second determination module 403 is used to determine the distribution of data records in the database that meet a preset extraction condition in the at least one database shard;

[0112] The adjustment module 404 is used to adjust at least one of the database shards based on the distribution of the data records to obtain a target database shard.

[0113] In the above device, the first determining module 401 is further used to determine any column of the database table to be processed as a shard column of the database to be processed.

[0114] In the above device, the slicing module 402 further includes:

[0115] A first determination submodule, configured to determine a first step length of the shard column based at least on a maximum value, a minimum value, and a predetermined number of concurrencies of the shard column;

[0116] The sharding submodule is used to shard the database table based on the first step to obtain at least one database shard.

[0117] In the above device, the adjustment module 404 is further used to merge or split the database shards whose distribution of the data records does not meet the specific conditions to obtain the target database shards.

[0118] In the above device, the adjustment module 404 further includes:

[0119] A second determination submodule is used to determine an absolute value of a distribution quantity difference of the data records in each of the database shards;

[0120] The merging or splitting submodule is used to merge or split the database shards to which the data records corresponding to the absolute value belong when the absolute value does not meet the specific condition, so as to obtain the target database shard.

[0121] In the above device, the merging or splitting submodule further includes:

[0122] an adjusting unit, configured to adjust the step size of the database shard to which the data record corresponding to the absolute value belongs when the absolute value does not satisfy the specific condition, so as to obtain a second step size;

[0123] A merging or splitting unit is used to merge or split the database shards to which the data records corresponding to the absolute value belong based on the second step size to obtain the target database shard.

[0124] In the above device, the device further comprises:

[0125] A generation module is used to generate corresponding structured query language statements based on the target database shards to read data in each of the database shards.

[0126] An embodiment of the present application further provides a data processing device, which includes the modules included, and the sub-modules and units included in each module, which can be implemented by a processor in a terminal; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0127] Correspondingly, an embodiment of the present application provides a terminal, Figure 5 This is a schematic diagram of the structure of the terminal described in the embodiment of the present application, such as Figure 5 As shown, the terminal 500 at least includes: a controller 501 and a storage medium 502 configured to store executable instructions, wherein:

[0128] The controller 501 is configured to execute stored executable instructions, where the executable instructions are used to implement the provided data processing method.

[0129] It should be noted that the description of the above terminal embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the terminal embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0130] Correspondingly, an embodiment of the present application provides a computer storage medium, in which computer executable instructions are stored, and the computer executable instructions are configured to execute the data processing method provided by other embodiments of the present application.

[0131] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0132] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0133] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0135] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0137] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0138] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0139] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the sharding columns of the database to be processed; Based on at least one number in the sharding column that satisfies a preset threshold, sharding the database table to obtain at least one database shard; Determine the distribution of data records in the database that meet a preset extraction condition in the at least one database shard; Based on the distribution of the data records, adjusting at least one of the database shards to obtain a target database shard; The adjusting, based on the distribution of the data records, at least one of the database shards to obtain a target database shard includes: Merging or splitting the database shards whose distribution of the data records does not satisfy a specific condition to obtain the target database shards; The merging or splitting of the database shards whose distribution of the data records does not satisfy a specific condition to obtain the target database shards includes: Determine an absolute value of a distribution quantity difference of the data record in each of the database shards; When the absolute value does not satisfy the specific condition, the database shards to which the data records corresponding to the absolute value belong are merged or split to obtain the target database shard.

2. The method according to claim 1, characterized in that The step of determining the shard columns of the database to be processed includes: Any column of the database table to be processed is determined as a shard column of the database to be processed.

3. The method according to claim 1, characterized in that The sharding the database table based on at least one number in the sharding column that meets a preset threshold to obtain at least one database shard includes: Determine the first step length of the shard column based at least on the maximum value, the minimum value and the predetermined concurrency number of the shard column; Based on the first step, the database table is sharded to obtain at least one database shard.

4. The method according to claim 1, characterized in that When the absolute value does not satisfy the specific condition, merging or splitting the database shards to which the data records corresponding to the absolute value belong to, to obtain the target database shards, includes: When the absolute value does not satisfy the specific condition, adjusting the step size of the database shard to which the data record corresponding to the absolute value belongs to obtain a second step size; Based on the second step length, the database shards to which the data records corresponding to the absolute value belong are merged or split to obtain the target database shard.

5. The method according to claim 1, characterized in that After adjusting at least one of the database shards based on the distribution of the data records to obtain a target database shard, the method further includes: Based on the target database shards, a corresponding structured query language statement is generated to read the data in each of the database shards.

6. A data processing device, characterized in that: The device comprises: A first determination module, used to determine the shard columns of the database to be processed; A sharding module, configured to shard the database table based on at least one number in the sharding column that meets a preset threshold, to obtain at least one database shard; A second determination module, used to determine the distribution of data records in the database that meet a preset extraction condition in the at least one database shard; An adjustment module, configured to adjust at least one of the database shards based on the distribution of the data records to obtain a target database shard; The adjustment module is used to merge or split the database shards whose distribution of the data records does not meet the specific conditions to obtain the target database shards; The adjustment module further includes: A second determination submodule is used to determine an absolute value of a distribution quantity difference of the data records in each of the database shards; The merging or splitting submodule is used to merge or split the database shards to which the data records corresponding to the absolute value belong when the absolute value does not meet the specific condition, so as to obtain the target database shard.

7. A data processing terminal, characterized in that: The terminal at least includes: a controller and a storage medium configured to store executable instructions, wherein: The controller is configured to execute stored executable instructions, wherein the executable instructions are configured to execute the data processing method provided in any one of claims 1 to 5 above.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. The computer-readable storage medium stores computer-executable instructions. The computer-executable instructions are configured to execute the data processing method provided in any one of claims 1 to 5 above.

Images