Method for sharding table for data management and electronic device therefor

Abstract

Disclosed are a method for sharding a table for data management and an electronic device therefor. The method for sharding a table according to the present disclosure may comprise the steps of: acquiring first input data including a data source, a table schema, a test query, and an expected response time; calculating an optimal table size on the basis of the table schema, the test query, and the expected response time; and calculating an optimal sharding number for a single table including the data source on the basis of the data source, the table schema, and the optimal table size.

Description

Method for partitioning tables for data management and electronic device therefor

[0001] The present disclosure relates to a technique for exploring a method for efficiently partitioning a table containing a large amount of data.

[0002] In a database managed by a relational database management system (RDBMS), the size of data within a single table may become excessively large as data accumulates over time.

[0003] To provide rapid responses to various external queries, query processing speed must be fast. However, query processing speed is determined by the size of the data contained in the table. Therefore, managing excessively large data in a single table can negatively impact query response performance and, by extension, the overall performance of database-based services.

[0004] Accordingly, there is a need to shard large-sized data into multiple tables of appropriate sizes, and naturally, there is a need to decide how many smaller tables to shard a single table into.

[0005] However, currently, there is a problem that it takes a lot of time to determine the sharding number for a single table, as it requires setting multiple sharding numbers and experimentally testing the performance of queries for each sharding number.

[0006] In this regard, reference may be made to prior literature such as KR 2022-0124075 A.

[0007] The present disclosure is proposed to solve the above-described problem, and aims to provide a device and method that can easily search for optimal conditions for dividing a large amount of data into multiple tables.

[0008] The technical tasks to be achieved by this embodiment are not limited to the technical tasks described above, and other technical tasks can be inferred from the following embodiments.

[0009] According to one aspect of the disclosure, a table sharding method performed by an electronic device may be provided, the method comprising: obtaining first input data including a data source, a table schema, a test query, and an expected response time; calculating an optimal table size based on the table schema, the test query, and the expected response time; and calculating an optimal sharding number for a single table including the data source based on the data source, the table schema, and the optimal table size.

[0010] According to one embodiment, in the step of calculating the optimal table size, the electronic device may perform a query test using the test query on a table defined according to the table schema, and optimize the table to satisfy a test objective related to the expected response time.

[0011] According to one embodiment, the test objective may be set such that the difference between the average response time according to the query test and the expected response time is within a preset error range.

[0012] According to one embodiment, the step of calculating the optimal table size may include: creating the table according to the table schema; checking the data size of the table; inputting sample data (mock data) into the table according to the data size of the table; performing the query test using the test query targeting the table; determining whether the result of the query test satisfies the test objective; calculating the data size of the table at the time when the query test is performed as the optimal table size when the result of the query test satisfies the test objective; and adjusting the data size of the table when the result of the query test does not satisfy the test objective.

[0013] According to one embodiment, in the step of adjusting the data size of the table, the electronic device may update the data size of the table based on the average response time according to the query test, the expected response time, and the data size of the table at the time when the query test was performed.

[0014] According to one embodiment, the step of adjusting the data size of the table may include the steps of: setting a proportional constant between a response time for a query and a data size; calculating a solution to an equation with one unknown variable defined using the proportional constant, the average response time, the expected response time, and the data size of the table at the time when the query test is performed; and updating the calculated solution of the equation to the data size of the table.

[0015] In one embodiment, the equation may be defined such that a first ratio between the average response time and the data size of the table at the time the query test is performed, and a second ratio between the expected response time and the unknown may be defined as the proportional constant.

[0016] According to one embodiment, the step of calculating the optimal number of divisions may include: setting a maximum number of divisions and a minimum number of divisions for the single table based on the data source and the optimal table size; calculating a data size of each of a plurality of tables according to division of the single table based on the table schema and a number of divisions having a value between the maximum number of divisions and the minimum number of divisions; determining whether a predefined optimal division condition is satisfied by inputting the data size of each of the plurality of tables; calculating the number of divisions as the optimal number of divisions when the optimal division condition is satisfied; and adjusting the number of divisions when the optimal division condition is not satisfied.

[0017] According to one embodiment, the maximum number of divisions may be set to a minimum prime number greater than or equal to a value obtained by dividing the data size corresponding to the data source by the optimal table size, and the minimum number of divisions may be set to a minimum prime number greater than or equal to a square root of the maximum number of divisions.

[0018] According to one embodiment, the number of divisions can be adjusted within a range greater than or equal to the minimum number of divisions, with the maximum number of divisions as an initial value, and in the step of adjusting the number of divisions, the electronic device can update the largest prime number that is less than or equal to the current number of divisions as a new number of divisions.

[0019] According to one embodiment, the optimal partitioning condition may be defined as a minimum sum of the absolute value of the difference between the average of the data sizes of each of the plurality of tables and the optimal table size, and the standard deviation of the data sizes of each of the plurality of tables.

[0020] According to another aspect disclosed, an electronic device for sharding a table may be provided, comprising a memory storing instructions and a processor, wherein the processor is connected to the memory to obtain first input data including a data source, a table schema, a test query, and an expected response time, calculate an optimal table size based on the table schema, the test query, and the expected response time, and calculate an optimal sharding number for a single table including the data source based on the data source, the table schema, and the optimal table size.

[0021] Another aspect of the present disclosure may provide a computer-readable recording medium having recorded thereon a program for implementing a method performed by an electronic device.

[0022] Specific details of other embodiments are included in the detailed description and drawings.

[0023] According to the disclosed embodiment, by optimizing the table size based on input data and calculating the optimal number of partitions using the optimal table size, it is possible to explore suitable conditions for partitioning a large amount of data into multiple tables without experimentally conducting query tests according to various numbers of partitions.

[0024] The effects of the invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0025] FIG. 1 is a schematic diagram illustrating a system in which an electronic device is driven according to one embodiment.

[0026] FIG. 2 is a flowchart of a table division method of an electronic device according to one embodiment.

[0027] Figure 3 is a flowchart specifically explaining a process for calculating an optimal table size according to one embodiment.

[0028] Figure 4 is a flowchart specifically explaining a process for calculating an optimal number of divisions according to one embodiment.

[0029] Figure 5 is an exemplary diagram schematically illustrating the entire process for table partitioning, from data input to calculation of the optimal number of partitions.

[0030] Figure 6 is an exemplary graph showing the process of adjusting the data size of a table.

[0031] Figure 7 is an exemplary code that implements an algorithm for calculating the optimal number of divisions.

[0032] Figure 8 shows a block diagram of an electronic device according to one embodiment.

[0033] Hereinafter, specific embodiments will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples, and the disclosed embodiments are not limited thereto.

[0034] In describing the embodiments, if it is determined that a detailed description of related known technology may unnecessarily obscure the gist of the disclosed embodiments, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the disclosed embodiments, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing the embodiments and should never be limited. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0035] The terms used in the embodiments have been selected from widely used and common terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the present disclosure.

[0036] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software, and, unlike the illustrated examples, may not be clearly distinguished in their specific operations.

[0037] The expression "at least one of a, b, and c" described throughout the specification may encompass 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c', or 'all of a, b, and c'.

[0038] The "terminal" mentioned below may be implemented as a computer or portable terminal that can connect to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, laptop, etc. equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that guarantees portability and mobility, and may include all types of handheld-based wireless communication devices such as communication-based terminals such as IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution), smartphones, tablet PCs, etc.

[0039] In the following description, the terms “transmission,” “communication,” “sending,” “receiving,” or other similar meanings of signals or information include not only direct transmission of signals or information from one component to another, but also transmission via another component.

[0040] In particular, "transmitting" or "sending" a signal or information as a single component indicates the final destination of the signal or information, not the direct destination. The same applies to "receiving" a signal or information. Furthermore, in this specification, "relating" two or more pieces of data or information means that, upon acquiring one piece of data (or information), at least a portion of the other piece of data (or information) can be acquired based on that piece of data (or information).

[0041] Additionally, while terms such as first, second, etc. may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another.

[0042] For example, without departing from the scope of the present invention, a first component could be referred to as a second component, and similarly, a second component could also be referred to as a first component.

[0043] In describing the embodiments, descriptions of technical details that are well-known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to avoid obscuring the gist of the present invention by omitting unnecessary explanations and to convey it more clearly.

[0044] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.

[0045] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0046] It will be appreciated that each block of the flowchart drawings and combinations of the flowchart drawings can be performed by computer program instructions. These computer program instructions can be installed in a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, such that the instructions, when executed by the processor of the computer or other programmable data processing equipment, create a means for performing the functions described in the flowchart block(s). These computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing equipment to perform the functions in a specific manner, such that the instructions stored in the computer-available or computer-readable memory can produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). Since the computer program instructions may be installed on a computer or other programmable data processing device, a series of operational steps may be performed on the computer or other programmable data processing device to create a computer-executable process, and the instructions that cause the computer or other programmable data processing device to perform the steps for performing the functions described in the flowchart block(s) may also provide steps for performing the functions described in the flowchart block(s).

[0047] Additionally, each block may represent a module, segment, or portion of code that contains one or more executable instructions for performing a specific logical function(s). It should also be noted that in some alternative implementation examples, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on their respective functions.

[0048] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. The methods illustrated in FIGS. 1 to 4 may be performed, for example, by an electronic device (100) described with reference to FIGS. 1 and 8. That is, the electronic device (100) is an exemplary device for dividing a table in the present disclosure, and may be a single device, but depending on the embodiment, it may also be a system composed of multiple sub-devices that communicate with each other via a wired or wireless network.

[0049] FIG. 1 is a schematic diagram illustrating a system in which an electronic device is driven according to one embodiment.

[0050] Referring to FIG. 1, the system includes a structure in which an electronic device (100) communicates with a data storage medium via a network (500). The data storage medium communicating with the electronic device (100) may include at least one of a database (200), a memory (300), or an object storage service (400). Meanwhile, the system illustrated in FIG. 1 only illustrates components related to the present embodiment. Therefore, those skilled in the art related to the present embodiment will understand that, in addition to the components illustrated in FIG. 1, other general-purpose components may be further included.

[0051] The electronic device (100) may acquire a large amount of data from a data storage medium according to a user's command, and search for table partitioning conditions that are efficient for processing a query request for the large amount of data based on initial conditions set according to the user's input. Specifically, the electronic device (100) searching for table partitioning conditions may include a process of optimizing at least one parameter value required for table partitioning. For example, the electronic device (100) may optimize the data size of a table after partitioning, or optimize the sharding number for a single table including the large amount of data. According to an embodiment, the electronic device (100) may be provided with an input interface for receiving a user's input or command, and may also be provided with an output interface for externally providing information produced through the above-described series of processes.

[0052] In FIG. 1, the electronic device (100) is illustrated as one server, but this is according to one embodiment, and the electronic device (100) may also refer to a server group including multiple servers located in different locations.

[0053] Meanwhile, a database (200) refers to a collection of systematically accumulated data, and in the present disclosure, it refers to a disk on which data is stored to distinguish it from a memory (300). In the present disclosure, a database (200) may be used to mean a database management system (DBMS).

[0054] Meanwhile, memory (300) refers to an in-memory database (IMDB) or a main memory database (MMDB), and may also refer to a volatile medium in which data is temporarily stored, depending on the embodiment. Although memory (300) has a faster data access speed than database (200), stored data may be deleted if power is not supplied to the server managing memory (300).

[0055] Meanwhile, the object storage service (400) is a general term for a service that provides storage for storing and managing data in an unstructured format called an object. The object storage service (400), which is built to allow electronic devices (100) to access data through a virtual storage repository, may also be referred to as a cloud object storage system.

[0056] The network (500) includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, a contents delivery network (CDN), and a combination thereof, and is a comprehensive data communication network that enables each network component illustrated in FIG. 1 to communicate smoothly with each other, and may include wired Internet, wireless Internet, and mobile wireless communication networks. Wireless communication may include, but is not limited to, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, Zigbee, WFD (Wi-Fi Direct), UWB (ultra wideband), infrared communication (IrDA, infrared Data Association), NFC (Near Field Communication), etc., for example.

[0057] FIG. 2 is a flowchart of a table division method of an electronic device according to one embodiment.

[0058] In step S210, the electronic device (100) may obtain first input data including a data source, a table schema, a test query, and an expected response time. In the present disclosure, a 'data source' refers to a set of data that the electronic device (100) reads from a data storage medium via a network, a 'table schema' refers to information defining the structure of a table that is generated during the process of segmentation or will be generated as a result of segmentation, and a 'test query' refers to a set of queries to be requested targeting a table that is generated during the process of segmentation or will be generated as a result of segmentation. The 'test query' may be a separate query from a query used by the electronic device (100) to read a 'data source' from the data storage medium. Meanwhile, the 'expected response time' refers to an expected response time when a request is transmitted using a test query to a table that is generated during the process of segmentation or will be generated as a result of segmentation. In the present disclosure, the 'table schema' and the 'test query' may be information received from a data storage medium, may be information previously stored in the electronic device (100), or may be information input from a user of the electronic device (100) to the electronic device (100). In addition, in the present disclosure, the 'expected response time' may be information determined by at least some of the 'data source', the 'table schema', or the 'test query', or may be information input from a user of the electronic device (100) to the electronic device (100).

[0059] In step S220, the electronic device (100) can calculate an optimal table size based on the table schema, test query, and expected response time.

[0060] In one embodiment, the electronic device (100) may perform a query test using a test query on a table defined according to a table schema to derive an optimal table size, and may optimize the table to satisfy a test objective related to an expected response time. For example, if the result of the query test does not satisfy the test objective, the electronic device (100) may adjust the table size so that the result of the query test comes closer to the test objective.

[0061] In relation to the above embodiment, the electronic device (100) may set a test goal such that the difference between the average response time and the expected response time according to the query test is within a preset error range. For example, the electronic device (100) may transmit a request to a table using a test query, and if the difference between the average response time for the request and the expected response time is within the error range, the electronic device may determine that the test goal has been met. On the other hand, if the difference between the average response time for the request and the expected response time exceeds the error range, the electronic device may determine that the test goal has not been met.

[0062] With regard to the above-described embodiment, it will be described later with reference to FIG. 3 below.

[0063] In step S230, the electronic device (100) can calculate an optimal sharding number for a single table including the data source based on the data source, table schema, and optimal table size.

[0064] Specifically, the electronic device (100) may use the optimal table size calculated as a result of performing step S220 as one of the initial values ​​for calculating the optimal number of divisions. In one embodiment, the electronic device (100) may use a predefined function when performing step S230, and the function may be a function that uses a data source, a table schema, and an optimal table size as input parameters and outputs the optimal number of divisions as at least a part of the result.

[0065] With regard to the above-described embodiment, it will be described later with reference to FIG. 4 below.

[0066] Figure 3 is a flowchart specifically explaining a process for calculating an optimal table size according to one embodiment.

[0067] In step S310, the electronic device (100) can create a table according to the table schema.

[0068] In step S320, the electronic device (100) can check the data size of the generated table. In the present disclosure, the data size may refer to the capacity of data. Accordingly, the data size of the table may refer to the capacity of data that can be included in the table. However, depending on the embodiment, the data size of the table may refer to the capacity of data that can be included in each column of the table, the capacity of data that can be included in each row of the table, or the capacity of data that can be included in each unit area constituting the table. In addition, the unit of the data size of the table may be set differently depending on the embodiment, and may generally be set to megabytes (MB).

[0069] In step S330, the electronic device (100) can input sample data (mock data) into the table according to the data size of the confirmed table. Of course, the electronic device (100) can input the data source acquired as the first input data into the table, but the optimal table size can be calculated simply by filling the table with arbitrary data.

[0070] In step S340, the electronic device (100) can perform a query test using a test query targeting a table into which sample data has been entered. For example, the electronic device (100) can transmit a request according to the test query targeting a table into which sample data has been entered and check the response time.

[0071] In step S350, the electronic device (100) can determine whether the result of the query test satisfies the test objective.

[0072] According to one embodiment, the electronic device (100) may determine that the result of the query test satisfies the test objective if the difference between the representative value of the response time taken to receive a response after transmitting a request according to the test query in step S340 and the expected response time is within a preset error range, and may determine that the result of the query test does not satisfy the test objective if the difference between the representative value and the expected response time exceeds the error range. For example, if the test query is composed of a set of multiple queries, the electronic device (100) may determine whether the test objective is met depending on whether the difference between the representative value (e.g., average, median, mode, etc.) of the response times taken in response to each query and the expected response time is within or exceeds the error range.

[0073] In one embodiment, the error range as a test target can be set to various values, typically 1 percent. A lower error range can increase the likelihood of repeating step S370 (adjusting the table data size) described below to determine the optimal table size.

[0074] In step S360, if the result of the query test satisfies the test objective, the electronic device (100) can calculate the data size of the table at the time the query test was performed as the optimal table size.

[0075] In step S370, the electronic device (100) can adjust the data size of the table if the result of the query test does not meet the test objective.

[0076] In one embodiment, the electronic device (100) can update the data size of the table based on the average response time according to the query test, the expected response time, and the data size of the table at the time the query test was performed.

[0077] For example, the electronic device (100) may set a proportional constant between the response time for a query and the data size. The electronic device (100) may then calculate a solution to an equation with one unknown variable, defined using the proportional constant, the average response time, the expected response time, and the data size of the table at the time the query test was performed. The electronic device (100) may then update the calculated solution to the equation with the data size of the table.

[0078] Among these, with respect to the above equation, the first ratio between the average response time and the table data size at the time the query test was performed, and the second ratio between the expected response time and an unknown variable can be defined as a proportional constant. This can be expressed in formulas as shown in Equations 1 to 3 below.

[0079]

[0080]

[0081]

[0082] Here, query_time is the response time corresponding to the request according to the query, data_size is the data size of the table, and a is a proportional constant. avg_time is the average response time, expect_time is the expected response time, cur_data_size is the data size of the table at the time the query test was performed, and expect_data_size is an unknown number that is the target of finding the solution to the equation and means the data size of the table to be newly updated.

[0083] That is, the electronic device (100) assumes that the response time corresponding to a request according to a query and the data size of the table are proportional to each other at a ratio a (Mathematical Expression 1), and by using this, the first ratio between the average response time and the data size of the table at the time when the query test is performed, and the second ratio between the expected response time and the unknown are defined as a (Mathematical Expression 2), and then the unknown of Mathematical Expression 2 is arranged on the right side of the equation and the remainder is arranged on the left side of the equation to obtain the value (solution) of the unknown (Mathematical Expression 3).

[0084] Thereafter, the electronic device (100) can re-enter sample data into the table whose data size has been adjusted through step S370, perform a query test, and re-determine whether the query test result satisfies the test objective (repeat from S330 onwards). This will be described in more detail later with reference to FIG. 6 below.

[0085] Figure 4 is a flowchart specifically illustrating a process for calculating an optimal number of partitions according to one embodiment. The process of Figure 4 assumes that the optimal table size has been calculated according to step S220.

[0086] In step S410, the electronic device (100) can set the maximum number of partitions and the minimum number of partitions for a single table based on the data source and the optimal table size.

[0087] In one embodiment, the electronic device (100) may set the maximum number of divisions to a minimum prime number greater than or equal to a value obtained by dividing the data size corresponding to the data source by the optimal table size. Additionally, the electronic device (100) may set the minimum number of divisions to a minimum prime number greater than or equal to the square root of the maximum number of divisions.

[0088] In step S420, the electronic device (100) can calculate the data size of each of the plurality of tables by dividing a single table based on the table schema and the number of divisions having a value between the maximum number of divisions and the minimum number of divisions.

[0089] In step S430, the electronic device (100) can input the data size of each of the plurality of tables and determine whether the predefined optimal segmentation condition is satisfied. Specifically, the electronic device (100) can input the data size of each of the plurality of tables into a predefined function or set of functions to determine whether the optimal segmentation condition is satisfied, and determine whether the optimal segmentation condition is satisfied through the output value of the corresponding function or the output value of some functions included in the set of functions.

[0090] In step S440, if the optimal division condition is satisfied, the electronic device (100) can calculate the number of divisions set at the current point in time as the optimal number of divisions.

[0091] In step S450, the electronic device (100) can adjust the number of divisions if the optimal division condition is not satisfied.

[0092] In one embodiment, the electronic device (100) may set the initial value of the number of divisions to the maximum number of divisions and adjust the number of divisions within a range greater than or equal to the minimum number of divisions. If the optimal division condition is not satisfied, the electronic device (100) may update the new number of divisions to the largest prime number less than or equal to the current number of divisions.

[0093] In one embodiment, the optimal partitioning condition may be defined as 1) the absolute value of the difference between the first representative value of the data sizes of each of the plurality of tables and the optimal table size, and 2) the sum of the second representative value of the data sizes of each of the plurality of tables being minimal. For example, the first representative value may be defined as an average, and the second representative value may be defined as a standard deviation. However, the types of representative values ​​are not limited thereto.

[0094] The process described above with reference to Fig. 4 will be exemplarily described later with reference to Fig. 7 below.

[0095] Figure 5 is an exemplary diagram schematically illustrating the entire process for table partitioning, from data input to calculation of the optimal number of partitions.

[0096] As illustrated in FIG. 5, if a data source, a table schema, a test query, and an expected response time are given as input, an optimal table size can be calculated based on at least some of these inputs. However, since the process of calculating the optimal table size is a process of optimizing the data size of the table through a query test on the table, a Structured Query Language (SQL) analysis process for the table schema, test query, etc. must be performed prior to the query test. The fact that the SQL analysis and query test processes are depicted as separate objects in FIG. 5 indicates that the processes may be performed within the electronic device (100), but may also be performed by a separate device linked to the electronic device (100) via a network.

[0097] After the query test is performed and the optimal table size is calculated, the optimal number of partitions can be calculated based on at least some of the input and the optimal table size. The electronic device (100) may return the optimal number of partitions as a result, but depending on the embodiment, it may additionally return the optimal average value of the data size of the table corresponding to the calculation time of the optimal number of partitions, or the optimal standard deviation value of the data size of the table.

[0098] Figure 6 is an exemplary graph illustrating the process of adjusting the data size of a table. In other words, Figure 6 visually illustrates the process of adjusting the data size of a table to derive an optimal table size and comparing the average response time measured as a result of a query test under the adjusted data size with the expected response time, which is part of the first input data.

[0099] The y-axis of the graph represents the average response time measured as a result of the query test, and the x-axis represents the number of times the data size was adjusted. However, since the initial data size is represented as x1, x2 represents the data size adjusted once, and x10 represents the data size adjusted nine times.

[0100] The slope of the proportional line represents the proportional constant that constitutes the equation. Since the ratio between the response time for a query request and the data size of the table is assumed to be constant, the slope of the proportional line in Figure 6 also appears constant.

[0101] Referring to Fig. 6, the y value at the intersection of the proportional line and the graph represents the average response time at the current table data size. Accordingly, it can be seen that as the number of times the data size is adjusted increases, the difference between the average response time and the expected response time decreases. If the difference between the average response time measured under the data size adjusted 10 times (x11) and the expected response time is within the error range, the x value at this time, i.e. the table data size, can be calculated as the optimal table size.

[0102] Figure 7 is an exemplary code that implements an algorithm for calculating the optimal number of divisions.

[0103] (for function)

[0104] This is a for loop to calculate the optimal number of partitions (bestShardingNum). The maximum number of partitions (maxTableCount) and the minimum number of partitions (minTableCount) can be used as input parameters for the execution of the for loop. The number of partitions (shardingNum) is a variable in the for loop, and its initial value can be set to maxTableCount. As long as shardingNum is greater than or equal to minTableCount, shardingNum is updated to the largest prime number less than or equal to the current shardingNum, and the for loop can be repeatedly executed.

[0105] Although not shown, maxTableCount can be defined as the smallest decimal value greater than or equal to the data size of the entire data source divided by the optimal table size, and minTableCount can be defined as the smallest decimal value greater than or equal to the square root of maxTableCount.

[0106] The tableSizeList array of Int data type can be the information output as a result of inputting the list of data sources (dataList), column values ​​for calculating the table index for each record (shardingColumns), and shardingNum to the calculateTableSizeByShardingNum function. In other words, the tableSizeList array can mean a list of data sizes for each of the multiple tables to be divided when the data source is divided according to shardingNum. In this case, shardingColumns can correspond to an element that determines the range of strings to be read as input to the record-specific hash function used to calculate the table index for each record.

[0107] avgTableSize of int data type is the average function applied to tableSizeList, and standardDeviation of int data type is the standard deviation function applied to tableSizeList.

[0108] The if statement can update the current avgTableSize to the optimal avgTableSize (bestAvgTableSize) if the return value of the judgetBest function described later is true, update the current standardDeviation to the optimal standardDeviation (bestStandardDeviation), and update the current shardingNum to bestShardingNum. On the other hand, the if statement will not be executed if the return value of the judgetBest function is false.

[0109] (judgeFun function)

[0110] The judgeFun function can take avgTableSize and standardDeviation as input variables and return the sum of the absolute value of the difference between avgTableSize and the optimal table size (suggestionTableSize) and standardDeviation.

[0111] (judgetBest function)

[0112] The judgeBest function takes the average (curAvgTableSize) of the data size of the current table and the standard deviation (curStandardDeviation) of the data size of the current table as input variables. If the return value of the judgeFun function when bestAvgTableSize and bestStandardDeviation are input and are greater than the return value of the judgeFun function when curAvgTableSize and curStandardDeviation are input, it returns a true value; otherwise, it returns a false value.

[0113] Figure 8 shows a block diagram of an electronic device according to one embodiment.

[0114] The electronic device (100) may include, according to one embodiment, a memory (110) and a processor (120). In one embodiment, the electronic device (100) may exchange data with the outside world through an input / output interface.

[0115] The processor (120) can perform at least one of the methods described above with reference to FIGS. 1 to 7. The memory (110) can store information for performing at least one of the methods described above with reference to FIGS. 1 to 7. The memory (110) can be a volatile memory or a non-volatile memory.

[0116] The processor (120) can control the electronic device (100) to execute a program and provide information. The code of the program executed by the processor (120) can be stored in the memory (110).

[0117] In one embodiment, the processor (120) is connected to the memory (110) to obtain first input data including a data source, a table schema, a test query, and an expected response time, calculate an optimal table size based on the table schema, the test query, and the expected response time, and calculate an optimal sharding number for a single table including the data source based on the data source, the table schema, and the optimal table size.

[0118] Additionally, the electronic device (100) according to one embodiment may further include an interface capable of providing information to a user of the electronic device (100).

[0119] The electronic device (100) illustrated in FIG. 8 only illustrates components related to the present embodiment. Therefore, those skilled in the art will understand that, in addition to the components illustrated in FIG. 8, other general-purpose components may be included.

[0120] The device according to the above-described embodiments may include a processor, a memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with an external device, a user interface device such as a touch panel, a key, a button, etc. The methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, the computer-readable recording medium includes a magnetic storage medium (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and an optical reading medium (e.g., CD-ROM, DVD: Digital Versatile Disc)). The computer-readable recording medium may be distributed to computer systems connected to a network, so that the computer-readable code may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed by a processor.

[0121] The present embodiment may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the embodiment may employ direct circuit configurations such as memory, processing, logic, look-up tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices. Similarly, the present embodiment may be implemented in a programming or scripting language such as C, C++, Java, assembler, etc., including various algorithms implemented as a combination of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms that execute on one or more processors. Furthermore, the present embodiment may employ conventional techniques for electronic configuration, signal processing, message processing, and / or data processing. Terms like "mechanism," "element," "means," and "composition" can be used broadly and are not limited to mechanical or physical components. These terms can also encompass a series of software routines, such as those associated with a processor.

[0122] The above-described embodiments are merely examples, and other embodiments may be implemented within the scope of the claims set forth below.

Claims

1. In a method performed by an electronic device, A step of obtaining first input data including a data source, table schema, test query and expected response time; A step of calculating an optimal table size based on the table schema, the test query, and the expected response time; and Comprising a step of calculating an optimal sharding number for a single table including the data source based on the data source, the table schema, and the optimal table size. How to split a table.

2. In paragraph 1, In the step of calculating the above optimal table size, The electronic device performs a query test using the test query on a table defined according to the table schema, and optimizes the table to satisfy a test goal related to the expected response time. How to split a table.

3. In paragraph 2, The above test objectives are: The difference between the average response time according to the above query test and the above expected response time is set to be within a preset error range. How to split a table.

4. In paragraph 2, The step of calculating the above optimal table size is: A step of creating the table according to the above table schema; Step of checking the data size of the above table; A step of entering sample data (mock data) into the table according to the data size of the table; A step of performing the query test using the test query targeting the table; A step of determining whether the result of the above query test satisfies the above test objective; If the result of the above query test satisfies the above test objective, a step of calculating the data size of the table at the time the above query test is performed as the optimal table size; and If the result of the above query test does not meet the above test objective, a step of adjusting the data size of the table is included. How to split a table.

5. In paragraph 4, In the step of adjusting the data size of the above table, The electronic device updates the data size of the table based on the average response time according to the query test, the expected response time, and the data size of the table at the time the query test is performed. How to split a table.

6. In paragraph 5, The steps for adjusting the data size of the above table are: A step for setting a proportional constant between the response time for a query and the data size; A step of calculating a solution to an equation with one unknown variable defined using the proportional constant, the average response time, the expected response time, and the data size of the table at the time when the query test is performed; and Including a step of updating the calculated solution of the above equation to the data size of the above table, How to split a table.

7. In paragraph 6, The above equation is, A first ratio between the average response time and the data size of the table at the time the query test was performed, and a second ratio between the expected response time and the unknown are defined as the proportional constants. How to split a table.

8. In paragraph 1, The step of calculating the above optimal number of divisions is: A step of setting the maximum number of partitions and the minimum number of partitions for the single table based on the data source and the optimal table size; A step of calculating the data size of each of a plurality of tables according to partitioning the single table based on the table schema and the number of partitions having a value between the maximum number of partitions and the minimum number of partitions; A step of determining whether the data size of each of the above multiple tables is input and whether it satisfies a predefined optimal division condition; When the above optimal division condition is satisfied, a step of calculating the number of divisions as the optimal number of divisions; and Including a step of adjusting the number of divisions when the above optimal division condition is not satisfied. How to split a table.

9. In paragraph 8, The maximum number of divisions above is The data size corresponding to the above data source is set to the minimum prime number greater than or equal to the value divided by the above optimal table size, The minimum number of divisions above is Set to the smallest prime number greater than or equal to the square root of the above maximum division number, How to split a table.

10. In paragraph 8, The above number of divisions is, With the above maximum number of divisions as the initial value, it is adjusted within a range greater than or equal to the above minimum number of divisions. In the step of adjusting the number of divisions, The electronic device updates the new division number with the largest prime number less than or equal to the current division number. How to split a table.

11. In paragraph 8, The above optimal division conditions are: The absolute value of the difference between the average of the data sizes of each of the plurality of tables and the optimal table size, and the sum of the standard deviations of the data sizes of each of the plurality of tables are defined to be the minimum. How to split a table.

12. A non-transitory computer-readable recording medium recording a program for executing the method of paragraph 1 on a computer.

13. As an electronic device for dividing a table, Includes a memory and a processor that stores instructions, The above processor is connected to the above memory, Obtain first input data including data source, table schema, test query and expected response time, Calculate the optimal table size based on the table schema, the test query, and the expected response time, An electronic device that calculates an optimal sharding number for a single table including the data source based on the data source, the table schema, and the optimal table size.

Images