A method, apparatus, and device for generating imbalanced data

Abstract

This specification discloses a method, apparatus, and device for generating unbalanced data. The method involves: acquiring the number of first nodes and data skew parameters; determining the number of second nodes corresponding to each first node based on the first node number and data skew parameters; generating a data table corresponding to the number of second nodes for each first node; and routing the data in the data table to each database node, thereby achieving the generation of controllable unbalanced data distribution in a database cluster.

Description

Technical Field

[0001] This specification relates to the field of Internet technology, and in particular to a method, apparatus, and device for generating unbalanced data. Background Technology

[0002] With the development of cloud technology, more and more enterprise applications and underlying database services are migrating to the cloud, where data is distributed across multiple database nodes. If data imbalance occurs across these databases, data migration and scheduling are required. To test this data migration and scheduling capability, imbalanced data needs to be generated within the database cluster.

[0003] Therefore, a scheme for generating imbalanced data is needed. Summary of the Invention

[0004] This specification provides an embodiment of an imbalanced data generation method, apparatus, device, and storage medium to address the following technical problem: the need for an imbalanced data generation scheme.

[0005] To solve the above-mentioned technical problems, one or more embodiments of this specification are implemented as follows:

[0006] In a first aspect, embodiments of this specification provide a method for generating unbalanced data, the method comprising: obtaining a number of first nodes and a data skew parameter; determining a number of second nodes corresponding to each first node based on the number of first nodes and the data skew parameter; generating a data table for each first node corresponding to the number of second nodes; and routing data partitions in the data table to each database node.

[0007] In a second aspect, embodiments of this specification provide an apparatus for generating unbalanced data. The apparatus includes: an acquisition module for acquiring a number of first nodes and data skew parameters; a determination module for determining a number of second nodes corresponding to each first node based on the number of first nodes and data skew parameters; a generation module for generating a data table corresponding to the number of second nodes for each first node; and a routing module for routing data partitions in the data table to each database node.

[0008] In a third aspect, embodiments of this specification provide an electronic device, including:

[0009] At least one processor; and,

[0010] A memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0012] In a fourth aspect, embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, which, when read by a computer from the storage medium, cause one or more processors to perform the method described in the first aspect.

[0013] The above-described at least one technical solution adopted in one or more embodiments of this specification can achieve the following beneficial effects: by obtaining the number of first nodes and data skew parameters; determining the number of second nodes corresponding to each first node based on the number of first nodes and data skew parameters; generating a data table corresponding to the number of second nodes for each first node; and routing the data partitions in the data table to each database node, thereby realizing the generation of controllable unbalanced data distribution in the database cluster, providing a basis for evaluating the data scheduling capabilities of the database. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating a method for generating imbalanced data as provided in an embodiment of this specification;

[0016] Figure 2a This is a schematic diagram illustrating the number of second nodes corresponding to a first node, as provided in an embodiment of this specification.

[0017] Figure 2b This is a schematic diagram illustrating the generation of a data table provided in an embodiment of this specification.

[0018] Figure 3a A schematic diagram of the architecture of a database node used in the embodiments of this specification;

[0019] Figure 3b This is a schematic diagram of the architecture of a data generation system provided in the embodiments of this specification;

[0020] Figure 4 A schematic diagram of the structure of an unbalanced data generation device provided in the embodiments of this specification;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0022] This specification provides an embodiment of a method, apparatus, device, and storage medium for generating unbalanced data.

[0023] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0024] In cloud environments, enterprises distribute data across multiple data centers, and distributed relational databases scale through data partitioning. However, the complex application scenarios in the cloud often require databases to have data scheduling capabilities under uneven distribution conditions. In evaluating database scheduling capabilities, each benchmark typically has its own defined data model, and data is generated based on this model during the evaluation process. However, traditional benchmarks do not consider the case of uneven data distribution. As relational databases increasingly evolve towards distributed systems, uneven data distribution across multiple nodes has become the norm, and conventional benchmarks cannot efficiently and quantitatively generate uneven data distributions in distributed clusters. Therefore, this specification provides an embodiment of an imbalanced data generation scheme.

[0025] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for generating imbalanced data as provided in an embodiment of this specification. Figure 1 The process may include the following steps:

[0026] S101: Obtain the number of first nodes and data skew parameters.

[0027] In different application scenarios, the first node and the second node can represent different objects. For example, in the scenario of online transaction processing in e-commerce, the first node can be a warehouse, and the second node can be the region under the jurisdiction of the warehouse; the database stores the order transaction data generated by the warehouse in various regions; in the scenario of securities trading, the first node can represent a securities company, and the second node can represent the securities trading object; in the scenario of parts sales management, the first node can represent the parts supplier, and the second node can represent the country and region where the parts are sold, etc.

[0028] The first and second nodes can be simulated in the data model. Taking an e-commerce online transaction processing scenario as an example, the number of first nodes (i.e., warehouses) can be configured. For example, the number of first nodes can be determined by providing configurable options through a parameter configurator.

[0029] The data skew parameter characterizes the degree of imbalance in the amount of data across database nodes. A data skew parameter S ≥ 1; the larger S is, the greater the skewness, meaning the more unbalanced the data is expected to be generated across the database nodes. In this application, the amount of data in each database node refers to the spatial volume occupied by the data contained in that database node, where the spatial volume occupied by the data is positively correlated with the number of rows of data.

[0030] In addition to configuring the number of first nodes and data skew parameters, the parameter configurator can also provide other types of configurable parameters. For example, configurable parameters may include database type, database size, and database partitioning strategy. Database types include distributed relational databases such as OceanBase, TiDB, and CockroachDB, while database size is used to configure the total amount of data generated.

[0031] S103, determine the number of second nodes corresponding to each first node based on the number of first nodes and the data skew parameter.

[0032] The data skew parameter determines the degree of imbalance in the amount of data in each database node. In practical applications, different calculation methods can be used to determine the number of second nodes corresponding to each first node.

[0033] For example, the number of first nodes can be multiplied by an amplification factor to obtain the total number of second nodes. Then, based on the data skew parameter as the variance, the aforementioned total can be allocated to each first node, such that the variance of the number of second nodes allocated to each first node is equal to the aforementioned data skew parameter.

[0034] In this approach, assuming the number of first nodes is 10, the data skew parameter is 5, and the amplification factor is K (K is a natural number greater than 1), then 10K second nodes can be randomly assigned to the first nodes, ensuring that the number of second nodes assigned to each of the 10 first nodes has a variance of 5. Under this implementation, the number of second nodes assigned to each first node is not uniform on an individual basis, but overall, it satisfies the constraint of the data skew parameter.

[0035] For example, the distribution of the first node, including the number of first nodes and the data skew parameter, can be determined first, and the number of the first node can be determined. Then, the number and the distribution of the first node determine the number of second nodes corresponding to the first node. In this implementation, the distribution of the first node is an unbalanced density distribution function related to the number of the first node, so the number of second nodes corresponding to each first node will also be unbalanced.

[0036] The distribution of the first node can be a pre-defined distribution. In one implementation, the distribution of the first node can be obtained in advance by performing statistics based on the actual situation (i.e., the distribution relationship between the first node and the second node in the actual implementation scenario).

[0037] For example, the first node distribution can be a Gaussian distribution, in which the data skew parameter can be the variance value of the Gaussian distribution. Another example is that the first node distribution satisfies the following distribution, and under the Zipfian distribution, it guarantees that each first node serves at least one second node, with the following density function constraints:

[0038]

[0039] Among them, M k This represents the number of second nodes corresponding to the first node numbered k, where ceil is the floor function, s is the data skew parameter, and N is the number of first nodes. Given N, S can also be given through the aforementioned distribution constraints.

[0040] In this case, the number of second nodes corresponding to each first node is inversely proportional to the number. If the number of second nodes corresponding to the first node with number 1 is M, then the number of second nodes corresponding to the first node with number n (n is a natural number greater than 1) is M / n, and the sum of the number of second nodes corresponding to all first nodes is 10N.

[0041] In this method, assuming the number of first nodes is 180 and the data skew parameter is 0.99, the second nodes can be randomly assigned to the first nodes. In this way, the number of second nodes assigned to the top ten numbered first nodes accounts for approximately 78% of the total. In this implementation, the number of second nodes assigned to each first node is uneven on an individual basis, while overall it meets the constraint of the data skew parameter.

[0042] like Figure 2a As shown, Figure 2aThis diagram illustrates the number of second nodes corresponding to a first node, as provided in an embodiment of this specification. In this diagram, the first node represents a warehouse, and the second node represents a service area. With 180 warehouse data points, the bar chart showing the number of service areas served by each warehouse is shown. Based on the different warehouse numbers, the number of service areas corresponding to each warehouse is determined. It can be seen that in this implementation, the number of second nodes corresponding to each first node is significantly uneven, with 77.2% of the areas concentrated in the top 10 warehouses.

[0043] S105, Generate a data table corresponding to the number of second nodes for each first node.

[0044] In this embodiment, the data to be generated should be controllable and quantitatively analyzable to ensure the fairness of the evaluation. The generated data table is actually linked to both the first node and the second node.

[0045] After determining the number of second nodes corresponding to each first node using the aforementioned method, the number of second nodes corresponding to any first node number can be obtained. Then, a data table corresponding to the number of second nodes can be generated for each node. The generation method, data type, and size of each data table should be consistent.

[0046] For example, in the context of online transaction processing in e-commerce, the first node can be a warehouse, and the second node can be the regions managed by that warehouse. Suppose we've determined that warehouse 1 corresponds to 100 regions, and region 2 corresponds to 50 warehouses. Then, we can generate 100 data tables for warehouse 1 and 50 data tables for region 2.

[0047] If we assume that each warehouse contains inventory information for A10,000 products, each region contains A2 customers and corresponding order information, and each order involves A3 to A4 products, then the following pseudocode algorithm can be used to generate data tables for each warehouse and region:

[0048] Loop through A1 to A2 to generate a data table (w, d, c) containing warehouse number w, region number d, and customer information c;

[0049] The process iterates from 1 to A2 to generate a data table (w, d, c) containing warehouse number w, region number d, and customer history information c.

[0050] The process iterates from 1 to A2, generating a data table (w, d, o) containing warehouse number w, region number d, and order information o. When o is greater than a specified value y, a data table (w, d, o) containing warehouse number w, region number d, and new order information o is generated.

[0051] From A3 to A4, select a random number ol (the random number ol represents the detailed information contained in an order), and generate a data table (w, d, o, ol) containing warehouse number w, region number d, order information / new order information, and the random number ol. For example... Figure 2b As shown, Figure 2b This diagram illustrates the generation of a data table provided in an embodiment of this specification. Given a cardinality W for the warehouses, the cardinality of other data tables will be expanded based on the cardinality and the number Z of regions corresponding to the warehouses. Since Z is unbalanced, the generated data tables will also be unbalanced, and ultimately, they will be unbalanced when routed to the database nodes.

[0052] In other words, when generating data tables, a pre-defined algorithm and parameters can be used to generate data tables corresponding to the number of second nodes for each warehouse. In practice, to ensure the convenience of the generated data, the same algorithm and parameters are usually used to generate all data tables, so that the data in each data table is basically uniform.

[0053] As mentioned earlier, since the number of second nodes corresponding to each first node is different, the number of data tables corresponding to each first node becomes unbalanced.

[0054] S107, route the data partitions in the data table to each database node.

[0055] Each data table will be divided into different data partitions according to the partition key. The database will maintain a routing table for the partitions based on the partitioning mode, and the data in each partition will be routed to different database nodes. Specifically, a corresponding partitioning strategy can be given in advance, thereby dividing the data in the data table corresponding to the same primary node into multiple data partitions, and routing the data in partitions within a range to the same database node.

[0056] In practical applications, a first node may correspond to multiple second nodes simultaneously. Furthermore, as mentioned earlier, the data table actually specifies a association between a first node and a second node. In distributed relational databases, multiple distributed database nodes are typically used simultaneously to store the data tables generated by the same first node and second node. For example, in e-commerce transactions, for warehouse A and city B, a set of database nodes would typically be used in city B to store the data tables (A, B, c). Figure 3a As shown, Figure 3aThis is a schematic diagram of a database node architecture used in the embodiments of this specification. Based on this structure, in a database cluster, database nodes may be divided into multiple subsets. Each subset's database nodes are relatively independent, used to store data generated by their corresponding first and second nodes. Since the number of data tables corresponding to each first node is uneven, after routing the data to each database node, the amount of data generated in each subset will inevitably be uneven as well. Furthermore, the method of partitioning subsets in the database cluster can be determined based on actual needs; for example, each database node can be divided into a subset, or every five database nodes can be divided into a subset.

[0057] By acquiring the number of first nodes and data skew parameters; determining the number of second nodes corresponding to each first node based on the number of first nodes and data skew parameters; generating a data table corresponding to the number of second nodes for each first node; and routing the data partitions in the data table to each database node, a controllable unbalanced distribution of data can be generated in the database cluster, providing a basis for evaluating the data scheduling capabilities of the database.

[0058] In one embodiment, when routing data partitions from the data table to each database node, a partitioning strategy can be predefined. For example, the partitioning strategy can be predefined by a parameter configurator, and the given partitioning strategy can also be modified or configured based on actual needs.

[0059] Configurable partitioning strategies can include primary key partitioning or hash partitioning. Primary key partitioning can further include uniform primary key partitioning or non-uniform primary key partitioning. Primary key partitioning refers to partitioning the data table based on the primary key sequence number to generate multiple data partitions, while hash partitioning refers to partitioning the data table based on the hash value of the primary key to generate multiple data partitions.

[0060] In a relational database, each row in a table is uniquely identified by a primary key. When creating a table, the columns that make up the primary key can be specified in advance; these columns are called primary key columns. Primary key columns must have values, and the combination of values ​​in the primary key columns can uniquely identify a row of data.

[0061] For example, for a transaction order, the columns TradeNo, UserID, Time, and Addr represent the transaction sequence number, user ID, transaction time, and shipping address, respectively. These columns are primary key columns that uniquely identify a transaction. The TradeNo, UserID, or Time column can be specified as the data partition key.

[0062] Hash partitioning refers to calculating the partition of a dataset based on the hash value of the primary key using a hash function. Under hash partitioning, data is distributed almost evenly across partitions. Hotspot partition keys can be distributed across different partitions and routed to different database nodes in the cluster, thus mitigating hotspot issues to some extent.

[0063] A standard primary key partition can be evenly partitioned using transaction numbers. For example, if a table contains 1000 records with transaction numbers from 0001 to 1000, the records can be evenly divided into 10 data partitions based on their transaction numbers and routed to the database nodes.

[0064] By pre-defining different partitioning strategies and specifying different primary key columns for partitioning, the amount of data in each partition can be affected differently. For example, by specifying the primary key index of the data table for non-uniform partitioning, multiple data partitions with uneven data sizes can be generated. The non-uniform partitioning can be done as follows: based on the primary key index of the data table, a basic coefficient K is specified, the first partition specifies K rows of data, the second partition specifies 2k rows of data, ..., the i-th partition specifies i*K rows of data, until the data in the data table is completely partitioned. In this way, multiple data partitions with uneven data sizes can be generated.

[0065] After partitioning, if there are multiple database nodes in the database subset corresponding to the first node, the data partition can be routed to the database nodes in the database subset based on certain rules (e.g., routing the data partition by taking the remainder of the data node number). This can achieve a quantitative imbalance in data distribution on each database node in the database subset.

[0066] In one embodiment, since the distribution relationship between the first node and the second node is given in the first node distribution, it is actually desirable that the distribution of data volume in each database node also matches the distribution of the first node. Based on this, the distribution of data volume in each database can also be statistically analyzed, and the distribution of data volume in the database can be compared with the distribution of the first node. If the difference between the two is large, it indicates that the distribution of the generated data volume may not be consistent with the real scenario.

[0067] Based on this, the data skew parameter can be adjusted to ensure that the distribution of data quantity in the database matches the distribution of the first node. For example, if the preset distribution is a Gaussian distribution, the variance value can be adjusted (the mean is usually kept the same as the actual value) to make the distribution closer to the actual distribution. As another example, under a Zipfian distribution, s can be adjusted to a suitable value so that the error between the distribution of data quantity in the database nodes and the actual distribution does not exceed 5%.

[0068] In practical applications, the aforementioned imbalanced data can be generated through component simulation, such as... Figure 3b As shown, Figure 3b This is a schematic diagram of the architecture of a data generation system provided in an embodiment of this specification. It includes a parameter configuration component, a database generation component, a database cluster component, and a distributed statistics component. The functions of each component are as follows:

[0069] The parameter configuration component is used by users to input relevant parameters, including database type, database size, database partitioning strategy, and data skew parameters.

[0070] The database generation component reads the aforementioned parameters and sends a request to the database cluster simulation component to create data tables. Based on the data model, it then generates different amounts of data for each warehouse and transmits the data to the database cluster.

[0071] As the data receiver, the database cluster component will route the data to the corresponding database nodes according to the pre-defined data partitioning rules.

[0072] The distributed statistics component counts the amount of data on each node in the database cluster, and uses the statistical results for display, verification, and adjustment.

[0073] Secondly, based on the same idea, one or more embodiments of this specification also provide apparatus and devices corresponding to the above-described methods, such as... Figure 4 , Figure 5 As shown.

[0074] like Figure 4 As shown, Figure 4 This is a schematic diagram of a device for generating imbalanced data according to an embodiment of this specification. The device includes:

[0075] Module 401 retrieves the number of first nodes and data skew parameters;

[0076] The determination module 403 determines the number of second nodes corresponding to each first node based on the number of first nodes and the data skew parameter;

[0077] The generation module 405 generates a data table for each first node corresponding to the number of second nodes;

[0078] The routing module 407 routes the data partitions in the data table to each database node.

[0079] Optionally, the determining module 403 determines a first node distribution including the number of first nodes and the data skew parameter; determines the number of the first node, and determines the number of second nodes corresponding to the first node based on the number and the first node distribution.

[0080] Optionally, the routing module 407 obtains a partitioning strategy, generates multiple data partitions based on the partitioning strategy for the data partitions in the data table, and routes the multiple data partitions to each database node.

[0081] Optionally, the routing module 407 can generate multiple data partitions by partitioning the data table according to the sequence number of the primary key, or by partitioning the data table according to the hash value of the primary key.

[0082] Optionally, the routing module 407 performs non-uniform partitioning based on the primary key number of the data table, generating multiple data partitions with non-uniform data volume.

[0083] Optionally, the device further includes an adjustment module 409, which statistically analyzes the distribution of data volume in each database and compares the distribution of data volume in the database with the distribution of the first node.

[0084] Optionally, the adjustment module 409 adjusts the data skew parameter according to the comparison result so that the distribution of the number of numbers in the database is consistent with the distribution of the number of the first node.

[0085] Optionally, in the device, the first node is a warehouse, and the second node is the region corresponding to the warehouse.

[0086] In the third aspect, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. The device includes:

[0087] At least one processor; and,

[0088] A memory communicatively connected to the at least one processor; wherein,

[0089] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0090] In a fourth aspect, based on the same idea, embodiments of this specification also provide a non-volatile computer storage medium corresponding to the above method, storing computer-executable instructions, which, when read by a computer from the storage medium, cause one or more processors to execute the method described in the first aspect.

[0091] In the 1990s, improvements to a technology could be clearly categorized into hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) and software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be seen as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0092] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0093] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0094] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0095] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0100] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0101] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0102] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0104] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0105] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0106] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for generating imbalanced data, the method comprising: Obtain the number of first nodes and the data skew parameter, wherein the data skew parameter characterizes the degree of imbalance in the amount of data in each database node as desired; The number of second nodes corresponding to each first node is determined based on the number of first nodes and the data skew parameter. The first node represents a business processing object, and the second node represents the object served by the corresponding business processing object. Generate a data table corresponding to the number of second nodes for each first node; The data in the data table is partitioned and routed to each database node to generate unevenly distributed data in the database cluster.

2. The method as described in claim 1, wherein, The number of second nodes corresponding to each first node is determined based on the number of first nodes and the data skew parameter, including: Determine the first node distribution, which includes the number of first nodes and the data skew parameter; Determine the number of the first node, and determine the number of second nodes corresponding to the first node based on the number and the distribution of the first node.

3. The method as described in claim 1, wherein, The data partitions in the data table are routed to each database node, including: Obtain the partitioning strategy, and generate multiple data partitions based on the partitioning strategy in the data table; The multiple data partitions are routed to each database node.

4. The method of claim 3, wherein, Multiple data partitions are generated from the data in the data table according to the partitioning strategy, including: Multiple data partitions can be generated by partitioning the data table based on the primary key sequence number, or by partitioning the data table based on the hash value of the primary key.

5. The method of claim 4, wherein, Multiple data partitions are generated based on the primary key sequence number of the data table, including: The data table is partitioned non-uniformly based on its primary key number, resulting in multiple data partitions with uneven data volumes.

6. The method of claim 2, wherein, The method further includes: The distribution of data volume in each database is statistically analyzed, and the distribution of data volume in the databases is compared with the distribution of the first node.

7. The method of claim 6, wherein, The method further includes: The data skew parameter is adjusted based on the comparison results so that the distribution of the number of numbers in the database is consistent with the distribution of the number of numbers in the first node.

8. The method of claim 1, wherein, In an e-commerce transaction scenario, the first node is the warehouse, and the second node is the region corresponding to the warehouse.

9. An apparatus for generating imbalanced data, the apparatus comprising: The acquisition module acquires the number of first nodes and data skew parameters, wherein the data skew parameters characterize the degree of imbalance in the amount of data in each database node as desired. The determination module determines the number of second nodes corresponding to each first node based on the number of first nodes and the data skew parameter. The first node represents a business processing object, and the second node represents the object served by the corresponding business processing object. The generation module generates a data table for each first node corresponding to the number of second nodes; The routing module routes the data partitions in the data table to each database node, thereby generating unevenly distributed data in the database cluster.

10. An electronic device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, such that the at least one processor can perform the method as described in any one of claims 1 to 8.

Images