Data table distribution method and device, electronic equipment and storage medium

Abstract

The invention discloses a data table distribution method and device, electronic equipment and a storage medium. The method comprises the following steps: determining target mapping information stored in a Redis cluster in a source database; the target mapping information is a mapping relationship between the value of the associated field and the candidate database, the target mapping information is determined according to the value of the associated field corresponding to the association table, the value of the distribution field corresponding to the association table and the target database, and the association table is a table which is associated when the main table is distributed and contains the distribution field; the value of a target associated field of a target main table in the source database is obtained, a target database is determined according to the value of the target associated field and target mapping information, the target main table is distributed to the target database, and the target database is one of the candidate databases. According to the technical scheme, the problem of low query efficiency caused by the fact that a large number of table association relationships need to be set in the source database is solved, and the distribution efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer database data migration and synchronization, and particularly to a method, apparatus, electronic device, and storage medium for distributing data tables. Background Art

[0002] Today, with the rapid development of digital technology, data is becoming an increasingly important asset for enterprises and institutions, and the need for data to be migrated and synchronized between different business systems in enterprises and institutions is also becoming more widespread.

[0003] With the complexity of data, the currently commonly used mode for data migration and synchronization is the associated condition distribution mode. The associated condition distribution mode distributes and synchronizes multiple data tables with an associated relationship in the source database to multiple target databases, and the distribution fields do not exist in each data table to be synchronized.

[0004] For the associated condition distribution mode, since the distribution fields are not in the currently synchronized table, it is necessary to associate with other tables containing the distribution fields. Therefore, it is necessary to set table association relationships in the source database, so as to distribute the synchronized data tables from the source database to the target database according to the table association relationships. However, with the increase in the amount of data, the number of data tables will also increase, so a large number of table association relationships need to be set in the source database, which greatly affects the query efficiency. In addition, data is a dynamic change process. With the increase in the amount of data, there will also be errors in setting table association relationships, thus affecting the efficiency and accuracy of distribution. Summary of the Invention

[0005] The present invention provides a method, apparatus, electronic device, and storage medium for distributing data tables to solve the problem of low query efficiency caused by the need to set a large number of table association relationships in the source database, and to improve the efficiency and accuracy of distribution.

[0006] According to an aspect of the present invention, there is provided a method for distributing data tables, the method including:

[0007] Determine the target mapping information stored in the Redis cluster in the source database; the target mapping information is the mapping relationship between the values of the associated fields and the candidate databases, and the target mapping information is determined according to the values of the associated fields corresponding to the associated table, the values of the distribution fields corresponding to the associated table, and the target databases, and the associated table is the table containing the distribution fields associated when the main table is distributed;

[0008] Obtain the values of the target associated fields of the target main table in the source database, determine the target database according to the values of the target associated fields and the target mapping information, and distribute the target main table to the target database, where the target database is one of the candidate databases.

[0009] According to another aspect of the present invention, there is provided a data table distribution device, which includes:

[0010] An information determination module for determining target mapping information stored in a Redis cluster in a source database; the target mapping information is a mapping relationship between the values of associated fields and candidate databases, and the target mapping information is determined according to the values of the associated fields corresponding to the association table, the values of the distribution fields corresponding to the association table, and the target database, and the association table is a table containing distribution fields associated when the main table is distributed;

[0011] A distribution module for obtaining the values of the target associated fields of the target main table in the source database, determining the target database according to the values of the target associated fields and the target mapping information, and distributing the target main table to the target database, where the target database is one of the candidate databases.

[0012] According to another aspect of the present invention, there is provided an electronic device, which includes:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data table distribution method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the data table distribution method according to any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention determines the target mapping information stored in the Redis cluster in the source database; the target mapping information is determined according to the values of the associated fields corresponding to the association table, the values of the distribution fields corresponding to the association table, and the target database, and the association table is the table containing the distribution fields associated when the main table is distributed; because the target mapping information is the mapping relationship between the values of the associated fields and the candidate databases, when determining the target database subsequently, it is not necessary to first determine the value of the target associated field of the target main table, then call the corresponding association table according to the value of the target associated field of the target main table, find the distribution field in the association table, and then determine the target database according to the value of the distribution field. Instead, the value of the target associated field of the target main table in the source database is obtained, and the target database is directly queried from the target mapping information according to the value of the target associated field, and the target main table is distributed to the target database, and the target database is one of the candidate databases, which solves the problem of low query efficiency caused by the need to set a large number of table association relationships in the source database, and improves the efficiency and accuracy of distribution.

[0018] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Brief Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 is a flowchart of a method for distributing data tables according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of the overall process of distributing data tables applicable to the embodiment of the present invention;

[0022] Figure 3 is a schematic structural diagram of a device for distributing data tables according to an embodiment of the present invention;

[0023] Figure 4 is a schematic structural diagram of an electronic device for implementing the method for distributing data tables according to an embodiment of the present invention. Detailed Embodiments

[0024] To enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solution in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0025] It should be noted that the terms "first", "second", "target", "candidate", etc. in the description and claims of the present invention and the above accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] Figure 1 As shown in the flowchart of a method for distributing a data table provided in an embodiment of the present invention, this embodiment is applicable to the case of distributing a data table in an associated condition distribution mode from a source database to a target database. This method can be executed by a data table distribution device, which can be implemented in the form of hardware and / or software, and the data table distribution device can be configured in any electronic device with network communication functions. As Figure 1 shown, the method for distributing a data table of the present invention includes:

[0028] S110. Determine the target mapping information stored in the Redis cluster in the source database; the target mapping information is the mapping relationship between the values of the associated fields and the candidate databases, and the target mapping information is determined according to the values of the associated fields corresponding to the associated table, the values of the distribution fields corresponding to the associated table, and the target database. The associated table is the table containing the distribution fields associated when the main table is distributed.

[0029] Among them, a large number of associated tables and main tables are stored in the source database. The main table is the table to be distributed and synchronized currently. The distribution fields involved in the distribution conditions are not in this table, and it is necessary to perform an associated query with other large tables with distribution fields to determine the target database. The main table includes the values of the associated fields. The associated table is the table containing the distribution fields associated with the main table during distribution. The associated table includes the values of the associated fields and the values of the distribution fields. There is a corresponding association relationship between the values of the distribution fields and the target database. The distribution field is the field involved in the distribution conditions in the associated table.

[0030] For example, as shown in Table 1 and Table 2, Table 2 is the main table, Table 1 is the associated table. The Deptno field in Table 1 is the associated field, and the dname field is the distribution field; the deptno in Table 2 is the associated field.

[0031] Table 1

[0032]

[0033] Table 2

[0034]

[0035] Specifically, there is a corresponding association relationship between the values of the distribution fields in the associated table and the target database. After determining the values of the associated fields corresponding to the associated table and the values of the distribution fields corresponding to the associated table, the mapping relationship between the values of the associated fields corresponding to the associated table and the target database can be established according to the corresponding association relationship, so as to accurately determine the target mapping information and store the target mapping information in the Redis cluster.

[0036] Optionally, the target mapping information can be represented in the form of key-value pairs, such as "value of the associated field - number of the target database". Specifically, the Redis cluster disperses the target mapping information to multiple nodes through data sharding. The method of Hash Slot is used to allocate data. There are a total of 16384 hash slots in the Redis cluster. For example, when a key-value pair is to be stored in the cluster, first calculate the CRC16 algorithm for the key to obtain a 16-bit number, and then take the modulus of this number by 16384. The result obtained is the hash slot number where this key-value pair should be stored. Each node is responsible for a part of the hash slots. For example, node A may be responsible for hash slots numbered 0 - 5000, and node B is responsible for hash slots numbered 5001 - 10000, etc. This method enables the data to be evenly distributed on each node, facilitating data management and query.

[0037] In this embodiment, optionally, determining the target mapping information stored in the Redis cluster in the source database includes steps A1 - A4:

[0038] Step A1: Control the values of the associated fields and distribution fields of the association table extracted by the first extraction thread, and send the values of the associated fields and distribution fields of the association table to the first cache queue.

[0039] Among them, the first extraction thread can be a thread for extracting the association table. The first cache queue can be a list for storing the values of the associated fields and distribution fields of the association table.

[0040] In this embodiment, optionally, before sending the values of the associated fields and distribution fields of the association table extracted by the extraction thread to the cache queue, the method includes: determining the data volume carried by each association table in the source database and the total number of association tables stored in the source database; starting at least one first extraction thread according to the data volume carried by the association table and the total number of association tables; each first extraction thread is connected to a first cache queue; starting at least one first loading thread according to the idle state of the first cache queue and the writing speed of the Redis cluster; there is a one-to-one correspondence between the first extraction thread, the first cache queue, and the first loading thread. This embodiment ensures the normal startup of the first extraction thread and the first loading thread, so as to effectively execute the caching process of the target mapping information.

[0041] Among them, the data volume carried by the association table can be understood as the amount of data stored in the association table. The total number of association tables is the number of association tables stored in the source database.

[0042] Step A2: When there is a first loading thread in an idle state, control the first loading thread to extract the values of the associated fields and distribution fields of the association table from the first cache queue.

[0043] Among them, the first loading thread can be a thread for extracting the values of the associated fields and distribution fields of the association table from the first cache queue.

[0044] Specifically, judging whether there is a first loading thread in an idle state is to accurately control the first loading thread to extract the values of the associated fields and distribution fields of the association table from the first cache queue, and avoid invalidly extracting the values of the associated fields and distribution fields of the association table from the first cache queue when there is no first loading thread in an idle state.

[0045] Step A3: Obtain the corresponding association relationship between the value of the distribution field and the candidate database, and determine the candidate database corresponding to the value of the distribution field according to the corresponding association relationship.

[0046] Step A4: Convert the value of the distribution field into the number information of the candidate database, establish the target mapping information between the number information of the candidate database and the value of the associated field of the association table, and write the target mapping information into the Redis cluster.

[0047] In the technical solution of this embodiment, the values of the associated fields and the distribution fields of the associated table extracted by the first extraction thread are controlled, and the values of the associated fields and the distribution fields of the associated table are sent to the first cache queue, so as to facilitate the subsequent extraction of the required information quickly and conveniently; when the state of the first loading thread is idle, the first loading thread is controlled to extract the values of the associated fields and the distribution fields of the associated table from the first cache queue to ensure the efficient extraction and processing of data. Further, the corresponding association relationship between the value of the distribution field and the candidate databases is obtained, and the candidate database corresponding to the value of the distribution field is determined according to the corresponding association relationship; the value of the distribution field is converted into the number information of the candidate database, and the target mapping information between the number information of the candidate database and the value of the associated field of the associated table is established, and the target mapping information is written into the Redis cluster to accurately determine the target mapping information, and the Redis cluster is used for storage to facilitate data management and query.

[0048] S120. Obtain the value of the target associated field of the target main table in the source database, determine the target database according to the value of the target associated field and the target mapping information, and distribute the target main table to the target database, where the target database is one of the candidate databases.

[0049] Among them, the target main table can be understood as the main table that needs to be distributed currently among all the main tables in the source database.

[0050] Specifically, the process of distributing the main table in the source database to the candidate database is as follows: determine the target main table to be distributed currently, obtain the value of the target associated field of the target main table, call the target mapping information, and query the corresponding target database from the target mapping information according to the value of the target associated field, so as to distribute the target main table to the target database.

[0051] In this embodiment, optionally, obtaining the value of the target associated field of the target main table in the source database and determining the target database according to the value of the target associated field and the target mapping information includes steps B1 - B3:

[0052] Step B1. Control the second extraction thread to extract the value of the associated field of the target main table, and send the value of the associated field of the target main table to the second cache queue.

[0053] Among them, the second extraction thread can be a thread for extracting the main table. The second cache queue can be a list for storing the values of the associated fields of the main table.

[0054] In this embodiment, optionally, before controlling the second extraction thread to extract the values of the associated fields of the target main table, the method includes: determining the data volume carried by each main table in the source database and the total number of main tables stored in the source database; starting at least one second extraction thread according to the data volume carried by the main table and the total number of main tables; each second extraction thread connecting to a second cache queue; starting at least one second loading thread according to the idle state of the second cache queue and the writing speed of the candidate database; there is a one-to-one correspondence between the second extraction thread, the second cache queue, and the second loading thread. This embodiment ensures the normal start of the second extraction thread and the second loading thread, so as to effectively distribute the main table.

[0055] Among them, the data volume carried by the main table can be understood as the data volume stored in the main table. The total number of main tables is the number of main tables stored in the source database.

[0056] Step B2: When the state of a second loading thread is idle, control the second loading thread to extract the values of the associated fields of the target main table from the second cache queue.

[0057] Among them, the second loading thread can be a thread that extracts the values of the associated fields of the main table from the second cache queue.

[0058] Specifically, determining whether there is a second loading thread in an idle state is to accurately control the second loading thread to extract the values of the associated fields of the target main table from the second cache queue, and to avoid invalidly extracting the values of the associated fields of the target main table from the second cache queue when there is no second loading thread in an idle state.

[0059] Step B3: Query the target mapping information according to the values of the associated fields of the target main table to determine the target database.

[0060] Specifically, after determining the target database according to the values of the target associated fields and the target mapping information, before distributing the target main table to the target database, the method includes establishing a connection relationship between the target database and the second loading thread, so that after determining the target database, the channel between the target database and the second loading thread can be accurately connected according to the connection relationship, that is, according to the connection relationship, the target main table can be accurately distributed to the target database.

[0061] In the technical solution of this embodiment, the second extraction thread is controlled to extract the values of the associated fields of the target main table and send the values of the associated fields of the target main table to the second cache queue, so as to facilitate the subsequent extraction of the required information quickly and conveniently; when the status of the second loading thread is idle, the second loading thread is controlled to extract the values of the associated fields of the target main table from the second cache queue; ensuring efficient data extraction and processing. Further, the target database is determined by querying the target mapping information according to the values of the associated fields of the target main table, realizing the accurate determination of the target database and improving the efficiency of determining the target database.

[0062] In the technical solution of the embodiment of the present invention, the target mapping information stored in the Redis cluster in the source database is determined; the target mapping information is determined according to the values of the associated fields corresponding to the associated table, the values of the distribution fields corresponding to the associated table, and the target database. The associated table is the table containing the distribution fields associated when the main table is distributed; because the target mapping information is the mapping relationship between the values of the associated fields and the candidate databases, therefore, when determining the target database subsequently, it is not necessary to first determine the values of the target associated fields of the target main table, and then call the corresponding associated table according to the values of the target associated fields of the target main table to find the distribution fields in the associated table, and then determine the target database according to the values of the distribution fields. Instead, the values of the target associated fields of the target main table in the source database are obtained, and the target database is directly queried from the target mapping information according to the values of the associated fields, and the target main table is distributed to the target database. The target database is one of the candidate databases, solving the problem of low query efficiency caused by the need to set a large number of table association relationships in the source database, and improving the efficiency and accuracy of distribution.

[0063] Embodiment 2

[0064] Figure 2 It is a schematic diagram of the overall process of distributing data tables applicable to the embodiment of the present invention. The technical solution of this embodiment further describes the process of distributing the data tables in the associated condition distribution mode from the source database to the target database on the basis of the above embodiment.

[0065] As Figure 2 shown, the entire process of distributing and synchronizing data tables can be divided into two stages. Stage P1: The caching process of target mapping information. Stage P2: The main table distribution and synchronization process.

[0066] Stage P1: The caching process of target mapping information.

[0067] Step P1S1: According to the data volume carried by the associated table, start multiple first extraction threads according to the number of shards of the source database or a finer granularity. When each first extraction thread starts, a connection to the source database is established.

[0068] Step P1S2: Start an appropriate number of first loading threads based on the free status of the first buffer queue and the speed of writing to the Redis cluster. When each first loading thread starts, establish a connection with the Redis cluster to prepare for data transmission.

[0069] Step P1S3: Each first extraction thread queries the values of the associated fields and distribution fields of the association table within the record range assigned to the current first extraction thread, and sends the values of the associated fields and distribution fields of the association table to the first buffer queue.

[0070] When the first loading thread is idle, it takes a row of unprocessed values of the associated fields and distribution fields of the association table from the first buffer queue, converts the value of the distribution field into the number of the candidate database according to the corresponding association relationship between the value of the distribution field and the candidate database, uses the key-value pair information of "value of the associated field - number of the candidate database" as the target mapping information, and writes the target mapping information into the Redis cluster.

[0071] Repeat Step P1S3 and Step P1S4 until the values of the associated fields and distribution fields of the association table assigned to all threads are processed to obtain the target mapping information, and the target mapping information is written into the Redis cluster, and the data in the Redis cluster is marked as available.

[0072] Phase P2: Master table distribution synchronization process.

[0073] Step P2S1: Start multiple second extraction threads according to the data volume carried by the master table, in accordance with the number of shards of the source database or at a finer granularity. When each second extraction thread starts, establish a connection with the source database.

[0074] Step P2S2: Start an appropriate number of loading threads based on the free status of the second buffer queue and the speed of writing to the candidate database. When each second loading thread starts, establish a connection with the Redis cluster and a connection with the target database to prepare for data transmission.

[0075] Step P2S3: Each second extraction thread queries the value of the associated field of the target master table within the record range assigned to the current second extraction thread, and sends the value of the associated field of the target master table to the second buffer queue.

[0076] When each second loading thread is idle, it takes a row of unprocessed values of the associated fields of the target master table from the second buffer queue, queries the Redis cluster according to the value of the associated field to obtain the number of the target database to be written, then obtains the connection of the target database, and loads the target master table into this target database.

[0077] Step P2S5: Repeat Step P2S3 and Step P2S4 until the values of the associated fields of the target master table assigned to all threads match the corresponding target database, and the target master table is distributed to the corresponding target database.

[0078] The technical solution of the embodiment of the present invention accurately determines the mapping relationship between the value of the associated field and the candidate database through the caching process of the target mapping information; so that during the master table distribution and synchronization process, after obtaining the value of the associated field of the target master table, the target database of the target master table can be accurately located from the candidate database according to the mapping relationship between the value of the associated field and the candidate database, realizing efficient query of the target database, solving the problem of low query efficiency caused by the need to set a large number of table association relationships in the source database, and improving the efficiency and accuracy of distribution.

[0079] Embodiment III

[0080] Figure 3 It is a structural schematic diagram of a data table distribution device provided by an embodiment of the present invention. This embodiment is applicable to the situation of distributing a data table in an associated condition distribution mode from a source database to a target database. The data table distribution device can be implemented in the form of hardware and / or software, and the data table distribution device can be configured in any electronic device with network communication functions. As Figure 3 shown, the data table distribution device of the present invention includes:

[0081] An information determination module 210, configured to determine target mapping information stored in a Redis cluster in the source database; the target mapping information is the mapping relationship between the value of the associated field and the candidate database, and the target mapping information is determined according to the value of the associated field corresponding to the associated table, the value of the distribution field corresponding to the associated table, and the target database. The associated table is a table containing a distribution field associated when the master table is distributed;

[0082] A distribution module 220, configured to obtain the value of the target associated field of the target master table in the source database, determine the target database according to the value of the target associated field and the target mapping information, and distribute the target master table to the target database, where the target database is one of the candidate databases.

[0083] Based on the above embodiment, optionally, the information determination module is configured to:

[0084] Control the values of the associated field and the distribution field of the associated table extracted by the first extraction thread, and send the values of the associated field and the distribution field of the associated table to the first cache queue;

[0085] When the status of the first loading thread is idle, control the first loading thread to extract the values of the associated fields and distribution fields of the association table from the first cache queue;

[0086] Obtain the corresponding association relationship between the value of the distribution field and the candidate databases, and determine the candidate database corresponding to the value of the distribution field according to the corresponding association relationship;

[0087] Convert the value of the distribution field into the number information of the candidate database, establish the target mapping information between the number information of the candidate database and the value of the associated field of the association table, and write the target mapping information into the Redis cluster.

[0088] On the basis of the above embodiments, optionally, the information determination module includes a first thread determination unit, and the first thread determination unit is used to determine the data volume carried by each association table in the source database and the total number of association tables stored in the source database; according to the data volume carried by the association table and the total number of association tables, start at least one first extraction thread; each first extraction thread is connected to a first cache queue; according to the idle status of the first cache queue and the writing speed of the Redis cluster, start at least one first loading thread; there is a one-to-one correspondence between the first extraction thread, the first cache queue and the first loading thread.

[0089] On the basis of the above embodiments, optionally, the distribution module includes a target database determination unit, and the target database determination unit is used to: control the second extraction thread to extract the value of the associated field of the target main table and send the value of the associated field of the target main table to the second cache queue; when the status of the second loading thread is idle, control the second loading thread to extract the value of the associated field of the target main table from the second cache queue; query the target mapping information according to the value of the associated field of the target main table to determine the target database.

[0090] On the basis of the above embodiments, optionally, the distribution module includes a second thread determination unit, and the second thread determination unit is used to: determine the data volume carried by each main table in the source database and the total number of main tables stored in the source database; according to the data volume carried by the main table and the total number of main tables, start at least one second extraction thread; each second extraction thread is connected to a second cache queue; according to the idle status of the second cache queue and the writing speed of the candidate database, start at least one second loading thread; there is a one-to-one correspondence between the second extraction thread, the second cache queue and the second loading thread.

[0091] Based on the above embodiments, optionally, the distribution module includes a connection relationship determination unit, and the connection relationship determination unit is configured to: establish a connection relationship between the target database and the second loading thread.

[0092] Based on the above embodiments, optionally, the distribution module is further configured to: distribute the target main table to the target database according to the connection relationship.

[0093] The data table distribution device provided by the embodiments of the present invention can execute the data table distribution method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0094] Embodiment 4

[0095] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0096] Figure 4 The structural schematic diagram of the electronic device that can be used to implement the data table distribution method of the embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0097] As Figure 4 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by at least one processor, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0098] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0099] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for distributing data tables.

[0100] In some embodiments, the method for distributing data tables can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for distributing data tables described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the method for distributing data tables in any other suitable manner (e.g., by means of firmware).

[0101] The various embodiments of the systems and technologies described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), system-on-chip systems (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, and the programmable processor can be a special or general programmable processor, can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] A computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer program can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0103] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0106] The computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0107] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0108] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for distributing a data table, characterized in that: The method comprises: Determine the target mapping information stored in the Redis cluster in the source database; the target mapping information is the mapping relationship between the value of the associated field and the candidate database, and the target mapping information is determined according to the value of the associated field corresponding to the associated table, the value of the distribution field corresponding to the associated table, and the target database. The associated table is a table containing the distribution field associated with the main table when distributing; The value of the target associated field of the target main table in the source database is obtained, the target database is determined according to the value of the target associated field and the target mapping information, and the target main table is distributed to the target database, which is one of the candidate databases.

2. The method according to claim 1, characterized in that: Determine the target mapping information stored in the Redis cluster in the source database, including: Controlling the first extraction thread to extract the value of the association field and the value of the distribution field of the association table, and sending the value of the association field and the value of the distribution field of the association table to the first cache queue; When the state of the first loading thread is idle, controlling the first loading thread to extract the value of the association field and the value of the distribution field of the association table from the first cache queue; Acquire a corresponding association relationship between a value of a distribution field and a candidate database, and determine a candidate database corresponding to the value of the distribution field according to the corresponding association relationship; The value of the distribution field is converted into the number information of the candidate database, target mapping information is established between the number information of the candidate database and the value of the association field of the association table, and the target mapping information is written into the Redis cluster.

3. The method according to claim 2, characterized in that Before sending the value of the association field and the value of the distribution field of the association table extracted by the extraction thread to the cache queue, the method includes: Determine the amount of data carried by each association table in the source database and the total number of association tables stored in the source database; According to the amount of data carried by the association table and the total number of the association tables, at least one first extraction thread is started; each first extraction thread is connected to a first cache queue; At least one first loading thread is started according to the idle state of the first cache queue and the write speed of the Redis cluster; there is a one-to-one correspondence between the first extraction thread, the first cache queue and the first loading thread.

4. The method according to claim 1, characterized in that: Acquiring the value of the target associated field of the target main table in the source database, and determining the target database according to the value of the target associated field and the target mapping information, including: Controlling the second extraction thread to extract the value of the associated field of the target main table, and sending the value of the associated field of the target main table to the second cache queue; When the state of the second loading thread is idle, controlling the second loading thread to extract the value of the associated field of the target main table from the second cache queue; The target mapping information is queried according to the value of the associated field of the target main table to determine the target database.

5. The method according to claim 4, characterized in that Before controlling the second extraction thread to extract the value of the associated field of the target main table, the method includes: Determine the amount of data carried by each main table in the source database and the total number of main tables stored in the source database; According to the amount of data carried by the main table and the total number of the main tables, at least one second extraction thread is started; each second extraction thread is connected to a second cache queue; At least one second loading thread is started according to the idle state of the second cache queue and the writing speed of the candidate database; there is a one-to-one correspondence between the second extraction thread, the second cache queue and the second loading thread.

6. The method according to claim 4, characterized in that After determining the target database according to the value of the target association field and the target mapping information, and before distributing the target master table to the target database, the method includes: A connection relationship between the target database and the second loading thread is established.

7. The method according to claim 6, characterized in that Distributing the target master table to the target database includes: The target master table is distributed to the target database according to the connection relationship.

8. A data table distribution device, characterized in that: The device comprises: An information determination module is used to determine the target mapping information stored in the Redis cluster in the source database; the target mapping information is the mapping relationship between the value of the associated field and the candidate database, and the target mapping information is determined according to the value of the associated field corresponding to the associated table, the value of the distribution field corresponding to the associated table, and the target database. The associated table is a table containing the distribution field associated with the main table when distributing; The distribution module is used to obtain the value of the target associated field of the target main table in the source database, determine the target database according to the value of the target associated field and the target mapping information, and distribute the target main table to the target database, which is one of the candidate databases.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data table distribution method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data table distribution method according to any one of claims 1 to 7 when executed.

Images