Data association method, apparatus, device, and storage medium

Abstract

The present disclosure provides a data association method, device, equipment and storage medium, which can be applied to the technical field of data processing. The method comprises: setting an index variable for a driven table and setting an association threshold; determining whether the association variable in the driven table contains the index variable and whether the driven table takes the index variable as a distribution key; wherein the data quantity of the driven table is less than that of the driven table; in response to the association variable containing the index variable and the driven table taking the index variable as the distribution key, performing deduplication processing on the association variable corresponding to the index variable in the driven table; counting the number of records of the index variable corresponding to the deduplicated association variable; in response to the number of records being not greater than the association threshold, associating the deduplicated driven table with the driven table to form a temporary table; wherein the temporary table takes the deduplicated association variable as the distribution key; and associating the driven table with the temporary table by the original association variable in the driven table to obtain a final connection table.

Description

Technical Field

[0001] This disclosure relates to the field of data processing and can be applied to the field of financial technology, and in particular to a data association method, apparatus, device, medium and program product. Background Technology

[0002] Greenplum is a relational database for data warehouse applications, and it commonly uses three join methods: hash join, nested circular join, and combined sort join. Currently, database-based analytics tools generally provide join steps for users to choose from. This approach relies on the user's data analysis skills and requires a certain level of expertise in selecting the join method, which may lead to inaccurate choices. Furthermore, setting fixed join methods for specific scenarios fails to support flexible join analysis scenarios. Summary of the Invention

[0003] In view of the above problems, this disclosure provides data association methods, apparatus, devices, media and program products.

[0004] According to a first aspect of this disclosure, a data association method is provided, comprising: setting an index variable and an association threshold for a driven table; determining whether an association variable in a driving table contains the index variable and whether the driving table uses the index variable as a distribution key; wherein the data volume of the driving table is less than the data volume of the driven table; in response to the association variable containing the index variable and the driving table using the index variable as a distribution key, performing deduplication processing on the association variables in the driving table corresponding to the index variable; counting the number of records corresponding to the index variable after deduplication; in response to the number of records not being greater than the association threshold, associating the deduplicated driving table with the driven table to form a temporary table; wherein the temporary table uses the deduplicated association variable as a distribution key; and associating the driving table with the temporary table using the original association variables in the driving table to obtain a final join table.

[0005] According to embodiments of this disclosure, setting index variables for the driven table includes: setting B-tree index variables for the driven table based on the distribution key of the database.

[0006] According to embodiments of this disclosure, setting the association threshold includes setting association thresholds of different sizes for driven tables with different data volumes and / or different index distributions.

[0007] According to embodiments of this disclosure, setting the association threshold further includes: obtaining a first association time for the driving table to associate with the driven table, wherein the first association time is the time required for the driving table to directly associate with the driven table; obtaining a second association time for the driving table to associate with the driven table, wherein the second association time is the time required for the driving table to associate with the driven table using the data association method described above; obtaining the maximum allowed association threshold for the driven table; and adjusting the current association threshold of the driven table based on the first association time, the second association time, and the maximum association threshold.

[0008] According to an embodiment of this disclosure, adjusting the current association threshold of the driven table based on the first association time, the second association time, and the maximum association threshold includes: calculating the time optimization rate of the driving table's association with the driven table based on the first association time and the second association time; and increasing the current association threshold of the driven table in response to the time optimization rate being greater than a preset optimization rate and the current association threshold of the driven table being less than the maximum association threshold.

[0009] According to embodiments of this disclosure, the step of calculating the time optimization rate of the driving table's association with the driven table based on the first association time and the second association time includes: calculating the ratio between the first association time and the second association time as the time optimization rate; or, calculating the absolute value of the difference between the first association time and the second association time as the time optimization rate; or, calculating the ratio between the absolute value of the difference between the first association time and the second association time and the second association time as the time optimization rate.

[0010] According to embodiments of this disclosure, a nested loop join is used to associate the deduplicated driving table with the driven table; and / or, a nested loop join is used to associate the driving table with the temporary table using the original association variables in the driving table.

[0011] According to embodiments of this disclosure, the method further includes: in response to the association variable containing the index variable and / or the distribution key of the driving table not being the index variable, directly associating the driving table with the driven table in a scale manner; or, in response to the number of records being greater than the association threshold, directly associating the driving table with the driven table in a scale manner.

[0012] A second aspect of this disclosure provides a data association apparatus, comprising: a setting module for setting an index variable and an association threshold for a driven table; a determining module for determining whether an association variable in a driving table contains the index variable and whether the driving table uses the index variable as a distribution key; wherein the data volume of the driving table is less than the data volume of the driven table; a deduplication module for deduplicating the association variables corresponding to the index variable in the driving table in response to the association variable containing the index variable and the driving table using the index variable as a distribution key; a statistics module for counting the number of records corresponding to the index variable after deduplication; a first association module for associating the deduplicated driving table with the driven table to form a temporary table in response to the number of records not exceeding the association threshold; wherein the temporary table uses the deduplicated association variable as a distribution key; and a second association module for associating the driving table with the temporary table using the original association variables in the driving table to obtain a final join table.

[0013] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0014] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0015] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0016] According to the data association method, apparatus, device, medium, and program products provided in this disclosure, by setting index variables and association thresholds for the driven table, the association variables corresponding to the index variables in the driving table are deduplicated based on the index variables. Then, data association is performed based on the number of records in the index variables corresponding to the deduplicated association variables and the size of the association threshold. Since data association is based on the number of records in the index variables corresponding to the deduplicated association variables and the size of the association threshold, it eliminates the need for manual selection of specific association methods, and the association methods are not fixed. Therefore, it at least partially solves the problem that users need a certain level of expertise, manual observation and calculation when choosing association methods, and that it cannot support flexible association analysis scenarios. This achieves the technical effect of accurate and flexible data association. Attached Figure Description

[0017] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of data association methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0019] Figure 2 A flowchart illustrating a data association method according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 3 A flowchart illustrating the setting of an association threshold based on association time according to an embodiment of the present disclosure is shown schematically.

[0021] Figure 4 A flowchart illustrating a data association method according to another embodiment of this disclosure is shown schematically;

[0022] Figure 5 This illustration schematically shows the adoption of an embodiment according to the present disclosure. Figures 2 to 4 The average duration distribution of data association after the data association method is shown in the figure.

[0023] Figure 6 A schematic block diagram of a data association apparatus according to an embodiment of the present disclosure is shown; and

[0024] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a data association method according to an embodiment of the present disclosure. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0029] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0030] In the technical solution disclosed herein, the acquisition, collection, storage, use, processing, transmission, provision, disclosure, and application of data all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0031] Embodiments of this disclosure provide a data association method, including: setting an index variable for a driven table and setting an association threshold. The method involves determining whether the association variable in the driving table contains an index variable and whether the driving table uses the index variable as a distribution key, wherein the data size of the driving table is smaller than the data size of the driven table. In response to the association variable containing an index variable and the driving table using the index variable as a distribution key, deduplication is performed on the association variables in the driving table corresponding to the index variable. The number of records corresponding to the index variable of the deduplicated association variable is counted. In response to the number of records not exceeding the association threshold, the deduplicated driving table is associated with the driven table to form a temporary table, wherein the temporary table uses the deduplicated association variable as a distribution key. The driving table is then associated with the temporary table using the original association variables in the driving table to obtain the final joined table.

[0032] Figure 1 The diagram illustrates an application scenario of the data association method and apparatus according to embodiments of the present disclosure.

[0033] like Figure 1 As shown, application scenario 100 according to this embodiment may include database 101, network 102, and server 103. Database 101 includes driving tables and driven tables. Network 102 is a medium used to provide communication links between database 101 and server 103, and between driving tables and driven tables.

[0034] Database 101 stores various types of data used for association; for example, the amount of data in the driving table can be less than the amount of data in the driven table.

[0035] Network 102 can include various connection types, such as wired, wireless communication links, or fiber optic cables. Wired connections can be made using cables and any of the following interfaces: Fibre Channel, infrared interface, Type-D data interface, serial interface, USB interface, USB Type-C interface, or Dock interface. Wireless connections can be made using wireless communication, such as Bluetooth, Wi-Fi, Infrared, ZigBee, or other wireless technology standards.

[0036] Server 103 can be a server that provides various services, such as a backend management server for data association between different tables in a database (for example only). The backend management server can set index variables and association thresholds for the driven table, and perform data association between the driving table and the driven table based on the index variables and association thresholds as judgment conditions.

[0037] It should be noted that the data association method provided in this disclosure embodiment can generally be executed by server 103. Correspondingly, the data association device provided in this disclosure embodiment can generally be located in server 103. The data association method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 103 but can communicate with database 101 and / or server 103. Correspondingly, the data association device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 103 but can communicate with database 101 and / or server 103.

[0038] It should be understood that Figure 1 The number of databases, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of databases, networks, and servers can be included.

[0039] The data association method provided in the embodiments of this disclosure can be applied to the field of data processing, specifically to the field of fintech. For example, to meet the data management needs of banks, it is necessary to perform data table association on various data stored in the database to obtain the required data table. There are three commonly used association join methods: hash join, nested loop join, and merge join.

[0040] In a hash join, the right table is scanned first and loaded into a hash table, using the join column as the hash key. Next, the left table is scanned, and the join attribute value of each row found during the scan is used as the hash key to search for matching rows in the hash table.

[0041] In nested loop joins, the right table is scanned once for each row found in the left table. This strategy is the easiest to implement but can be very time-consuming.

[0042] In a combined sort join, each table is sorted according to the join columns before the join begins. The two tables are then scanned in parallel, and matching rows are combined into a join row. Because each table is scanned only once in this type of join, the required sorting can be achieved through an explicit sorting step or by scanning the relationships in the appropriate order using an index on the join key.

[0043] However, in general, bank staff choose the corresponding data association method based on their own data analysis capabilities. In this scenario, due to the differences in the data analysis capabilities of bank staff, the currently selected data association method may not be the optimal one. This may result in inaccurate data connection tables that cannot well meet business needs, and the data association efficiency may be low, affecting the efficient operation of business.

[0044] Based on this scenario, the concept of this disclosure is to automatically determine the status of the left and right tables when a small table is related to a large table, set indexes and association thresholds for the driven table, and transform the initial association statement into a step-by-step process.

[0045] The following will be based on Figure 1 The described scene, through Figures 2-5 The data association method of the disclosed embodiments will be described in detail.

[0046] Figure 2 A flowchart illustrating a data association method according to an embodiment of the present disclosure is shown.

[0047] like Figure 2 As shown, the data association method in this embodiment includes operations S210 to S260.

[0048] In operation S210, index variables and association thresholds are set for the driven table.

[0049] In operation S220, determine whether the associated variable in the driving table contains the index variable and whether the driving table uses the index variable as the distribution key.

[0050] In operation S230, in response to the fact that the associated variable contains the index variable and the driving table uses the index variable as the distribution key, the associated variable corresponding to the index variable in the driving table is deduplicated.

[0051] In operation S240, count the number of records of the index variable corresponding to the deduplicated associated variable.

[0052] In operation S250, in response to the number of records not being greater than the association threshold, the deduplicated driving table is associated with the driven table to form a temporary table.

[0053] In operation S260, the driving table is associated with the temporary table using the original association variables in the driving table to obtain the final join table.

[0054] In the embodiments of this disclosure, the amount of data in the driving table is less than the amount of data in the driven table, and the temporary table uses the deduplicated association variable as the distribution key.

[0055] It should be understood that the driving table can also be understood as the left table, and the driven table as the right table. This data association method leverages the fact that nested loop joins perform significantly better than hash joins in certain scenarios. It collects data information from the left table and uses the parameter settings of the right table to determine whether to use an optimized association method. Therefore, a nested loop join can be used to associate the deduplicated driving table with the driven table. Alternatively, a nested loop join can be used to associate the driving table with a temporary table using the original association variables in the driving table.

[0056] It should be understood that a join variable can also be understood as a join key or join field. The role of a join key or join field is to establish a connection between data tables when data is joined. A distribution key can be understood as the data distributed and stored in a distributed database by specifying a distribution key.

[0057] The data association method according to embodiments of this disclosure sets index variables and association thresholds for the driven table. Based on the index variables, it deduplicates the association variables corresponding to the index variables in the driving table. Then, it performs data association based on the number of records in the index variables corresponding to the deduplicated association variables and the association threshold. Since data association is based on the number of records in the index variables corresponding to the deduplicated association variables and the association threshold, it eliminates the need for manual selection of specific association methods, and the association methods are not fixed. Therefore, it at least partially solves the problem that users need a certain level of expertise to choose an association method, requiring manual observation and calculation, and cannot support flexible association analysis scenarios. This achieves the technical effect of accurate and flexible data association.

[0058] Furthermore, in embodiments of this disclosure, setting an index variable for the driven table may include setting a B-tree index variable for the driven table based on the distribution key of the database.

[0059] B-tree is a balanced m-way search tree that utilizes multiple branch nodes (subtree nodes) to reduce the number of nodes traversed when querying data, thereby saving access time. Therefore, setting a B-tree index variable for the driven table can reduce data join time and improve the efficiency of data joins.

[0060] Furthermore, in embodiments of this disclosure, setting an association threshold for the driven table may include setting association thresholds of different sizes for driven tables with different data volumes and / or different index distributions.

[0061] In the embodiments of this disclosure, due to significant differences in the indexing and data conditions of the driven table (right table), different parameter controls are achieved by configuring different association thresholds (limits on the number of records in the main table) for different right tables (large tables). Examples of association threshold settings are shown in Table 1:

[0062] Table 1

[0063]

[0064] As can be clearly seen from Table 1, different association thresholds were set according to the index and data conditions of different right tables, which can better adapt to the data association of different data tables.

[0065] Furthermore, in the embodiments of this disclosure, the association time and the status of the associated tables between the driving table and the driven table can also be recorded, and an association threshold can be set based on the association time.

[0066] Figure 3 A flowchart illustrating the setting of an association threshold based on association time according to an embodiment of the present disclosure is shown.

[0067] like Figure 3 As shown, setting the association threshold based on the association time may include operations S310 to 340.

[0068] In operation S310, the first association time of the driving table with the driven table is obtained.

[0069] In operation S320, obtain the second association time of the driving table and the driven table.

[0070] In operation S330, obtain the maximum allowed association threshold of the driven table.

[0071] In operation S340, the current association threshold of the driven table is adjusted based on the first association time, the second association time, and the maximum association threshold.

[0072] In the embodiments of this disclosure, the first association time can be understood as the time required for the driving table to directly associate with the driven table. Direct association can be understood as not using the parameter setting information of the driven table for data association provided in the embodiments of this disclosure, but directly associating the driving table with the driven table according to the scale.

[0073] In the embodiments of this disclosure, the associated time can be understood as the time required for the driving table to associate with the driven table using the data association method provided by the embodiments of this disclosure through the parameter setting information of the driven table.

[0074] The association time can be obtained from the log table generated during the data association process. The log table records the fields as shown in Table 2:

[0075] Table 2

[0076] field name illustrate Logical library name Right table logic library Table name Right table name Model Number Model number for this step Number of records in the main table Number of records in the main table Related fields Right table related fields Optimize? Whether to use optimization Runtime Step execution time

[0077] As can be clearly seen from Table 2, the field name "Optimization" in the log table can directly reflect whether it is a direct association or the data association method provided by the embodiments of this disclosure is used, and the field name "Running Time" can directly reflect the first association time or the second association time.

[0078] The embodiments of this disclosure set an association threshold based on the association time. By continuously adjusting the association threshold of the driven table (i.e., the limit on the number of records in the primary table of the driven table) by recording the time of the association steps, the association threshold of the driven table is adjusted to a more suitable record position based on the time change. This makes data association judgment based on the association threshold more accurate, thereby improving the accuracy and efficiency of data association. Furthermore, since the optimization of the association method is controlled by setting an association threshold, and the association threshold can be flexibly adjusted, this data association method can achieve flexible judgment, thus supporting flexible association analysis scenarios for users.

[0079] Preferably, in embodiments of this disclosure, adjusting the current association threshold of the driven table based on the first association time, the second association time, and the maximum association threshold may include the following operations:

[0080] The time optimization rate of the driving table's association with the driven table is calculated based on the first association time and the second association time.

[0081] If the time optimization rate is greater than the preset optimization rate and the current association threshold of the driven table is less than the maximum association threshold, increase the current association threshold of the driven table.

[0082] According to embodiments of this disclosure, the time optimization rate is used as a criterion to determine whether to adjust the association threshold. Since the time optimization rate can more accurately and intuitively reflect the changes in association time compared to association time, the association threshold can be adjusted more efficiently and accurately, thereby improving the data association efficiency.

[0083] Furthermore, calculating the time optimization rate of the driving table's association with the driven table based on the first association time and the second association time may include one of the following operations:

[0084] Calculate the ratio between the first association time and the second association time, and use it as the time optimization rate.

[0085] Alternatively, the absolute value of the difference between the first and second association times can be calculated as the time optimization rate.

[0086] Alternatively, the ratio of the absolute value of the difference between the first and second association times to the second association time can be used as the time optimization rate.

[0087] According to embodiments of this disclosure, since multiple methods for calculating the time optimization rate are provided, users can flexibly choose according to actual application needs or application scenarios.

[0088] To further improve the accuracy of time optimization rate calculation, the first and second association times can also be determined by calculating the average time.

[0089] For example, the segment values ​​for calculating the number of records in the driven table can be set as follows: below 10,000 records is 0, 10,000 to 50,000 records is 1, 50,000 to 100,000 records is 2, and so on, with above 500,000 records being 11. For the same model, the average running time of the data association is calculated for the segment values ​​of the number of records in the driving table and the driven table, and then the time optimization rate is calculated based on the average time.

[0090] For example, the ratio between the first association time and the second association time can be used as the time optimization rate, with a preset optimization rate of 1 and a maximum record limit of 500,000 for the driven table. If the time optimization rate is greater than 1 and the currently set association threshold (record limit) of the driven table is less than or equal to 500,000, the association threshold of the driven table can be increased. It should be understood that the increased association threshold cannot exceed 500,000.

[0091] Figure 4 A flowchart illustrating a data association method according to another embodiment of this disclosure is shown schematically.

[0092] like Figure 4 As shown, the data association method may include, for example, operations S410 to S420.

[0093] In operation S410, in response to the association variable containing an index variable and / or the distribution key of the driving table not being an index variable, the driving table is directly associated with the driven table in a scale manner.

[0094] In operation S420, in response to the number of records exceeding the association threshold, the driving table is directly associated with the driven table in a scale manner.

[0095] According to embodiments of this disclosure, when the data and index conditions of the data table do not meet the conditions for optimizing the association method, a direct association method is used for data association, which enables flexible selection of association methods and supports flexible association analysis scenarios for users.

[0096] To further illustrate the advantages of the data association method provided in the embodiments of this disclosure, a set of data is provided for illustration.

[0097] Figure 5 This illustration schematically shows the adoption of an embodiment according to the present disclosure. Figures 2 to 4 The distribution of average data association duration after the data association method is shown.

[0098] like Figure 5 As shown, curve A represents the number of visitors, and curve B represents the average duration. After optimizing data association based on index variables and association thresholds using the embodiments of this disclosure, the average duration of data association operations is significantly reduced. This indicates that optimizing data association based on index variables and association thresholds significantly improves the efficiency of data association.

[0099] In summary, the data association method of this disclosure, for scenarios involving small tables joining large tables, automatically optimizes the association method based on index variables and association thresholds. While maintaining flexibility during the association process, it solves the problem that users need a certain level of expertise and manual observation and calculation to select the association method. Furthermore, by setting parameters for the driven table and adjusting the record count limit for analysis and computation time, it achieves dynamic support for optimizing parameter settings, thereby realizing accurate and flexible data association.

[0100] Based on the above data association method, this disclosure also provides a data association apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0101] Figure 6 A schematic block diagram of a data association apparatus according to an embodiment of the present disclosure is shown.

[0102] like Figure 6 As shown, the data association device 600 of this embodiment includes a setting module 610, a determining module 620, a deduplication module 630, a statistics module 640, a first association module 650, and a second association module 660.

[0103] The setting module 610 is used to set index variables and association thresholds for the driven table. In one embodiment, the setting module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0104] The determining module 620 is used to determine whether the associated variables in the driving table contain index variables and whether the driving table uses the index variables as the distribution key. The data size of the driving table is smaller than the data size of the driven table. In one embodiment, the determining module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0105] The deduplication module 630 is used to deduplicate the associated variables in the driving table corresponding to the index variables in response to the association variable containing the index variable and the driving table using the index variable as the distribution key. In one embodiment, the deduplication module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0106] The statistics module 640 is used to count the number of records of the index variable corresponding to the deduplicated associated variable. The statistics module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0107] The first association module 650 is used to associate the deduplicated driving table with the driven table in response to the number of records not exceeding the association threshold, forming a temporary table. The temporary table uses the deduplicated association variable as the distribution key. The first association module 650 can be used to execute the operation S250 described above, which will not be repeated here.

[0108] The second association module 660 is used to associate the driving table with the temporary table using the original association variables in the driving table, to obtain the final join table. The second association module 660 can be used to perform the operation S260 described above, which will not be repeated here.

[0109] According to embodiments of this disclosure, any multiple modules among the setting module 610, determining module 620, deduplication module 630, statistics module 640, first association module 650, and second association module 660 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the setting module 610, determining module 620, deduplication module 630, statistics module 640, first association module 650, and second association module 660 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the setting module 610, determining module 620, deduplication module 630, statistics module 640, first association module 650 and second association module 660 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0110] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a data association method according to an embodiment of the present disclosure.

[0111] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0112] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0113] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0114] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0115] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0116] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the item recommendation method provided in the embodiments of this disclosure.

[0117] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0118] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0119] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0120] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0123] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data association method, comprising: Set index variables and join thresholds for the driven table; Determine whether the associated variable in the driving table contains the index variable and whether the driving table uses the index variable as the distribution key; wherein, the data volume of the driving table is smaller than the data volume of the driven table, and the distribution key is used to distribute the data in the driving table for storage. In response to the fact that the associated variable contains the index variable and the driving table uses the index variable as the distribution key, the associated variables in the driving table corresponding to the index variable are deduplicated. Count the number of records in the index variable corresponding to the deduplicated associated variable; In response to the fact that the number of records is not greater than the association threshold, the deduplicated driving table is associated with the driven table to form a temporary table; wherein, the temporary table uses the deduplicated association variable as the distribution key; and The driving table is associated with the temporary table using the original association variables in the driving table to obtain the final join table; The setting of the association threshold also includes: Obtain the first association time between the driving table and the driven table, wherein the first association time is the time required for the driving table to directly associate with the driven table; Obtain the second association time of the driving table associating with the driven table, wherein the second association time is the time required from setting the index variable and the association threshold for the driven table to associating the driving table with the temporary table using the original association variable in the driving table to obtain the final join table; Obtain the maximum allowed association threshold of the driven table; The current association threshold of the driven table is adjusted based on the first association time, the second association time, and the maximum association threshold.

2. The data association method according to claim 1, wherein, Setting index variables for the driven table includes: Set a B-tree index variable for the driven table based on the distribution key of the database.

3. The data association method according to claim 1, wherein, The setting of the association threshold includes: Set different join thresholds for driven tables with different data volumes and / or different index distributions.

4. The data association method according to claim 1, wherein, The step of adjusting the current association threshold of the driven table based on the first association time, the second association time, and the maximum association threshold includes: Calculate the time optimization rate for associating the driving table with the driven table based on the first association time and the second association time; and In response to the time optimization rate being greater than the preset optimization rate and the current association threshold of the driven table being less than the maximum association threshold, the current association threshold of the driven table is increased.

5. The data association method according to claim 4, wherein, The step of calculating the time optimization rate of the driving table's association with the driven table based on the first association time and the second association time includes: The ratio of the first associated time to the second associated time is calculated as the time optimization rate; Alternatively, the absolute value of the difference between the first associated time and the second associated time can be calculated as the time optimization rate. Alternatively, the ratio of the absolute value of the difference between the first correlation time and the second correlation time to the second correlation time can be calculated as the time optimization rate.

6. The data association method according to claim 1, wherein, The deduplicated driving table is associated with the driven table using a nested loop join method. And / or, using a nested loop join join method, the driving table is associated with the temporary table using the original association variables in the driving table.

7. The data association method according to claim 1, further comprising: In response to the association variable containing the index variable and / or the distribution key of the driving table not being the index variable, the driving table is directly associated with the driven table in a scale manner. Alternatively, in response to the number of records being greater than the association threshold, the driving table is directly associated with the driven table in a scale manner.

8. A data association device, comprising: The configuration module is used to set index variables and join thresholds for the driven table. A determination module is used to determine whether the associated variable in the driving table contains the index variable and whether the driving table uses the index variable as the distribution key; wherein, the data volume of the driving table is smaller than the data volume of the driven table, and the distribution key is used to distribute the data in the driving table for storage. A deduplication module is used to perform deduplication processing on the associated variables in the driving table corresponding to the index variable in response to the fact that the associated variable contains the index variable and the driving table uses the index variable as the distribution key; The statistics module is used to count the number of records of the index variable corresponding to the deduplicated associated variable; The first association module is configured to, in response to the number of records not exceeding the association threshold, associate the deduplicated driving table with the driven table to form a temporary table; wherein the temporary table uses the deduplicated association variable as the distribution key; and The second association module is used to associate the driving table with the temporary table using the original association variables in the driving table to obtain the final join table; The setting of the association threshold also includes: Obtain the first association time between the driving table and the driven table, wherein the first association time is the time required for the driving table to directly associate with the driven table; Obtain the second association time of the driving table associating with the driven table, wherein the second association time is the time required from setting the index variable and the association threshold for the driven table to associating the driving table with the temporary table using the original association variable in the driving table to obtain the final join table; Obtain the maximum allowed association threshold of the driven table; The current association threshold of the driven table is adjusted based on the first association time, the second association time, and the maximum association threshold.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images