Data association method, apparatus, storage medium, and terminal

Abstract

This specification discloses a data association method, apparatus, storage medium, and terminal. The method involves acquiring a main table shard containing incrementally updated data within a target period, determining the target index partition of the main table shard corresponding to the target period in the index table, identifying the association task corresponding to the target index partition, and associating the bypass table partition corresponding to the target index partition with the main table shard based on the association task to obtain a bypass period scheduling table. Based on the bypass period scheduling table, and combining the data in the main table shard and the data in the bypass period scheduling table, a main table wide table is obtained. Since the index table contains index partitions corresponding to each period, during association, the corresponding bypass table can be indexed using the index partition corresponding to that period. Then, the data from the main table and the bypass table are associated to obtain the required main table wide table. This reduces the computational load during data association by using an index table that only records the index paths between multiple tables.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data association method, apparatus, storage medium, and terminal. Background Technology

[0002] Providing services to users often involves generating large amounts of data. As transactions evolve and time progresses, the data volume grows, leading to increased resource requirements and longer processing times for data processing tasks. Storing large volumes of data typically involves using multiple database tables to store different types of data and establishing relationships between these tables. This allows for successful retrieval of target data from each table when calculations require data from multiple tables simultaneously. Therefore, a data association method is needed to improve the efficiency of data association. Summary of the Invention

[0003] This specification provides a data association method, apparatus, storage medium, and terminal, which can solve the technical problems of poor accuracy and high computational resource consumption in related technologies.

[0004] Firstly, embodiments of this specification provide a data association method, which includes:

[0005] Obtain the main table shards that contain incrementally updated data within the target period, and determine the index partitions in the index table corresponding to the above target period as the target index partitions of the above main table shards;

[0006] Determine the associated tasks corresponding to the target index partitions mentioned above, and associate the bypass table partitions corresponding to the target index partitions with the main table shards based on the associated tasks to obtain the bypass periodic scheduling table.

[0007] Based on the above bypass periodic scheduling table, the main table wide table is obtained by combining the data in the above main table shards and the data in the above bypass periodic scheduling table.

[0008] In one possible implementation, the index table is configured to generate an index partition corresponding to the current period every preset period, and the index partition of each period is used to perform indexing within its respective period.

[0009] In one possible implementation, the above-mentioned acquisition of the main table shard containing incrementally updated data within the target period and determination of the index partition in the index table corresponding to the target period as the target index partition of the main table shard includes: dividing the target period into at least two sub-periods; at the end of each sub-period, acquiring the first main table shard containing incrementally updated data within the previous ending target sub-period, and determining the index partition in the index table corresponding to the target period as the first target index partition of the first main table shard; at the end of the target period, acquiring the second main table shard containing incrementally updated data within the last target sub-period of the period, and determining the index partition in the index table corresponding to the target period as the second target index partition of the second main table shard.

[0010] In one possible implementation, the process involves determining the associated task corresponding to the target index partition, and associating the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain a bypass periodic scheduling table. This includes: at the end of each sub-cycle, determining the first associated task corresponding to the first target index partition, and associating the bypass table partition corresponding to the first target index partition with the first main table shard based on the first associated task to obtain a bypass first scheduling table; at the end of the target cycle, determining the second associated task corresponding to the second target index partition, and associating the bypass table partition corresponding to the second target index partition with the second main table shard based on the second associated task to obtain a bypass second scheduling table; and deduplicating the bypass second scheduling table based on the bypass first scheduling table to obtain a bypass periodic scheduling table.

[0011] In one possible implementation, the above method further includes: setting a data volume threshold for the running data of each associated task and / or the number of consecutive periods in which the running data is located based on preset association requirements, so that each index partition corresponds to only one associated task, and each associated task corresponds to at least one index partition.

[0012] In one possible implementation, the above method further includes: calculating the feature value of the main table association key corresponding to each index partition in the above index table, and performing data bucketing and intra-bucket sorting on each index partition based on each feature value.

[0013] Secondly, embodiments of this specification provide a data association device, the device comprising:

[0014] The index partition determination module is used to obtain the main table shards that contain incrementally updated data in the target period, and determine the index partition in the index table corresponding to the above target period as the target index partition of the above main table shards.

[0015] The associated task distribution module is used to determine the associated task corresponding to the target index partition, and associate the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain the bypass periodic scheduling table.

[0016] The data association module is used to obtain the main table wide table based on the above bypass periodic scheduling table, combined with the data in the above main table shards and the data in the above bypass periodic scheduling table.

[0017] In one possible implementation, the index table is configured to generate an index partition corresponding to the current period every preset period, and the index partition of each period is used to perform indexing within its respective period.

[0018] In one possible implementation, the index partition determination module is further configured to divide the target period into at least two sub-periods, and at the end of each sub-period, obtain the first main table shard containing incrementally updated data in the previous target sub-period, and determine the index partition in the index table corresponding to the target period as the first target index partition of the first main table shard; at the end of the target period, obtain the second main table shard containing incrementally updated data in the last target sub-period of the period, and determine the index partition in the index table corresponding to the target period as the second target index partition of the second main table shard.

[0019] In one possible implementation, the aforementioned associated task distribution module is further configured to: at the end of each sub-cycle, determine the first associated task corresponding to the first target index partition; associate the bypass table partition corresponding to the first target index partition with the first main table shard based on the first associated task to obtain a bypass first scheduling table; at the end of the target cycle, determine the second associated task corresponding to the second target index partition; associate the bypass table partition corresponding to the second target index partition with the second main table shard based on the second associated task to obtain a bypass second scheduling table; and deduplicate the bypass second scheduling table based on the bypass first scheduling table to obtain a bypass cycle scheduling table.

[0020] In one possible implementation, the above apparatus further includes: an associated task setting module, used to set the data volume threshold of the running data of each associated task and / or the number of consecutive cycles in which the running data is located based on preset associated requirements, so that each index partition corresponds to only one associated task, and each associated task corresponds to at least one index partition.

[0021] In one possible implementation, the above apparatus further includes: a main table sorting module, used to calculate the feature value of the main table association key corresponding to each index partition in the above index table, and to perform data bucketing and intra-bucket sorting on each index partition based on each feature value.

[0022] Thirdly, embodiments of this specification provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of the method described above.

[0023] Fourthly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method described above.

[0024] Fifthly, embodiments of this specification provide a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being adapted to be loaded by the processor and to execute the steps of the method described above.

[0025] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0026] This specification provides a data association method. It involves obtaining the main table shard containing incrementally updated data within a target period, determining the target index partition of the main table shard corresponding to the target period in the index table, identifying the association task corresponding to the target index partition, and associating the bypass table partition corresponding to the target index partition with the main table shard based on the association task to obtain a bypass period scheduling table. Based on the bypass period scheduling table, and combining the data in the main table shard and the data in the bypass period scheduling table, a main table wide table is obtained. Since the index table contains index partitions corresponding to each period, when associating the main table shard containing incrementally updated data within a period with the bypass table, the corresponding bypass table can be indexed using the index partition corresponding to that period. Then, based on the association task corresponding to the index partition, the data in the main table and the bypass table are associated to obtain the required main table wide table. This reduces the amount of data computation when associating data between different tables by using an index table that only records the index paths between multiple tables, without recording the associated data, thus improving the efficiency of cross-table and cross-period data association. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 An exemplary system architecture diagram of a data association method provided in the embodiments of this specification;

[0029] Figure 2A flowchart illustrating a data association method provided in an embodiment of this specification;

[0030] Figure 3 A logical diagram of an index table for data association provided in an embodiment of this specification;

[0031] Figure 4 A flowchart illustrating a data association method provided in another embodiment of this specification;

[0032] Figure 5 A logical flowchart of a data association method provided in the embodiments of this specification;

[0033] Figure 6 A structural block diagram of a data association device provided in the embodiments of this specification;

[0034] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of this specification. Detailed Implementation

[0035] To make the features and advantages of the embodiments of this specification more apparent and understandable, the technical solutions of the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the embodiments of this specification.

[0036] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those described in this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments described in this specification as detailed in the appended claims.

[0037] With the development of the internet, online activities generate massive amounts of data. As the content of transactions evolves and time progresses, the data volume grows, leading to increased resource requirements and longer processing times for data processing tasks. Storing all data in a single full table would result in excessive data, causing database errors, processing delays, and data loss. These failures could compromise data integrity and accuracy. Therefore, when storing large volumes of data, multiple database tables are often used to store different types of data, and relationships are established between these tables. This allows for the retrieval of target data from each table when calculations require data from multiple tables simultaneously. For example, in a product sales system, the main table stores order information and transaction user information, while a side table stores product information and price information. When calculating the transaction history for a product, the main table is linked with the side table to retrieve the necessary fields, resulting in the order information, transaction user information, product information, and price information for the same product. The transaction history is then calculated based on this complete product information.

[0038] Currently, to better manage data, data is recorded according to fixed periods, such as a week, a day, or 12 hours. Data association operations are also performed periodically. When performing data association for a period, if both the main table and the side table have incrementally updated data, then this updated data will be directly associated. However, if the side table does not have incrementally updated data in that period, the main table cannot associate with the side table data of the corresponding fields that have not been updated, resulting in inaccurate data after association.

[0039] Therefore, this specification provides a data association method that obtains the main table shard containing incrementally updated data within a target period, determines the target index partition of the main table shard corresponding to the target period in the index table, determines the association task corresponding to the target index partition, and associates the bypass table partition corresponding to the target index partition with the main table shard based on the association task to obtain a bypass period scheduling table. Based on the bypass period scheduling table, and combining the data in the main table shard and the data in the bypass period scheduling table, a main table wide table is obtained to solve the technical problems of poor accuracy and high computational resource consumption in the above-mentioned data association.

[0040] Please see Figure 1 , Figure 1 This is an exemplary system architecture diagram of a data association method provided in the embodiments of this specification.

[0041] like Figure 1As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0042] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartwatches, smartphones, tablets, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0043] In the embodiments of this specification, terminal 101 first obtains the main table shard containing incremental update data within the target period, and determines the target index partition of the main table shard corresponding to the target period in the index table; further, terminal 101 determines the associated task corresponding to the target index partition, and associates the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain the bypass period scheduling table; then, terminal 101 obtains the main table wide table based on the bypass period scheduling table, combined with the data in the main table shard and the data in the bypass period scheduling table.

[0044] Server 103 can be a business server providing various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0045] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification can be applied to a system structure that only includes terminal 101. The embodiments of this specification do not limit this.

[0046] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.

[0047] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data association method provided in an embodiment of this specification. The execution entity in this embodiment can be a terminal executing the data association, a processor within the terminal executing the data association method, or a data association service within the terminal executing the data association method. For ease of description, the following example uses a processor within a terminal as the execution entity to illustrate the specific execution process of the data association method.

[0048] like Figure 2 As shown, data association methods can include at least:

[0049] S202. Obtain the main table shards that contain incrementally updated data within the target period, and determine the index partition in the index table corresponding to the target period as the target index partition of the main table shards.

[0050] Optionally, when performing offline computing tasks, it is usually necessary to join the main table with the bypass tables of each domain to obtain the fields and data required by the offline computing task, ultimately obtaining the main table width table required by the offline computing task. However, the main table width table obtained after data joining is updated by the data acquisition tool after the corresponding online tables are updated. If the main table and the joined bypass tables do not have incremental data added on the same day, the incremental data in the main table cannot be joined with the corresponding data in the bypass tables. This inability to join data will lead to accuracy issues in the main table width table. For example, the main table width table needs product information, product price, and product transaction records. The online main table stores product transaction records, while the bypass tables store product prices and product information. If the product transaction record data of product A in the main table changes during the current period, then the main table data and bypass table data of product A need to be joined in this period. However, if the product information data and product price data of product A in the bypass tables have not changed during this period, then the product transaction record data of product A in this period cannot be joined with the product information data and product price data.

[0051] Optionally, when joining data between a main table and a bypass table that do not both have incremental changes within the same period, cross-period data joining is required. This means the main table needs to join with bypass table data from previous periods. In multi-table data management scenarios, cross-period data joining helps integrate data from different points in time, ensuring data integrity and facilitating subsequent comparison, analysis, and reporting.

[0052] Alternatively, in most cases, if it's necessary to join data from different periods, the conventional approach is to read the full offline tables of the relevant tables and use the primary key as the join key. However, with data volumes in the tens or hundreds of billions, reading and joining the full tables daily becomes extremely time-consuming and resource-intensive. Another approach is to maintain an incomplete full table. While there is still updatable content, the relevant tables are joined when detailed data is first recorded, and the incomplete data details are stored in this incomplete full table. If the main table undergoes an incremental update while the bypass tables remain unchanged, this incomplete full table can be joined to obtain the necessary information. If a data item's status is still incomplete, it can be overwritten and updated back to a complete full table. However, over time, the amount of data in the incomplete full table will gradually increase, leading to a gradual increase in the resources required for the task and a longer processing time.

[0053] Furthermore, when performing cross-period data association, in order to quickly find the cross-period bypass table data items corresponding to the main table data, an index table can be used between the main table and the bypass table. The index table records the index path of the association relationship between the main table and the bypass table. By using the index table to find the main table and the corresponding bypass table, the data of the main table and the bypass table can be quickly associated, reducing the computing resources consumed in the data association process and improving the efficiency and accuracy of data association.

[0054] Specifically, the index table does not directly store the original data, but rather stores index keys and pointers or references to the actual data. This structure allows the index table to quickly locate data in the actual data table, improving query performance. The data in the index table is generally sorted and organized according to specific index fields (or keys), typically including: the index key, which is the core of the index table and identifies each row of data in the table; and pointers or references, which usually contain pointers or references to the corresponding rows in the actual data table, used for quick access to the actual data when needed. These index fields are selected based on requirements and database query performance considerations, and are usually certain columns in the table. In the embodiments described in this specification, the index field is the unique identifier key ID of the data item in the main table.

[0055] In the embodiments of this specification, data association tasks are performed periodically. Therefore, the unique identifier key ID of each data item in the corresponding main table should also be generated based on the corresponding period of the data item, so that the index field is the main table association key ID related to the period. This allows the data association task to be completed through the index table according to the period information in the main table association key ID. When indexing, the index table uses the index partition corresponding to each period to index the main table and bypass tables for that period. If there are no incremental updates to the online main table data within the target period, no repeated association is needed. If there are incremental updates to the online main table data, it needs to be associated with the relevant data in the bypass tables to obtain the latest online data in a timely manner. Therefore, within each period, the main table shard with incrementally updated data within the target period is first obtained, and then the index partition in the index table corresponding to the target period is determined as the target index partition of the main table shard.

[0056] It should be noted that the period can be set according to the needs of the actual scenario. If the data volume is large or the data changes frequently in a short period of time, a shorter data association period can be set, such as half a day or one day, to alleviate the problem of strained computing resources when a large amount of data is associated simultaneously. If the data volume is small or the data changes infrequently in a short period of time, a shorter data association period can be set, such as one week or one month. The embodiments in this specification do not impose specific limitations on the setting of the period.

[0057] S204. Determine the associated task corresponding to the target index partition, and associate the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain the bypass periodic scheduling table.

[0058] Optionally, when performing data association, the index partition has a pre-generated association task. The association task is used to complete the association between the main table and the bypass table corresponding to the index partition. That is, after determining the target index partition corresponding to the target period, the association task corresponding to the target index partition is determined. Based on the association task, the bypass table partition corresponding to the target index partition is associated with the main table shard. The bypass period scheduling table composed of the data items that need to be associated by the main table shard is obtained from the bypass table partition.

[0059] For related tasks, thresholds for the amount of data used in each task and / or the number of consecutive periods the data is in can be set based on preset related requirements. Limiting the amount of data in related tasks prevents long-tail tasks, ensuring that each index partition corresponds to only one related task, and each related task corresponds to at least one index partition. Furthermore, SQL tuning interfaces can be provided for all related tasks to fine-tune parameters based on their execution. When using multiple parallel related tasks for data association, if a task fails due to resource issues or other unexpected problems, only the failed task needs to be rerun, avoiding rerunning all tasks requiring data association and improving the timeliness and fault tolerance of related task execution.

[0060] S206. Based on the bypass periodic scheduling table, the main table wide table is obtained by combining the data in the main table shards and the data in the bypass periodic scheduling table.

[0061] Optionally, after obtaining the bypass periodic scheduling table, the bypass periodic scheduling table contains the data items that are associated with the main table shards and retrieved from the bypass table. Based on this, the main table wide table is obtained by combining the data in the main table shards and the data in the bypass periodic scheduling table. Each data item in the main table wide table contains all the data corresponding to the primary key in the main table shards and the bypass table. This completes the data association between the main table and the bypass table. By using an index table that only records the index paths between multiple tables, the amount of data calculation when associating data between different data tables can be reduced, and the efficiency of cross-table and cross-period data association can be improved.

[0062] This specification provides a data association method. The method involves obtaining the main table shard containing incrementally updated data within a target period, determining the target index partition of the main table shard corresponding to the target period in the index table, determining the association task corresponding to the target index partition, and associating the bypass table partition corresponding to the target index partition with the main table shard based on the association task to obtain a bypass period scheduling table. Based on the bypass period scheduling table, and combining the data in the main table shard and the data in the bypass period scheduling table, a main table wide table is obtained. Since the index table contains index partitions corresponding to each period, when associating the main table shard containing incrementally updated data within a period with the bypass table, the corresponding bypass table can be indexed using the index partition corresponding to that period. Then, based on the association task corresponding to the index partition, the data in the main table and the bypass table are associated to obtain the required main table wide table. This reduces the amount of data computation when associating data between different data tables by using an index table that only records the index paths between multiple tables, without recording the associated data, thus improving the efficiency of cross-table and cross-period data association.

[0063] As described in the above embodiments, when performing cross-period data association between different tables, using an index table that records the association relationship between the main table and the bypass tables to partition and associate the bypass tables corresponding to the main table's sharded index each time can save computational resources consumed during the data association process. As time progresses and new periods emerge, the index table also needs to be updated accordingly to ensure that data generated in each period can be associated.

[0064] Please see Figure 3 , Figure 3 This is a logical diagram of an index table for data association provided in an embodiment of this specification.

[0065] Optionally, starting from its creation time, the index table generates corresponding index partitions for each period to complete data association within that period. That is, the index table is set to generate an index partition for the current period every preset period, with each period's index partition used for indexing within its respective period. When generating index partitions, data is cleaned from the bypass table to obtain the primary table's join key ID. This primary table join key ID and the latest index partition are then written to the index table, ensuring that the index table contains the latest period's index partition. When multiple bypass tables exist, their indexes can be placed in the same index table, avoiding multiple reads of different index tables during data association and reducing data read pressure.

[0066] For example, such as Figure 3 As shown, taking a period of one day as an example, in the main table, the main table join key ID of the main table shard created on May 1st is 0501a, and the main table join key ID of the main table shard created on May 3rd is 0503b. In the index table, the main table join key ID corresponds to the index partition. After the data item "0503b..." is created on May 3rd and undergoes an incremental change, it should correspond to the 0503 partition corresponding to the period on May 3rd, and the data join should be performed at the end of May 3rd. The index partition corresponding to the data item "0501a..." in the creation period is the 0501 partition. However, if the data item "0501a..." undergoes another incremental change on May 5th, then the index partition corresponding to the data item "0501a..." should be updated to the 0505 partition corresponding to the period on May 5th. Different index partitions can use the same or different join tasks, but the same index partition can only use the same join task. The join task can join the main table data to the bypass table data item with the same main table join key ID, thereby completing the data join between the main table and the bypass table.

[0067] Optionally, when dealing with large amounts of data, the characteristic values ​​of the main table join keys corresponding to each index partition in the index table are calculated. Based on these characteristic values, data is bucketed and sorted within each index partition to improve the join efficiency of the index table. It should be noted that the characteristic values ​​can be calculated using a hash algorithm or other characteristic value algorithms; this specification does not specifically limit the methods used in the embodiments.

[0068] In the embodiments of this specification, a method for setting up an index table for data association is provided. The index table is set to generate an index partition corresponding to the current period every preset period. Each period's index partition is used for indexing within its own period. The index table records the latest partition of the bypass table but does not record the specific values. By indexing the bypass table partition through the index table, it is not necessary to put all the data in the same full table for data association, thus reducing the computational load of reading the full table. When the data volume is large, the feature value of the main table association key corresponding to each index partition in the index table is calculated. Based on each feature value, the data of each index partition is bucketed and sorted within the bucket, thereby improving the association efficiency of the index table.

[0069] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data association method provided in another embodiment of this specification.

[0070] like Figure 4 As shown, data association methods can include at least:

[0071] S402. Divide the target period into at least two sub-periods. At the end of each sub-period, obtain the first main table shard that contains incrementally updated data in the previous target sub-period. Determine the index partition in the index table corresponding to the target period as the first target index partition of the first main table shard.

[0072] Optionally, when there is a large amount of incremental data within a single period, performing data association all at the end of the period may lead to excessive resource consumption and affect data timeliness. Therefore, to address the issues of long processing time and high resource consumption in sharded parallel association tasks, the period can be further subdivided into multiple sub-periods. Data association can be performed in advance within each sub-period, and deduplication can be performed after the complete period ends, using the latest incremental data to retain the most accurate data association result. The sub-periods can be divided according to the needs and data volume in the actual scenario. For example, if the period length is 24 hours, the complete period can be divided into 3 equal-length sub-periods, that is, each sub-period is 8 hours long, and the incremental data generated in the previous 8 hours can be associated every 8 hours.

[0073] Specifically, please refer to Figure 5 , Figure 5This is a logical flowchart of a data association method provided in an embodiment of this specification. Figure 5 As shown, the target period is divided into at least two sub-periods. The main table, which is based on a complete period, is used as the main table for the period, and the partial main table, which is based on a sub-period, is used as the main table for the sub-period. At the end of each sub-period, the first main table shard in the sub-period main table that contains incrementally updated data in the previous target sub-period is obtained. At this time, the index partition in the index table corresponding to the target period is determined as the first target index partition of the first main table shard. Based on the first target index partition, the data association of the first main table shard is performed in advance. In other words, the index partition of this period directly performs data association for the incremental data of the main table in the sub-period in advance at the end of the sub-period, reducing the computational resource consumption at the end of the complete period.

[0074] S404. At the end of the target period, obtain the second main table shard containing incrementally updated data in the last target sub-period of the period, and determine the index partition in the index table corresponding to the target period as the second target index partition of the second main table shard.

[0075] Alternatively, please continue reading Figure 5 At the end of the complete cycle, the second main table shard containing incrementally updated data in the last target sub-cycle of the cycle is obtained from the main table of the cycle. The index partition of the target cycle in the index table is determined as the second target index partition of the second main table shard. Then, the index partition of the cycle can be used to perform data association with the main table shard data of the last sub-cycle.

[0076] S406. At the end of each sub-cycle, determine the first associated task corresponding to the first target index partition, and associate the bypass table partition corresponding to the first target index partition with the first main table shard based on the first associated task to obtain the bypass first scheduling table.

[0077] Alternatively, please continue reading Figure 5 At the end of each sub-cycle, the index partition creates the parallel association task corresponding to the sub-cycle in advance, that is, determines the first association task corresponding to the first target index partition. Then, based on the first association task, the bypass table partition corresponding to the first target index partition is associated with the first main table shard. The main table shard data and bypass table partition data in the sub-cycle are associated in advance to obtain the bypass first scheduling table.

[0078] S408. At the end of the target period, determine the second associated task corresponding to the second target index partition, and associate the bypass table partition corresponding to the second target index partition with the second main table shard based on the second associated task to obtain the bypass second scheduling table.

[0079] Alternatively, please continue reading Figure 5At the end of the target period, the second associated task corresponding to the second target index partition is determined. Based on the second associated task, the bypass table partition corresponding to the second target index partition is associated with the second main table shard to obtain the bypass second scheduling table. At this time, the bypass second scheduling table is the bypass scheduling table of the last sub-cycle in the target period.

[0080] S410. Based on the first bypass scheduling table, the second bypass scheduling table is deduplicated to obtain the bypass periodic scheduling table.

[0081] Alternatively, please continue reading Figure 5 There may be some data items that have undergone incremental changes in multiple sub-cycles within a single cycle. In such cases, the final data association result should be based on the last change. The duplicate data that has been repeatedly associated should be deduplicated. This means deduplicating the second bypass scheduling table based on the first bypass scheduling table to obtain the final bypass cycle scheduling table.

[0082] S412. Based on the bypass periodic scheduling table, the main table wide table is obtained by combining the data in the main table shards and the data in the bypass periodic scheduling table.

[0083] For details regarding step S412, please refer to the description in step S206; it will not be repeated here.

[0084] In the embodiments of this specification, a data association method is provided, which further subdivides the period into multiple sub-periods, performs data association in advance in each sub-period, and performs deduplication after the complete period ends, using the latest incremental data as the standard to retain the most accurate data association result. This solves the problems of long time consumption and high resource consumption of the piecewise parallel association task, and reduces the computational resource consumption at the end of the complete period.

[0085] Please see Figure 6 , Figure 6 This is a structural block diagram of a data association device provided in an embodiment of this specification. Figure 6 As shown, the data association device 600 includes:

[0086] The index partition determination module 610 is used to obtain the main table shards that contain incrementally updated data in the target period, and determine the index partition in the index table corresponding to the target period as the target index partition of the main table shard.

[0087] The associated task distribution module 620 is used to determine the associated task corresponding to the target index partition, and associate the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain the bypass periodic scheduling table.

[0088] The data association module 630 is used to obtain the main table wide table based on the bypass periodic scheduling table, combined with the data in the main table shards and the data in the bypass periodic scheduling table.

[0089] Optionally, the index table is configured to generate an index partition corresponding to the current period every preset period, and the index partition of each period is used for indexing within its respective period.

[0090] Optionally, the index partition determination module 610 is further configured to divide the target period into at least two sub-periods, and at the end of each sub-period, obtain the first main table shard containing incrementally updated data in the previous target sub-period, and determine the index partition in the index table corresponding to the target period as the first target index partition of the first main table shard; at the end of the target period, obtain the second main table shard containing incrementally updated data in the last target sub-period of the period, and determine the index partition in the index table corresponding to the target period as the second target index partition of the second main table shard.

[0091] Optionally, the associated task distribution module 620 is further configured to: at the end of each sub-cycle, determine the first associated task corresponding to the first target index partition; associate the bypass table partition corresponding to the first target index partition with the first main table shard based on the first associated task to obtain a bypass first scheduling table; at the end of the target cycle, determine the second associated task corresponding to the second target index partition; associate the bypass table partition corresponding to the second target index partition with the second main table shard based on the second associated task to obtain a bypass second scheduling table; and deduplicate the bypass second scheduling table based on the bypass first scheduling table to obtain a bypass cycle scheduling table.

[0092] Optionally, the data association device 600 further includes: an association task setting module, used to set the data volume threshold of the running data of each association task and / or the number of consecutive periods in which the running data is located based on preset association requirements, so that each index partition corresponds to only one association task, and each association task corresponds to at least one index partition.

[0093] Optionally, the data association device 600 further includes: a master table sorting module, used to calculate the feature value of the master table association key corresponding to each index partition in the index table, and to perform data bucketing and intra-bucket sorting on each index partition based on each feature value.

[0094] In this embodiment, a data association apparatus is provided, comprising: an index partition determination module for obtaining the main table shard containing incrementally updated data within a target period, and determining the target index partition of the main table shard corresponding to the target period in the index table; an association task distribution module for determining the association task corresponding to the target index partition, and associating the bypass table partition corresponding to the target index partition with the main table shard based on the association task to obtain a bypass period scheduling table; and a data association module for obtaining a main table wide table based on the bypass period scheduling table, combined with the data in the main table shard and the data in the bypass period scheduling table. Since the index table contains index partitions corresponding to each period, when associating the main table shard containing incrementally updated data within a period with the bypass table, the corresponding bypass table can be indexed using the index partition corresponding to that period. Then, based on the association task corresponding to the index partition, the data of the main table and the bypass table are associated to obtain the required main table wide table. This reduces the amount of data computation when associating data between different data tables by using an index table that only records the index paths between multiple tables, without recording the associated data, thus improving the efficiency of cross-table and cross-period data association.

[0095] This specification provides a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of any of the methods described above.

[0096] This specification also provides a computer storage medium that can store multiple instructions adapted for loading by a processor and executing the steps of any of the methods described in the above embodiments.

[0097] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a terminal provided as an embodiment of this specification. For example... Figure 7 As shown, terminal 700 may include: at least one terminal processor 701, at least one network interface 704, user interface 703, memory 705, and at least one communication bus 702.

[0098] The communication bus 702 is used to enable communication between these components.

[0099] The user interface 703 may include a display screen and a camera. Optionally, the user interface 703 may also include a standard wired interface and a wireless interface.

[0100] The network interface 704 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0101] The terminal processor 701 may include one or more processing cores. The terminal processor 701 connects to various parts within the terminal 700 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 705, and by calling data stored in the memory 705. Optionally, the terminal processor 701 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The terminal processor 701 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the terminal processor 701.

[0102] The memory 705 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 705 may include a non-transitory computer-readable storage medium. The memory 705 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 705 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 705 may also be at least one storage device located remotely from the aforementioned terminal processor 701. Figure 7 As shown, the memory 705, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data association program.

[0103] exist Figure 7In the terminal 700 shown, the user interface 703 is mainly used to provide an input interface for the user and to obtain the user's input data; while the terminal processor 701 can be used to call the data association program stored in the memory 705 and specifically perform the following operations:

[0104] Obtain the main table shards containing incrementally updated data within the target period, and determine the target index partition of the main table shard corresponding to the target period in the index table;

[0105] Determine the associated tasks corresponding to the target index partition, and associate the bypass table partition corresponding to the target index partition with the main table shard based on the associated tasks to obtain the bypass periodic scheduling table;

[0106] Based on the bypass periodic scheduling table, the main table wide table is obtained by combining the data in the main table shards and the data in the bypass periodic scheduling table.

[0107] In some embodiments, the index table is configured to generate an index partition corresponding to the current period every preset period, and the index partition of each period is used to perform indexing within its respective period.

[0108] In some embodiments, when the terminal processor 701 performs the following steps when acquiring the main table shard containing incrementally updated data in the target period and determining the index partition in the index table corresponding to the target period as the target index partition of the main table shard: dividing the target period into at least two sub-periods; at the end of each sub-period, acquiring the first main table shard containing incrementally updated data in the previous ending target sub-period and determining the index partition in the index table corresponding to the target period as the first target index partition of the first main table shard; at the end of the target period, acquiring the second main table shard containing incrementally updated data in the last target sub-period of the period and determining the index partition in the index table corresponding to the target period as the second target index partition of the second main table shard.

[0109] In some embodiments, when the terminal processor 701 executes the following steps when determining the association task corresponding to the target index partition and associating the bypass table partition corresponding to the target index partition with the main table shard based on the association task to obtain a bypass periodic scheduling table: at the end of each sub-cycle, a first association task corresponding to the first target index partition is determined, and the bypass table partition corresponding to the first target index partition is associating with the first main table shard based on the first association task to obtain a bypass first scheduling table; at the end of the target cycle, a second association task corresponding to the second target index partition is determined, and the bypass table partition corresponding to the second target index partition is associating with the second main table shard based on the second association task to obtain a bypass second scheduling table; and the bypass second scheduling table is deduplicated based on the bypass first scheduling table to obtain a bypass periodic scheduling table.

[0110] In some embodiments, the terminal processor 701 further performs the following steps: setting the data volume threshold of the running data of each associated task and / or the number of consecutive cycles in which the running data is located based on preset association requirements, so that each index partition corresponds to only one associated task, and each associated task corresponds to at least one index partition.

[0111] In some embodiments, the terminal processor 701 further performs the following steps: calculating the feature value of the main table association key corresponding to each index partition in the index table, and performing data bucketing and sorting within each index partition based on each feature value.

[0112] In the several embodiments provided in this specification, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0113] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0114] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0115] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0116] In addition, it should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0117] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0118] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0119] The above is a description of a data association method, apparatus, storage medium, and terminal provided in the embodiments of this specification. For those skilled in the art, based on the ideas of the embodiments of this specification, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this specification.

Claims

1. A data association method, the method comprising: Obtain the main table shard that contains incrementally updated data within the target period, and determine the index partition in the index table corresponding to the target period as the target index partition of the main table shard; Determine the associated task corresponding to the target index partition, and associate the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain the bypass periodic scheduling table; Based on the bypass periodic scheduling table, the main table wide table is obtained by combining the data in the main table shards and the data in the bypass periodic scheduling table.

2. The method according to claim 1, wherein the index table is configured to generate an index partition corresponding to the current period every preset period, and the index partition of each period is used for indexing within its respective period.

3. The method according to claim 2, wherein obtaining the main table shard containing incrementally updated data within the target period and determining the index partition in the index table corresponding to the target period as the target index partition of the main table shard includes: The target period is divided into at least two sub-periods. At the end of each sub-period, the first main table shard containing incrementally updated data in the previous target sub-period is obtained, and the index partition in the index table corresponding to the target period is determined as the first target index partition of the first main table shard. At the end of the target period, the second main table shard containing incrementally updated data in the last target sub-period of the period is obtained, and the index partition in the index table corresponding to the target period is determined as the second target index partition of the second main table shard.

4. The method according to claim 3, wherein determining the associated task corresponding to the target index partition, and associating the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain a bypass periodic scheduling table, includes: At the end of each sub-cycle, the first associated task corresponding to the first target index partition is determined, and the bypass table partition corresponding to the first target index partition is associated with the first main table shard based on the first associated task to obtain the bypass first scheduling table; At the end of the target period, the second associated task corresponding to the second target index partition is determined, and the bypass table partition corresponding to the second target index partition is associated with the second main table shard based on the second associated task to obtain the bypass second scheduling table; Based on the first bypass scheduling table, the second bypass scheduling table is deduplicated to obtain the bypass periodic scheduling table.

5. The method according to claim 1, further comprising: Based on preset association requirements, set the data volume threshold of each associated task's running data and / or the number of consecutive periods in which the running data is located, so that each index partition corresponds to only one associated task, and each associated task corresponds to at least one index partition.

6. The method according to claim 1, further comprising: Calculate the feature value of the main table association key corresponding to each index partition in the index table, and perform data bucketing and sorting within each index partition based on each feature value.

7. A data association apparatus, the apparatus comprising: The index partition determination module is used to obtain the main table shards that contain incrementally updated data in the target period, and determine the index partition in the index table corresponding to the target period as the target index partition of the main table shard. The associated task distribution module is used to determine the associated task corresponding to the target index partition, and associate the bypass table partition corresponding to the target index partition with the main table shard based on the associated task to obtain the bypass periodic scheduling table. The data association module is used to obtain the main table wide table based on the bypass periodic scheduling table, combined with the data in the main table shards and the data in the bypass periodic scheduling table.

8. A computer program product comprising instructions that, when run on a computer or processor, causes the computer or processor to perform the steps of the method as claimed in any one of claims 1 to 6.

9. A computer storage medium storing a plurality of instructions adapted for loading by a processor and performing the steps of the method as claimed in any one of claims 1 to 6.

10. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as claimed in any one of claims 1 to 6.

Images