Incremental label data synchronization processing method and device, equipment and storage medium

Abstract

The invention relates to an incremental label data synchronization processing method and device, computer equipment, a storage medium and a computer program product, and relates to the technical field of big data. The resource cost can be reduced, and the synchronization timeliness of the incremental label data can be improved. The method comprises the steps of obtaining respective data distribution characteristics of each sub-domain increment label data table, determining data distribution similarity among the sub-domain increment label data tables according to the respective data distribution characteristics of the sub-domain increment label data tables, and generating a distribution similarity index tree of each sub-domain increment label data table according to the data distribution similarity, and according to the distribution approximation index tree, carrying out step-by-step fusion on each sub-domain increment label data table to obtain a global increment label data table, and writing the global increment label data table into a data warehouse.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and particularly to an incremental tag data synchronization processing method, device, computer device, storage medium, and computer program product. Background Art

[0002] With the development and application of big data technology, related technologies for incremental tag data synchronization have emerged. Taking user portrait data as an example, user portrait data can be split into corresponding business domains and data domains. Incremental calculation is performed on each data domain to generate user tag data, and then it can be synchronized to the user tag retrieval system through a data synchronization tool such as DataX. As the user scale and business domains grow, the IO (Input / Output) capacity of the ES (ElasticSearch, a real-time distributed search and analysis engine) cluster in the user tag retrieval system approaches its limit, and the data synchronization performance becomes lower and lower.

[0003] To improve the incremental tag data synchronization speed of the ES cluster, some technologies fuse the incremental tag data in the Hive data warehouses of multiple sub-domains into a global incremental tag data table by introducing the HBase columnar storage snapshot, optimizing the decentralized synchronization of each sub-domain into centralized synchronization. However, this technology relies on the HBase (a distributed, column-oriented open-source database) cluster, and there are large-scale centralized IO queries, consuming the resources of the HBase cluster and affecting the ability of the HBase cluster to provide tag data query services externally. Summary of the Invention

[0004] Based on this, it is necessary to provide an incremental tag data synchronization processing method, device, computer device, storage medium, and computer program product for the above technical problems.

[0005] In a first aspect, the present application provides an incremental tag data synchronization processing method. The method includes:

[0006] Obtain the data distribution characteristics of each sub-domain incremental tag data table;

[0007] Determine the data distribution similarity between the sub-domain incremental tag data tables according to the data distribution characteristics of each sub-domain incremental tag data table;

[0008] Generate a distribution approximation index tree for each sub-domain incremental tag data table according to the data distribution similarity;

[0009] Gradually fuse each sub-domain incremental tag data table according to the distribution approximation index tree to obtain a global incremental tag data table;

[0010] Write the global incremental tag data table into the data warehouse.

[0011] In one embodiment, obtaining the data distribution characteristics of each sub-domain incremental label data table includes: performing re-partitioning processing on each sub-domain incremental label data table according to the same partitioning algorithm; for the sub-domain incremental label data tables in each sub-domain incremental label data table, determining the respective representative values of each partition according to the data in each partition corresponding to the sub-domain incremental label data table; and obtaining the data distribution characteristics of each sub-domain incremental label data table according to the respective representative values of each partition of each sub-domain incremental label data table.

[0012] In one embodiment, performing re-partitioning processing on each sub-domain incremental label data table according to the same partitioning algorithm includes: performing data sampling processing on each sub-domain incremental label data table according to a certain proportion to obtain the respective sampling data tables of each sub-domain incremental label data table; and performing re-partitioning processing on the respective sampling data tables of each sub-domain incremental label data table according to a preset partitioning algorithm. Determining the respective representative values of each partition according to the data in each partition corresponding to the sub-domain incremental label data table includes: determining the respective representative values of each partition according to the sampling data in each partition corresponding to the sub-domain incremental label data table.

[0013] In one embodiment, determining the respective representative values of each partition according to the sampling data in each partition corresponding to the sub-domain incremental label data table includes: obtaining the respective representative values of each partition according to the average value of the sampling data in each partition corresponding to the sub-domain incremental label data table.

[0014] In one embodiment, obtaining the data distribution characteristics of each sub-domain incremental label data table according to the respective representative values of each partition of each sub-domain incremental label data table includes: for each sub-domain incremental label data table, performing binarization processing on the respective representative values of each partition to obtain a bitmap vector; and compressing the bitmap vector into a numerical vector according to a certain number of bits to obtain the data distribution characteristics of each sub-domain incremental label data table.

[0015] In one embodiment, determining the data distribution similarity between each sub-domain incremental label data table according to the data distribution characteristics of each sub-domain incremental label data table includes: obtaining the Euclidean distance between the corresponding sub-domain incremental label data tables according to the data distribution characteristics of each sub-domain incremental label data table; and determining the data distribution similarity between each sub-domain incremental label data table according to the Euclidean distance.

[0016] In one embodiment, fusing the sub-domain incremental label data tables step by step according to the distribution approximation index tree to obtain a global incremental label data table, including: obtaining a fusion DAG task chain corresponding to the distribution approximation index tree based on a depth-first mechanism, a lazy merge operator, and a lazy query operator of each sub-domain incremental label data table; executing the fusion DAG task chain to fuse each sub-domain incremental label data table step by step to obtain a global incremental label data table.

[0017] In a second aspect, the present application also provides an incremental label data synchronization processing device. The device includes:

[0018] A feature acquisition module, configured to acquire the data distribution features of each sub-domain incremental label data table;

[0019] A similarity determination module, configured to determine the data distribution similarity between the sub-domain incremental label data tables according to the data distribution features of each sub-domain incremental label data table;

[0020] An index tree generation module, configured to generate a distribution approximation index tree of each sub-domain incremental label data table according to the data distribution similarity;

[0021] A global table obtaining module, configured to fuse each sub-domain incremental label data table step by step according to the distribution approximation index tree to obtain a global incremental label data table;

[0022] A data table writing module, configured to write the global incremental label data table into a data warehouse.

[0023] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0024] Acquire the data distribution features of each sub-domain incremental label data table; determine the data distribution similarity between the sub-domain incremental label data tables according to the data distribution features of each sub-domain incremental label data table; generate a distribution approximation index tree of each sub-domain incremental label data table according to the data distribution similarity; fuse each sub-domain incremental label data table step by step according to the distribution approximation index tree to obtain a global incremental label data table; write the global incremental label data table into a data warehouse.

[0025] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Obtain the data distribution characteristics of each sub - domain incremental label data table respectively; determine the data distribution similarity between the sub - domain incremental label data tables according to the data distribution characteristics of each sub - domain incremental label data table respectively; generate a distribution approximation index tree for each sub - domain incremental label data table according to the data distribution similarity; perform hierarchical fusion on each sub - domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table; write the global incremental label data table into the data warehouse.

[0027] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program which, when executed by a processor, implements the following steps:

[0028] Obtain the data distribution characteristics of each sub - domain incremental label data table respectively; determine the data distribution similarity between the sub - domain incremental label data tables according to the data distribution characteristics of each sub - domain incremental label data table respectively; generate a distribution approximation index tree for each sub - domain incremental label data table according to the data distribution similarity; perform hierarchical fusion on each sub - domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table; write the global incremental label data table into the data warehouse.

[0029] The above - mentioned incremental label data synchronization processing method, device, computer device, storage medium and computer program product obtain the data distribution characteristics of each sub - domain incremental label data table respectively, determine the data distribution similarity between the sub - domain incremental label data tables according to the data distribution characteristics of each sub - domain incremental label data table respectively, generate a distribution approximation index tree for each sub - domain incremental label data table according to the data distribution similarity, perform hierarchical fusion on each sub - domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table, and write the global incremental label data table into the data warehouse. This solution can determine the data distribution similarity between the sub - domain incremental label data tables according to the data distribution characteristics of each sub - domain incremental label data table respectively, and accordingly establish a distribution approximation index tree for each sub - domain incremental label data table. According to this index tree, hierarchical fusion is performed on each sub - domain incremental label data table to quickly obtain a global incremental label data table and write it into the data warehouse. Thus, the index tree constructed based on data distribution similarity can achieve the rapid / efficient synthesis of each sub - domain incremental label data table into a global incremental label data table and write it into the data warehouse on the spark on hive computing engine, so that the fusion of the global incremental label data table does not depend on the HBase cluster, reduces the resource consumption of the HBase cluster, avoids the impact on the external query service performance of the HBase cluster, and achieves the effect of reducing resource costs while improving the timeliness of incremental label data synchronization. Description of the Drawings

[0030] Figure 1 It is an application environment diagram of the incremental label data synchronization processing method in an embodiment;

[0031] Figure 2 It is a schematic flowchart of the incremental label data synchronization processing method in an embodiment;

[0032] Figure 3 It is a schematic flowchart of the steps for obtaining data distribution characteristics in an embodiment;

[0033] Figure 4 It is a schematic flowchart of the incremental label data synchronization processing method in another embodiment;

[0034] Figure 5 It is a schematic diagram of the sampling partition compression related processing in an embodiment;

[0035] Figure 6 It is a schematic diagram of the fused DAG task chain in an embodiment;

[0036] Figure 7 It is a structural block diagram of the incremental label data synchronization processing device in an embodiment;

[0037] Figure 8 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The incremental label data synchronization processing method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 This application environment may include a terminal and a server. The terminal can communicate with the server through a network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server, or can be placed in the cloud or other network servers. Among them, the terminal can be, but is not limited to, various personal computers, laptop computers, smart phones, and tablet computers. The server can be implemented by a server cluster composed of multiple servers. Among them, the incremental label data synchronization processing method of the present application can be executed by the server, and the terminal can query the label data synchronized by the server from the ES cluster.

[0040] In one embodiment, as shown in Figure 2 A method for incremental label data synchronization processing is provided, and the method may include the following steps:

[0041] Step S201: Obtain the data distribution characteristics of each sub - domain incremental label data table respectively.

[0042] In this step, the incremental label data tables of each sub - domain stored in the Hive data warehouse (denoted as sub - domain incremental label data tables) can be obtained. Here, the sub - domain can be a business domain. The incremental label data tables of each business domain can be obtained to get the incremental label data tables of each sub - domain, and then the data distribution characteristics of each sub - domain incremental label data table are obtained. Among them, the data distribution characteristic refers to the distribution characteristic of the data in the sub - domain incremental label data table. For example, the data distribution characteristic can be determined by analyzing the arrangement of data values and the mean value, etc. in the sub - domain incremental label data table. Analyzing the data distribution characteristic can be used to evaluate the similarity between each sub - domain incremental label data table, which is convenient for efficiently performing data table fusion processing in subsequent steps and overcoming the impact brought by the huge difference in the data volume of each sub - domain incremental label data table for data table fusion.

[0043] Step S202: Determine the data distribution similarity between each sub - domain incremental label data table according to the data distribution characteristics of each sub - domain incremental label data table respectively.

[0044] This step mainly determines the data distribution similarity between each sub - domain incremental label data table based on the data distribution characteristics of each sub - domain incremental label data table. The data distribution characteristics of each sub - domain incremental label data table can be represented by vectors, and then the data distribution similarity between each sub - domain incremental label data table is calculated by a method for measuring the similarity between vectors.

[0045] Step S203: Generate a distribution approximation index tree for each sub - domain incremental label data table according to the data distribution similarity.

[0046] In this step, according to the data distribution similarity between each sub - domain incremental label data table, a distribution approximation index tree that can represent the fusion path sequence of these sub - domain incremental label data tables can be generated. For example, referring to Figure 3 , according to the data distribution similarity between each sub - domain incremental label data table, several sub - domain incremental label data tables that meet the data distribution similarity conditions can be respectively mapped to several leaf nodes under the same branch node, etc., thereby generating a distribution approximation index tree for these sub - domain incremental label data tables.

[0047] Step S204: Perform hierarchical fusion on each sub - domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table.

[0048] In this step, starting from the leaf nodes of the distribution approximation index tree, the sub-domain incremental label data tables corresponding to the leaf nodes under the same branch node can be fused to obtain a temporary incremental label data table and mapped to new leaf nodes. Finally, the temporary incremental label data tables corresponding to all the leaf nodes under the root node are fused to obtain a global incremental label data table (i.e., the hierarchical fusion result of all sub-domain incremental label data tables).

[0049] Step S205: Write the global incremental label data table into the data warehouse.

[0050] In this step, the globally incrementally labeled data table obtained by final fusion can be written into the Hive data warehouse and can also be synchronized to the ES cluster.

[0051] The incremental label data synchronization processing method of this embodiment obtains the data distribution characteristics of each sub-domain incremental label data table, determines the data distribution similarity between each sub-domain incremental label data table according to the data distribution characteristics of each sub-domain incremental label data table, generates a distribution approximation index tree for each sub-domain incremental label data table according to the data distribution similarity, hierarchically fuses each sub-domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table, and writes the global incremental label data table into the data warehouse. This solution can determine the data distribution similarity between each sub-domain incremental label data table according to the data distribution characteristics of each sub-domain incremental label data table, and accordingly establish a distribution approximation index tree for each sub-domain incremental label data table. According to this index tree, each sub-domain incremental label data table can be hierarchically fused to quickly obtain a global incremental label data table and written into the data warehouse. Thus, the index tree constructed based on data distribution similarity can achieve quickly and efficiently synthesizing each sub-domain incremental label data table into a global incremental label data table and writing it into the data warehouse on the spark on hive computing engine, so that the fusion of the global incremental label data table does not depend on the HBase cluster, reduces the resource consumption of the HBase cluster, avoids impacting the external query service performance of the HBase cluster, and achieves the effect of reducing resource costs while improving the timeliness of incremental label data synchronization.

[0052] In one embodiment, as Figure 3 shown, obtaining the data distribution characteristics of each sub-domain incremental label data table in step S201 may include:

[0053] Step S301: Perform re-partitioning processing on each sub-domain incremental label data table according to the same partitioning algorithm.

[0054] In this step, the incremental label data tables of each sub-domain can be repartitioned respectively using a unified partitioning algorithm (such as a unified Hash partitioning algorithm, without restricting to a specific partitioning algorithm, and the same partitioning algorithm can be used), and thus the incremental label data corresponding to each sub-domain incremental label data table in each of its partitions can be obtained.

[0055] Step S302: For the sub-domain incremental label data tables in the incremental label data tables of each sub-domain, determine the representative value of each partition according to the data corresponding to the sub-domain incremental label data table in each partition.

[0056] In this step, for each sub-domain incremental label data table, the representative value corresponding to the sub-domain incremental label data table in each partition can be determined. Among them, for each sub-domain incremental label data table, the representative value of each partition can be determined according to the data corresponding to the sub-domain incremental label data table in each partition (that is, the incremental label data corresponding to the sub-domain incremental label data table in each partition). For example, the representative value of the partition can be determined according to statistical values such as the average value and median of the incremental label data in the partition. Thus, the representative value of each partition of each sub-domain incremental label data table can be obtained.

[0057] Step S303: Obtain the data distribution characteristics of each sub-domain incremental label data table according to the representative value of each partition of each sub-domain incremental label data table.

[0058] In this step, the representative value of each partition of each sub-domain incremental label data table can be used as the data distribution characteristics of each sub-domain incremental label data table.

[0059] The solution of this embodiment can repartition the sub-domain incremental label data tables based on the same partitioning algorithm, determine the representative value of each partition, and obtain the data distribution characteristics accordingly. It can obtain the data distribution characteristics that can accurately measure the similarity between data tables more efficiently, providing effective data support for improving the hierarchical fusion efficiency subsequently.

[0060] In one embodiment, the step of repartitioning the incremental label data tables of each sub-domain respectively according to the same partitioning algorithm in step S301 may include:

[0061] Perform data sampling processing on the incremental label data tables of each sub-domain respectively according to a certain ratio to obtain the sampling data tables of each sub-domain incremental label data table; repartition the sampling data tables of each sub-domain incremental label data table respectively according to the same partitioning algorithm.

[0062] This embodiment mainly focuses on the situation where the sampling data distribution approximates the overall data distribution, compresses the operation time, and improves the overall synchronization processing efficiency. In this embodiment, data sampling processing can be performed on each sub-domain incremental label data table at an equal ratio of 0.05 to obtain the sampling data table of each sub-domain incremental label data table (i.e., the data table obtained through data sampling processing). Then, a unified partitioning algorithm (such as a unified Hash partitioning algorithm) is used to perform re-partitioning processing on the sampling data tables of each sub-domain incremental label data table respectively.

[0063] Therefore, determining the representative value of each partition according to the data in each partition corresponding to the sub-domain incremental label data table in step S302 above may include: determining the representative value of each partition according to the sampling data in each partition corresponding to the sub-domain incremental label data table. Among them, the above-mentioned use of a unified partitioning algorithm to perform re-partitioning processing on the sampling data tables of each sub-domain incremental label data table respectively can obtain the data in each partition of each sampling data table, and then for each sampling data table, the representative value of each partition can be obtained according to the sampling data in each partition.

[0064] In one embodiment, determining the representative value of each partition according to the sampling data in each partition corresponding to the sub-domain incremental label data table in the above embodiment may include:

[0065] Obtaining the representative value of each partition according to the average value of the sampling data in each partition corresponding to the sub-domain incremental label data table.

[0066] In this embodiment, for each sub-domain incremental label data table, the average value of the sampling data in each partition of its corresponding sampling data table can be obtained by performing average value calculation on the sampling data in each partition respectively, and then this average value can be used as the representative value of each partition, so that the basic data for accurately measuring the data distribution characteristics and similarities in the subsequent process can be obtained more efficiently.

[0067] In one embodiment, obtaining the data distribution characteristics of each sub-domain incremental label data table according to the representative value of each partition of each sub-domain incremental label data table in step S303 may include:

[0068] For each sub-domain incremental label data table, perform binarization processing on the representative value of each partition to obtain a bitmap vector; compress the bitmap vector into a numerical vector according to a certain number of bits to obtain the data distribution characteristics of each sub-domain incremental label data table.

[0069] In this embodiment, the data distribution characteristics of each sub-domain incremental label data table are accurately obtained by forming a numerical vector. In this embodiment, for each sub-domain incremental label data table, the representative values (such as average values) of each partition of the sub-domain incremental label data table can be binarized to obtain a bitmap vector, and then the bitmap vector can be compressed into a numerical vector according to, for example, 10 bits (corresponding to a certain number of bits). The compressed numerical vector is used as the data distribution characteristic, so as to obtain the data distribution characteristics of each sub-domain incremental label data table.

[0070] In one embodiment, determining the data distribution similarity between each sub-domain incremental label data table according to the data distribution characteristics of each sub-domain incremental label data table in step S202 may include:

[0071] Obtain the Euclidean distance between the corresponding sub-domain incremental label data tables according to the data distribution characteristics of each sub-domain incremental label data table; determine the data distribution similarity between each sub-domain incremental label data table according to the Euclidean distance.

[0072] In this embodiment, the data distribution similarity between each sub-domain incremental label data table can be measured mainly by the Euclidean distance. Among them, the above-mentioned numerical vector can be used as the data distribution characteristic. According to the numerical vectors of each sub-domain incremental label data table, calculate the Euclidean distance between the corresponding sub-domain incremental label data tables, and determine the data distribution similarity between each sub-domain incremental label data table according to the magnitude of the Euclidean distance, so that the data distribution similarity between each sub-domain incremental label data table can be quickly and accurately determined.

[0073] In one embodiment, performing hierarchical fusion on each sub-domain incremental label data table according to the distribution approximation index tree to obtain the global incremental label data table in step S204 may include:

[0074] Based on the depth-first mechanism, the lazy merge operator, and the lazy query operator of each sub-domain incremental label data table, obtain the fusion DAG task chain corresponding to the distribution approximation index tree; execute the fusion DAG task chain to perform hierarchical fusion on each sub-domain incremental label data table to obtain the global incremental label data table.

[0075] In this embodiment, a corresponding fused DAG task chain (a DAG task chain that can execute Spark calculations; DAG, Directed Acyclic Graph) can be generated according to the distributed approximation index tree, and the global incremental label data table can be generated by executing the fused DAG task chain. Among them, the lazy query operator of each sub-domain incremental label data table can be mapped to the corresponding leaf node in the distributed approximation index tree, so that based on the depth-first mechanism, the lazy merge operator, and the lazy query operators of each sub-domain incremental label data table, a fused DAG task chain corresponding to the distributed approximation index tree is generated, and then the fused DAG task chain is executed to perform step-by-step fusion on each sub-domain incremental label data table to obtain the global incremental label data table. Among them, when the fused DAG task chain is executed, it traverses all non-leaf nodes except the root node of the distributed approximation index tree according to the depth-first mechanism, and preferentially executes the lazy query operators of all subordinate leaf nodes, then executes the lazy merge operator, and maps the temporarily generated incremental label data table after merging to a new leaf node. Finally, the temporarily generated incremental label data tables of all leaf nodes corresponding to the root node are executed with the lazy merge operator again to finally generate the global incremental label data table, and the global incremental label data table can be written into the Hive data warehouse.

[0076] In one embodiment, an incremental label data synchronization processing method is provided. Refer to Figures 4 to 6 , this method can efficiently fuse each sub-domain incremental label data table (such as the user portrait label data table of each business domain) into a global incremental label data table on the spark on hive computing engine. The incremental label data table can be label data generated with update frequencies such as daily update, weekly update, monthly update, etc. This method can include the following three processing parts:

[0077] In the first processing part, the range of incremental tag data to be fused can be verified first. The main purpose is to screen out the incremental tag data that needs to be imported into ES to build a retrieval index. Among them, the configuration information of each sub-domain tag data table (tag information table) can be loaded first, which can be used to load all sub-domain tag schema information. Then, the ES cluster index Mapping configuration information can be loaded, which can be used to load the tag schema information that needs to be imported into the ES cluster. Then, the progress information of the offline calculation task of each sub-domain tag data table can be loaded, which can be used to load the latest data partition information when the fusion of each sub-domain tag data table was completed last time. Next, the ES cluster push tag schema information and data legality verification for the currently to-be-fused data can be performed, which serves to prevent the fusion of non-compliant dirty data. Thus, the incremental tag data tables of each sub-domain that need to be fused can be obtained, and the query DSL for each sub-domain tag data table can be generated and encapsulated as a lazy query operator, which serves to determine the field range and data partition range for querying each sub-domain tag data table. Among them, this processing part can verify the configuration information of each sub-domain tag data table (tag information table), the current data partition of the tag data table corresponding to each sub-domain, and the ES cluster Mapping configuration information, generate the query DSL for each sub-domain tag data table, and encapsulate it as a lazy query operator.

[0078] In the second processing part, related processing for generating an index tree based on bitmap vectors can be performed. Its function is that, due to the huge difference in the data volume of each sub-domain incremental label data table, the processing for solving the adaptive selection of the Join fusion order can represent the data distribution similarity between each sub-domain incremental label data table through vector distance. Among them, each sub-domain incremental label data table can be sampled at a ratio of 0.05 respectively, so as to approximate the overall data distribution with the sampled data distribution and compress the operation time. Then, re-partitioning can be performed according to a unified Hash partitioning algorithm. Among them, there is no specific selection requirement for the Hash partitioning algorithm, as long as the same one is used. Then, the average value of the data in the same partition of each sub-domain incremental label data table can be calculated and binarized into a bitmap vector. Then, the binarized vector can be compressed into a numerical vector according to 10 bits. Then, a four-dimensional index tree can be generated based on the Euclidean distance measurement of this numerical vector, and thus the optimal fusion calculation path order of each sub-domain incremental label data table can be obtained. Among them, 1280 indicates the number of partitions calculated in the Spark cluster. The number of partitions can be determined according to the sub-domain incremental label data table with the largest data volume and the available memory or hard disk storage resources of the Spark cluster. If the selection is too large, the calculation time will be longer; if the selection is too small, an OOM exception (Out OfMemory Error, memory overflow exception) is likely to occur during the task calculation process. In the compression process, the 128 dimensions shown follow the binary bit width in the related hardware algorithm, and there is no mandatory requirement in the method of this application to compress it to 128 dimensions. Among them, four dimensions mean that four sub-domain incremental label data tables can be joined and fused at a time. In this processing part, an index tree can be generated based on the bitmap vector. Each sub-domain incremental label data table can be partitioned and sampled according to a unified partitioning rule, and the number of sampled data items in each partition can be converted into a bitmap vector. Then, an index tree can be generated according to the Hamming Distance or, after dimensionality reduction (merging the 10-bit binary values into a new column), according to the Euclidean distance.

[0079] In the third processing part, a DAG scheduling task chain (fused DAG task chain) of a custom Hash Table between multiple Hive tables can be generated based on a four-dimensional index tree. Among them, the lazy query operators of each sub-domain incremental label data table can be mapped to the corresponding leaf nodes in the four-dimensional index tree. Accordingly, it can be transformed into a DAG fusion task chain that can be executed by Spark. During the execution of the DAG fusion task chain, all non-leaf nodes of the four-dimensional index tree except the root node can be traversed according to the depth-first mechanism, and the lazy query operators of all subordinate leaf nodes (such as subtask 11 to subtask 14, etc.) are preferentially executed, and then the lazy merge operator (such as subtask 1, etc.) is executed, and the temporarily generated incremental label data table after merging is mapped to a new leaf node (such as subtask result 1, etc.). Thus, finally, the temporarily generated incremental label data tables of all child nodes corresponding to the root node are again executed with the lazy merge operator (root task), and the global incremental label data table (fusion result) is finally generated and written into the Hive data warehouse to complete the fusion of the overall global incremental label data table.

[0080] Based on the high-dimensional index algorithm of this embodiment, the sub-domain incremental label data tables with similar data distribution characteristics can be inserted into the index tree, and then according to the index tree, the sub-domain incremental label data tables of the same leaf node can be Hash Joined (hashed joined) in the order of sub-partition index numbers into a temporarily generated incremental label data table, so as to gradually fuse and finally generate the global incremental label data table, making the fusion of the global incremental label data table no longer dependent on the HBase cluster, avoiding the impact on the external query service performance of the HBase cluster, reducing the resource consumption of the HBase cluster, and making the running time greatly reduced and the consumption of server queue resources significantly reduced when executing the global incremental label data table fusion task.

[0081] It should be understood that although each step in the flowcharts involved in the above-described embodiments is sequentially shown according to the indication of the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same moment, but can be executed at different moments, and the execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0082] Based on the same inventive concept, an embodiment of the present application further provides an incremental tag data synchronization processing device for implementing the incremental tag data synchronization processing method involved above. The implementation solutions provided by this device for solving problems are similar to the implementation solutions recorded in the above method. Therefore, the specific limitations in one or more embodiments of the incremental tag data synchronization processing device provided below can refer to the limitations on the incremental tag data synchronization processing method in the above text, and will not be repeated here.

[0083] In one embodiment, as Figure 7 shown, an incremental tag data synchronization processing device is provided. The device 700 may include:

[0084] A feature acquisition module 701, configured to acquire the data distribution features of each sub-domain incremental tag data table;

[0085] A similarity determination module 702, configured to determine the data distribution similarity between the sub-domain incremental tag data tables according to the data distribution features of each sub-domain incremental tag data table;

[0086] An index tree generation module 703, configured to generate a distribution approximation index tree for each sub-domain incremental tag data table according to the data distribution similarity;

[0087] A global table obtaining module 704, configured to perform hierarchical fusion on each sub-domain incremental tag data table according to the distribution approximation index tree to obtain a global incremental tag data table;

[0088] A data table writing module 705, configured to write the global incremental tag data table into a data warehouse.

[0089] In one embodiment, the feature acquisition module 701 is configured to perform re-partitioning processing on each sub-domain incremental tag data table according to the same partitioning algorithm; for the sub-domain incremental tag data tables in each sub-domain incremental tag data table, determine the representative value of each partition according to the data in each partition corresponding to the sub-domain incremental tag data table; according to the representative values of each partition of each sub-domain incremental tag data table, obtain the data distribution features of each sub-domain incremental tag data table.

[0090] In one embodiment, the feature acquisition module 701 is configured to perform data sampling processing on each sub-domain incremental tag data table according to a certain ratio to obtain the sampling data tables of each sub-domain incremental tag data table; perform re-partitioning processing on the sampling data tables of each sub-domain incremental tag data table according to the same partitioning algorithm; the feature acquisition module 701 is configured to determine the representative value of each partition according to the sampling data in each partition corresponding to the sub-domain incremental tag data table.

[0091] In one embodiment, the feature acquisition module 701 is configured to obtain the representative value of each partition according to the average value of the sampled data corresponding to the sub-domain incremental label data table in each partition.

[0092] In one embodiment, the feature acquisition module 701 is configured to perform binarization processing on the representative value of each partition for each sub-domain incremental label data table to obtain a bitmap vector; compress the bitmap vector into a numerical vector according to a certain number of bits to obtain the data distribution feature of each sub-domain incremental label data table.

[0093] In one embodiment, the similarity determination module 702 is configured to obtain the Euclidean distance between the corresponding sub-domain incremental label data tables according to the data distribution features of the sub-domain incremental label data tables; determine the data distribution similarity between the sub-domain incremental label data tables according to the Euclidean distance.

[0094] In one embodiment, the global table obtaining module 704 is configured to obtain the fusion DAG task chain corresponding to the distribution approximation index tree based on the depth-first mechanism, the lazy merge operator, and the lazy query operator of each sub-domain incremental label data table; execute the fusion DAG task chain to perform hierarchical fusion on each sub-domain incremental label data table to obtain the global incremental label data table.

[0095] Each module in the above incremental label data synchronization processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0096] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as tag data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements an incremental tag data synchronization processing method.

[0097] Those skilled in the art can understand that Figure 8 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0098] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0099] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0100] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0102] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0103] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0104] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An incremental label data synchronization processing method, characterized in that, The method includes: Obtaining the data distribution characteristics of each sub-domain incremental label data table; Determining the data distribution similarity between the sub-domain incremental label data tables according to the data distribution characteristics of each sub-domain incremental label data table; Generating a distribution approximation index tree for each sub-domain incremental label data table according to the data distribution similarity; Hierarchically fusing each sub-domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table; Writing the global incremental label data table into a data warehouse.

2. The method according to claim 1, wherein The obtaining the data distribution characteristics of each sub-domain incremental label data table includes: Performing re-partitioning processing on each sub-domain incremental label data table respectively according to the same partitioning algorithm; For the sub-domain incremental label data tables in each sub-domain incremental label data table, determining the representative value of each partition according to the data corresponding to the sub-domain incremental label data table in each partition; Obtaining the data distribution characteristics of each sub-domain incremental label data table according to the representative values of each partition of each sub-domain incremental label data table.

3. The method according to claim 2, wherein The performing re-partitioning processing on each sub-domain incremental label data table respectively according to the same partitioning algorithm includes: Performing data sampling processing on each sub-domain incremental label data table respectively according to a certain proportion to obtain the sampling data tables of each sub-domain incremental label data table; Performing re-partitioning processing on the sampling data tables of each sub-domain incremental label data table respectively according to the same partitioning algorithm; The determining the representative value of each partition according to the data corresponding to the sub-domain incremental label data table in each partition includes: Determining the representative value of each partition according to the sampling data corresponding to the sub-domain incremental label data table in each partition.

4. The method according to claim 3, characterized in that, The determining the representative value of each partition according to the sampling data corresponding to the sub-domain incremental label data table in each partition includes: Obtaining the representative value of each partition according to the average value of the sampling data corresponding to the sub-domain incremental label data table in each partition.

5. The method according to any one of claims 2 to 4, characterized in that The obtaining the data distribution characteristics of each sub-domain incremental label data table according to the representative values of each partition of each sub-domain incremental label data table includes: For each sub-domain incremental label data table, performing binarization processing on the representative values of each partition to obtain a bitmap vector; Compressing the bitmap vector into a numerical vector according to a certain number of bits to obtain the data distribution characteristics of each sub-domain incremental label data table.

6. The method according to claim 5, wherein The determining the data distribution similarity between the sub-domain incremental label data tables according to the data distribution characteristics of each sub-domain incremental label data table includes: Obtaining the Euclidean distance between the corresponding sub-domain incremental label data tables according to the data distribution characteristics of each sub-domain incremental label data table; Determining the data distribution similarity between the sub-domain incremental label data tables according to the Euclidean distance.

7. The method according to claim 1, wherein The hierarchically fusing each sub-domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table includes: Obtain the fused DAG task chain corresponding to the distribution approximation index tree based on the depth-first mechanism, the lazy merge operator, and the lazy query operator for each sub-domain incremental label data table; Execute the fused DAG task chain to perform hierarchical fusion on each sub-domain incremental label data table to obtain a global incremental label data table.

8. An incremental label data synchronization processing device, characterized in that, The device includes: A feature acquisition module, configured to acquire the data distribution features of each sub-domain incremental label data table; A similarity determination module, configured to determine the data distribution similarity between the sub-domain incremental label data tables according to the data distribution features of each sub-domain incremental label data table; An index tree generation module, configured to generate a distribution approximation index tree for each sub-domain incremental label data table according to the data distribution similarity; A global table obtaining module, configured to perform hierarchical fusion on each sub-domain incremental label data table according to the distribution approximation index tree to obtain a global incremental label data table; A data table writing module, configured to write the global incremental label data table into a data warehouse.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When this computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images