Multi-source heterogeneous data fusion method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide a multi-source heterogeneous data fusion method and apparatus, an electronic device and a storage medium. The method comprises the steps of cleaning and formatting data acquired by multi-source heterogeneous increments on that day, and then writing the cleaned and formatted data into a cold region data table and a hot region data table; performing total data deduplication fusion on the cold region data in the cold region data table at an interval of N days, wherein N is a natural number greater than or equal to 2; identifying a new active account from the data acquired by the multi-source heterogeneous increment, and writing cold area data corresponding to the new active account in the cold area data table into a hot area data table; the hot area data in the hot area data table is subjected to full data deduplication fusion at an interval of M days, the hot area data table is emptied regularly, M is a natural number larger than or equal to 1, and M is smaller than N. According to the embodiment of the invention, the operation performance is effectively improved under limited big data server resources, the operation resources are saved, the system resource utilization rate is improved, and the server resource cost is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a method, device, electronic device, storage medium and computer program product for fusing multi-source heterogeneous data. Background Art

[0002] With the rapid development of information technology represented by big data and the deepening of digital transformation, data has become an important asset for enterprises. It has become a trend in technological development to integrate multi-source heterogeneous data with different storage modes, logical structures, and language semantics, eliminate biases or errors that may exist in a single data source, ensure data consistency and accuracy, manage massive, large, complex, and chaotic data in an efficient, unified, and low-cost manner, and mine data value to generate new application services.

[0003] With the increase in data sources, more and more multi-source heterogeneous data are being integrated, and the execution of workflow tasks is very time-consuming. It is a great challenge to complete the full integration of tens of billions of data on limited big data cluster servers.

[0004] Therefore, how to achieve efficient fusion of multi-source heterogeneous data has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, device, electronic device, storage medium and computer program product for fusing multi-source heterogeneous data, which can effectively improve computing performance, save computing resources, improve system resource utilization and reduce server resource costs under limited big data server resources.

[0006] According to a first aspect of an embodiment of the present application, a method for fusing multi-source heterogeneous data is provided, the method comprising: cleaning and formatting the data collected by multi-source heterogeneous increments on the same day and writing them into a cold zone data table and a hot zone data table respectively; performing full data deduplication fusion on the cold zone data in the cold zone data table at an interval of N days, where N is a natural number greater than or equal to 2; identifying new active accounts from the multi-source heterogeneous incrementally collected data, and writing the cold zone data corresponding to the new active account in the cold zone data table into the hot zone data table; performing full data deduplication fusion on the hot zone data in the hot zone data table at an interval of M days, and regularly clearing the hot zone data table, where M is a natural number greater than or equal to 1, and M is less than N.

[0007] In some embodiments of the present application, the data collected from multi-source heterogeneous increments on the same day are cleaned and formatted and then written into the cold zone data table and the hot zone data table respectively. This also includes: using the full data fusion table of the data warehouse summary layer as the cold zone data table, and adding a hot zone data table. The data business fields of the hot zone data table are consistent with those of the cold zone data table, and the partition field only retains the execution date partition.

[0008] In some embodiments of the present application, the cold zone data table has a dual partition structure, the outer layer is partitioned according to the execution date, and the inner layer is partitioned according to the modulus of 10 after hashing the data primary key or unique index; the cold zone data in the cold zone data table is deduplicated and merged at an interval of N days, specifically: the data in the partition with the latest execution date is obtained at an interval of N days, and deduplicated and merged cyclically according to the inner layer modulus partition.

[0009] In some embodiments of the present application, N is set based on at least one of the amount of fused data, server resources, and the execution time of the deduplication fusion workflow for the full amount of cold zone data.

[0010] In some embodiments of the present application, the hot zone data in the hot zone data table is deduplicated and merged at an interval of M days, and the hot zone data table is cleared regularly, and also includes: when the total amount of data in the hot zone data table reaches a preset threshold, the hot zone data table is cleared, and the preset threshold is determined based on server resources and running time.

[0011] In some embodiments of the present application, the method further includes: querying the business data of the dimensional information from the integrated hot zone data table based on the key dimensional information of the incremental business data of the new active account, and performing incremental calculation of data application statistical indicators based on the business data to update the statistical indicators.

[0012] According to a second aspect of an embodiment of the present application, a device for fusing multi-source heterogeneous data is provided, the device comprising: a data writing module, for cleaning and formatting the data collected from multi-source heterogeneous increments on the same day and writing them into a cold zone data table and a hot zone data table respectively; a cold zone fusion module, for performing full data deduplication fusion on the cold zone data in the cold zone data table at an interval of N days, where N is a natural number greater than or equal to 2; an account identification module, for identifying a new active account from the data collected from multi-source heterogeneous increments, and writing the cold zone data corresponding to the new active account in the cold zone data table into the hot zone data table; a hot zone fusion module, for performing full data deduplication fusion on the hot zone data in the hot zone data table at an interval of M days, and performing timed clearing of the hot zone data table, where M is a natural number greater than or equal to 1, and M is less than N.

[0013] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect above.

[0014] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect above is implemented.

[0015] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes computer-executable instructions, which, when executed, enable at least one processor to perform the method described in the first aspect above.

[0016] According to the above technical solution, the fusion solution of multi-source heterogeneous data in the embodiment of the present application cleans and formats the data collected by multi-source heterogeneous increments on the same day and writes them into the cold zone data table and the hot zone data table respectively. Identify new active accounts from the data collected by multi-source heterogeneous increments, and write the cold zone data corresponding to the new active accounts in the cold zone data table into the hot zone data table, so as to separate and store the cold and hot data. In the embodiment of the present application, the cold zone data in the cold zone data table is deduplicated and merged with full data at intervals of N days, and the hot zone data in the hot zone data table is deduplicated and merged with full data at intervals of M days, and the hot zone data table is cleared regularly, where M and N are natural numbers greater than or equal to 1, and M is less than N. Therefore, the cold zone data in the embodiment of the present application is deduplicated and merged with low frequency, and the daily incremental collection data and the cold zone data of the new active account are regarded as hot zone data, and high frequency full deduplication and fusion are performed. The embodiment of the present application effectively improves computing performance, saves computing resources, improves system resource utilization, and reduces server resource costs under limited big data server resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flow chart of a method for fusing multi-source heterogeneous data provided by an embodiment of the present application;

[0019] Figure 2 is a flow chart of a method for fusing multi-source heterogeneous data provided by another embodiment of the present application;

[0020] Figure 3 is a flowchart of a method for fusing multi-source heterogeneous data provided by another embodiment of the present application;

[0021] Figure 4 is a flowchart of a method for fusing multi-source heterogeneous data provided by another embodiment of the present application;

[0022] Figure 5 It is a structural schematic diagram of a multi-source heterogeneous data fusion device provided by an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the structure of an electronic device used in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0025] Reference is made to the accompanying drawings in the following detailed description, which form a part of the detailed description and illustrate exemplary embodiments. In addition, it is to be understood that other embodiments may be utilized and structural and / or logical changes may be made without departing from the scope of the claimed subject matter. It should also be noted that directions and references (e.g., up, down, top, bottom, etc.) may be used merely to facilitate the description of features in the accompanying drawings. Therefore, the following detailed description is not to be understood in a limiting sense, and the scope of the claimed subject matter is limited only by the appended claims and their equivalents.

[0026] In the following description, numerous details are set forth. However, it will be apparent to those skilled in the art that the embodiments herein can be practiced without these specific details. In some cases, known methods and devices are shown in block diagram form, rather than in detail, to avoid blurring the embodiments herein. References to "embodiment" or "one embodiment" or "some embodiments" throughout this specification mean that the specific features, structures, functions or characteristics described in conjunction with the embodiment are included in at least one embodiment herein. Therefore, the phrases "in an embodiment" or "in one embodiment" or "some embodiments" appearing throughout this specification do not necessarily refer to the same embodiment. In addition, in one or more embodiments, specific features, structures, functions or characteristics may be combined in any suitable manner. For example, the first embodiment may be combined with the second embodiment in any case where the specific features, structures, functions or characteristics associated with the two embodiments are not mutually exclusive.

[0027] As used in the description and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0028] The terms "coupled" and "connected" together with their derivatives may be used in this article to describe the functional or structural relationship between components. It should be understood that these terms are not intended to be synonymous with each other. On the contrary, in a particular embodiment, "connected" can be used to indicate that two or more elements are in direct physical, optical or electrical contact with each other. "Coupled" can be used to indicate that two or more elements are in direct or indirect (with other intermediate elements between them) physical or electrical contact with each other, and / or that two or more elements cooperate or interact with each other (e.g., as in a cause-and-effect relationship).

[0029] As used herein, the terms "above," "below," "between," and "on" refer to the relative position of one component or material with respect to other components or materials where such physical relationship is noteworthy. For example, in the context of materials, a material or materials disposed above or below another material may be in direct contact, or may have one or more intervening materials. Also, a material disposed between two materials or materials may be in direct contact with both layers, or may have one or more intervening layers. In contrast, a first material or material "above" a second material or material is in direct contact with the second material / materials. Similar distinctions are made in the context of component assembly.

[0030] As used throughout the description herein and in the claims, a list of items connected by the term "at least one of" or "one or more of" may mean any combination of the listed items. For example, the phrase "at least one of A, B, or C" may mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0031] The term "circuit" or "module" may refer to one or more passive and / or active components that are arranged to cooperate with each other to provide a desired functionality. The term "signal" may refer to at least one current signal, voltage signal, or magnetic signal. The terms "substantially," "close," "approximately," "close to," and "approximately" generally refer to within + / - 10% of a target value.

[0032] In general, based on the fusion processing of multi-source heterogeneous data at the level of tens of billions, according to the characteristics of storage media and structure of different data sources, sqoop (Sqoop is a tool for efficiently transmitting large amounts of data between Hadoop and relational database management systems. Its Chinese name can be directly translated as "Sqoop") offline synchronization, sparkstreaming (Spark streaming computing framework) consumer message queue near real-time synchronization or some tools are selected to achieve data collection, and incremental data is written into the big data cluster. Through data cleaning, noise and outliers in the data are removed to improve data quality, and then the incremental data is converted into a format, and the incremental data from multiple different data sources are unified into a data storage format, and the accumulated full fusion data is deduplicated, and new data fusion results are generated and stored in the fusion table of the dws layer (data warehouse summary layer), completing the full fusion of multi-source heterogeneous data.

[0033] Usually, for data security management, the dws layer data fusion table will be partitioned and stored according to the execution date, and all the incremental data after cleaning and format conversion of multi-source heterogeneous data sources will be written into the dws layer fusion table T-day partition, and then the full amount of data in the fusion table T-day partition will be deduplicated and written into the T+1 day partition, so as to repair the data when an exception occurs in the big data cluster or workflow task scheduling platform cluster server or application service. Therefore, the dws layer data fusion table will save the data of multiple execution date partitions and regularly delete the earlier execution date partitions.

[0034] The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention.

[0035] Fusion method of multi-source heterogeneous data

[0036] The present application provides a method for fusing multi-source heterogeneous data. The method for fusing multi-source heterogeneous data is described in detail below through multiple embodiments.

[0037] See also Figure 1 , the method comprising:

[0038] Step S1: Clean and format the multi-source heterogeneous incremental data collected on the day and write them into the cold area data table and the hot area data table respectively.

[0039] Specifically, the data collected by multi-source heterogeneous increments on the current day in the embodiment of the present application is the data collected by multi-source heterogeneous increments yesterday, that is, the data collected by multi-source heterogeneous increments one day before the current time point.

[0040] In some specific implementations of the embodiments of this application, see Figure 2 , the method further includes before step S1:

[0041] Step S0: Use the full data fusion table of the data warehouse summary layer as the cold zone data table, and add a hot zone data table. The data business fields of the hot zone data table are consistent with those of the cold zone data table, and the partition field only retains the execution date partition.

[0042] This application cleans and formats the data collected from multi-source heterogeneous incremental data on the same day and writes it into the cold zone data table. The cold zone data table is a full data fusion table at the data warehouse summary layer. The cold zone data accumulates multi-source heterogeneous incremental data collected on multiple dates, including data that is not frequently updated, and occupies a large storage space.

[0043] This application also cleans and formats the data collected from multiple sources on the same day and writes it into the hot zone data table. The data business fields of the hot zone data table are consistent with those of the cold zone data table, but the partition field only retains the execution date partition. Therefore, the hot zone data table does not need to be initialized, only includes frequently updated data, and occupies little storage space.

[0044] The embodiment of the present application performs cold and hot data partitioning through step S0, so that in the subsequent execution process, the cold area data table and the hot area data table are used to store different data respectively, so as to achieve full data deduplication and fusion processing with different frequencies for different data.

[0045] Step S2: deduplication and fusion of all cold zone data in the cold zone data table are performed at intervals of N days, where N is a natural number greater than or equal to 2.

[0046] Specifically, N is set according to at least one of the amount of fused data, server resources, and the execution time of the deduplication fusion workflow of the full amount of cold zone data.

[0047] In the embodiment of the present application, the cold zone data is fully deduplicated and integrated at intervals of N days, where N is a natural number greater than or equal to 2, so that the cold zone data is no longer fully deduplicated and integrated every day. The embodiment of the present application reduces the frequency of full data integration and releases the server computing resources occupied by daily data deduplication and integration.

[0048] Step S3: Identify new active accounts from the data collected from multi-source heterogeneous increments, and write the cold zone data corresponding to the new active accounts in the cold zone data table into the hot zone data table.

[0049] New active accounts are identified from multi-source heterogeneous incremental formatted data every day, and data about new active accounts is obtained from cold zone data and written into the hot zone data table. Therefore, the hot zone data not only contains the newly added or updated data of the new active account on the day, but also includes its historical data. When performing data deduplication and fusion, all data of the active account can be updated to ensure the integrity of the data and the accuracy of the calculation indicators in the subsequent incremental calculation.

[0050] Step S4: deduplication and fusion of all the hot zone data in the hot zone data table are performed at intervals of M days, and the hot zone data table is cleared regularly, where M is a natural number greater than or equal to 1, and M is less than N.

[0051] The hot zone data is deduplicated and merged every day. As the hot zone data gradually increases, the hot zone data table is cleared regularly.

[0052] According to the above technical solution, the cold zone data is deduplicated and fused at a low frequency, and the daily incremental collected data and the cold zone data of new active accounts are regarded as hot zone data, and high frequency full deduplication and fusion are performed. The embodiment of the present application effectively improves the computing performance under limited big data server resources, saves computing resources, improves the utilization rate of system resources, and reduces the server resource cost.

[0053] In some specific implementations of the embodiments of the present application, the cold zone data table has a dual partition structure, the outer layer is partitioned according to the execution date, and the inner layer is partitioned according to the hash value of the data primary key or unique index and modulo 10.

[0054] Step S2 is specifically:

[0055] Get the data in the partition with the latest execution date at intervals of N days, and perform de-duplicate fusion according to the inner modulus partition in a loop.

[0056] The amount of cold zone data in the embodiment of the present application is relatively large, usually including tens of billions of data. When the embodiment of the present application adopts the above-mentioned scheme to perform full deduplication and fusion on the cold zone data, the data in the partition with the latest execution date is taken, and the deduplication and fusion are performed cyclically according to the inner layer modulus partition, thereby improving the deduplication and fusion performance and efficiency.

[0057] In some specific implementations of the embodiments of this application, see Figure 3 , the method further comprising:

[0058] Step S5: Based on the key dimension information of the incremental business data of the new active account, query the business data of the dimension information from the integrated hot zone data table, perform incremental calculation of data application statistical indicators based on the business data, and update the statistical indicators.

[0059] Specifically, the embodiment of the present application identifies key dimension information of incremental business data of active accounts from multi-source heterogeneous incremental formatted data every day, and queries business data related to the dimension information from the integrated hot zone data table.

[0060] The above business data is used for subsequent related data application statistical indicator incremental calculation, and the related indicator results are updated. In this case, the business data is complete, and the server computing resources and execution time occupied by the full data indicators are reduced.

[0061] In some specific implementations of the embodiments of the present application, step S4 further includes:

[0062] When the total amount of data in the hot zone data table reaches a preset threshold, the hot zone data table is cleared. The preset threshold is determined based on server resources and running time.

[0063] As the hot zone data gradually increases, an assessment is performed based on existing server resources and operation time. When the total amount of hot zone data reaches a preset threshold, the hot zone data table is cleared periodically without affecting the daily data indicator increment calculation in step S5.

[0064] In some specific implementations of the embodiments of the present application, the above steps are all completed by a workflow task scheduling platform such as DolphinScheduler, without the need for human intervention, thus reducing the complexity and error rate of manual operations.

[0065] See also Figure 4 , the implementation of the embodiment of the present application is further illustrated below through a specific application scenario.

[0066] Collect multi-source heterogeneous incremental data;

[0067] Convert the format of multi-source heterogeneous incremental data to form multi-source incremental formatted data;

[0068] Write multi-source incremental formatted data into cold area data;

[0069] At intervals of N days, for example, the cold zone data is deduplicated and integrated at a regular low frequency to form cold zone integrated data;

[0070] Identify new active accounts from multiple sources of incremental formatted data;

[0071] Write the cold zone data corresponding to the new active account in the cold zone data table into the hot zone data;

[0072] At intervals of M days, for example, day T = 1, perform deduplication fusion on all hot zone data to form hot zone fusion data;

[0073] The hot zone fusion data is cleared regularly to form a hot zone empty table;

[0074] Based on the key dimension information of the incremental business data of the new active account, query the business data of the dimension information from the integrated hot zone data table;

[0075] Perform incremental calculations on data application statistical indicators based on business data and update statistical indicators.

[0076] Fusion device for multi-source heterogeneous data

[0077] Figure 5 FIG. 1 is a schematic diagram of a multi-source heterogeneous data fusion device provided by an embodiment of the present application. Figure 5 As shown, the multi-source heterogeneous data fusion device 500 may include:

[0078] The data writing module 501 is used to clean and format the data collected from multiple sources of heterogeneous increments on the day and write them into the cold area data table and the hot area data table respectively.

[0079] The cold zone fusion module 502 is used to perform deduplication fusion on the cold zone data in the cold zone data table at intervals of N days, where N is a natural number greater than or equal to 2.

[0080] The account identification module 503 is used to identify new active accounts from the data collected from multi-source heterogeneous increments, and write the cold zone data corresponding to the new active accounts in the cold zone data table into the hot zone data table.

[0081] The hot zone fusion module 504 is used to perform full data deduplication fusion on the hot zone data in the hot zone data table at an interval of M days, and to periodically clear the hot zone data table, where M is a natural number greater than or equal to 1, and M is less than N.

[0082] According to the above technical solution, the cold zone data of the embodiment of the present application is subjected to low-frequency full-volume deduplication and fusion, and the daily incremental collection data and the cold zone data of the new active accounts are regarded as hot zone data, and high-frequency full-volume deduplication and fusion are performed. The embodiment of the present application effectively improves the computing performance under limited big data server resources, saves computing resources, improves the utilization rate of system resources, and reduces the server resource cost.

[0083] Electronic devices

[0084] Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device. Figure 6As shown, the electronic device 600 may include: a processor 602, a communications interface 604, a memory 606, and a communication bus 608. Among them:

[0085] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0086] The communication interface 604 is used to communicate with other electronic devices or servers.

[0087] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps in any of the aforementioned tax management method embodiments.

[0088] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0089] The processor 602 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0090] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0091] The program 610 may be specifically used to enable the processor 602 to execute the communication control method in any of the aforementioned embodiments.

[0092] The specific implementation of each step in program 610 can refer to the corresponding description of the corresponding steps and units in any of the aforementioned method embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiments, which will not be repeated here.

[0093] Storage Media

[0094] In this embodiment, a computer-readable storage medium is provided, which stores instructions for causing a machine to execute the communication control method as described herein. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0095] In this case, the program code read from the storage medium itself can implement the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of the present application.

[0096] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0097] It should be noted that the above communication control method embodiment is implemented based on the aforementioned method embodiment. The specific content and beneficial effects can be found in the description of the aforementioned method embodiment, which will not be repeated here.

[0098] In the description of this application, the terms "first" and "second" are only used to facilitate the description of different components or names, and cannot be understood as indicating or implying a sequential relationship, relative importance, or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0099] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0100] It should be noted that although the specific embodiments of the present application are described in detail in conjunction with the accompanying drawings, this should not be construed as limiting the scope of protection of the present application. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of the present application.

[0101] The examples of the embodiments of the present application are intended to concisely illustrate the technical features of the embodiments of the present application so that those skilled in the art can intuitively understand the technical features of the embodiments of the present application, and are not intended to be improper limitations of the embodiments of the present application.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for fusing multi-source heterogeneous data, characterized in that: The method comprises: The multi-source heterogeneous incremental data collected on the day are cleaned and formatted, and then written into the cold zone data table and the hot zone data table respectively; Performing full data deduplication and fusion on the cold area data in the cold area data table at intervals of N days, where N is a natural number greater than or equal to 2; Identify new active accounts from the multi-source heterogeneous incrementally collected data, and write the cold zone data corresponding to the new active accounts in the cold zone data table into the hot zone data table; The hot zone data in the hot zone data table are deduplicated and merged at an interval of M days, and the hot zone data table is cleared regularly, where M is a natural number greater than or equal to 1, and M is less than N.

2. The method according to claim 1, characterized in that The data collected from the multi-source heterogeneous increments on the day are cleaned and formatted and then written into the cold zone data table and the hot zone data table respectively, which also includes: The full data fusion table of the data warehouse summary layer is used as the cold area data table, and the hot area data table is added. The data business fields of the hot area data table are consistent with those of the cold area data table, and the partition field only retains the execution date partition.

3. The method according to claim 2, characterized in that The cold zone data table has a dual partition structure, the outer layer is partitioned according to the execution date, and the inner layer is partitioned according to the modulus of 10 after hashing the data primary key or unique index; The cold zone data in the cold zone data table are deduplicated and merged at an interval of N days, specifically: Get the data in the partition with the latest execution date at intervals of N days, and perform de-duplicate fusion according to the inner modulus partition in a loop.

4. The method according to claim 1, characterized in that The N is set according to at least one of the amount of fused data, server resources, and the execution time of the deduplication fusion workflow of the full amount of cold zone data.

5. The method according to claim 1, characterized in that The step of performing deduplication fusion on all the hot zone data in the hot zone data table at an interval of M days and regularly clearing the hot zone data table further includes: When the total amount of data in the hot zone data table reaches a preset threshold, the hot zone data table is cleared. The preset threshold is determined according to server resources and operation time.

6. The method according to claim 1, characterized in that The method further comprises: According to the key dimension information of the incremental business data of the new active account, the business data of the dimension information is queried from the integrated hot zone data table, and the data application statistical indicator incremental calculation is performed based on the business data to update the statistical indicator.

7. A device for fusing multi-source heterogeneous data, characterized in that: The device comprises: The data writing module is used to clean and format the data collected from multiple sources and heterogeneous increments on the same day and write them into the cold zone data table and the hot zone data table respectively; A cold zone fusion module, used to perform full data deduplication fusion on the cold zone data in the cold zone data table at intervals of N days, where N is a natural number greater than or equal to 2; An account identification module, configured to identify new active accounts from the multi-source heterogeneous incrementally collected data, and write the cold zone data corresponding to the new active account in the cold zone data table into the hot zone data table; The hot zone fusion module is used to perform full data deduplication fusion on the hot zone data in the hot zone data table at an interval of M days, and to perform regular clearing of the hot zone data table, where M is a natural number greater than or equal to 1, and M is less than N.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the method as described in any one of claims 1-6.

9. A computer storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product is tangibly stored on a computer readable medium and comprises computer executable instructions which, when executed, cause at least one processor to perform the method according to any one of claims 1-6.

Images