A new method for multi-aperture data fusion analysis

Abstract

The application discloses a new method for multi-caliber data fusion analysis, and belongs to the technical field of data analysis. The method comprises the following steps: obtaining metadata of a business data source by using a data exploration technology, and establishing a basic dimension table; combining data calibers of various systems to construct a data caliber cube; establishing a data source layer by using a data aggregation technology; constructing a data cube by comprehensively combining the data source layer and the data caliber cube, and writing data indexes into the corresponding data caliber cube; based on a business data caliber, traversing the data caliber cube to obtain the data indexes, and querying corresponding data in the data cube according to the data indexes. The double-cube modeling technology of the data cube and the data caliber cube is used to realize a gradual data caliber standardization process, so that the difficulty of data fusion is reduced, meanwhile, a more flexible multi-caliber data usage mode is brought, and the differentiated development of various systems can be maximally satisfied.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a novel method for multi-caliber data fusion analysis. Background Technology

[0002] Currently, with the increasingly widespread application of databases and the growing diversity of business types across different fields, the demand for big data processing is also increasing. However, research and practice on existing technologies have revealed that they primarily employ a data platform, using ETL to aggregate data from various business systems, clean and store the data according to a unified enterprise data standard, and then provide it to various application systems. These existing technologies have the following drawbacks: the workload of unifying data standards across different business systems within an enterprise is substantial; they sever the original differences in data standards between systems; and they cannot meet users' customized requirements. Summary of the Invention

[0003] To overcome the problems of existing technologies, a progressive data standardization process is achieved by introducing data caliber cubes and bicubic data cube modeling techniques. This reduces the difficulty of data fusion and brings more flexible ways to use multi-caliber data, which can meet the differentiated development needs of various systems to the greatest extent.

[0004] To achieve the above objectives, this invention provides a novel method for multi-caliber data fusion analysis, comprising the following steps:

[0005] (1) Use data exploration technology to obtain metadata of business data sources and establish basic dimension tables;

[0006] (2) Based on the aforementioned basic dimension table, and in conjunction with the data scope of each system, construct a data scope cube;

[0007] (3) Use data aggregation technology to establish a data source layer;

[0008] (4) Combine the data source layer and the data aperture cube to construct a data cube, and write the data index into the corresponding data aperture cube;

[0009] (5) Based on the business data scope, traverse the data scope cube to obtain the data index, and use it to query the corresponding data in the data cube.

[0010] Furthermore, step 1 includes:

[0011] (1.1) Data exploration is performed using JDBC to obtain all metadata of the corresponding database;

[0012] (1.2) Based on the characteristics of specific business, extract the necessary metadata, construct a star schema through foreign key associations in relational databases, and establish basic dimension tables.

[0013] Furthermore, the construction of the data aperture cube in step 2 includes the following steps:

[0014] (2.1) Establish business data standards based on each business system, with each business data standard being a separate functional module;

[0015] (2.2) Establish standard data definitions for the analysis platform based on the analysis platform;

[0016] (2.3) Based on the business data caliber and the standard data caliber, establish a hierarchical hybrid data caliber and construct a data caliber cube.

[0017] Furthermore, the data aperture cube is stored using a graph database to realize the mapping relationship from the same dimension to multiple data apertures.

[0018] Furthermore, in step 3, the data aggregation technology uses DataX to construct the source layer.

[0019] Furthermore, step 4, constructing the data cube, includes:

[0020] (4.1) Traverse the data aperture cube and aggregate the data corresponding to each data aperture;

[0021] (4.2) Construct the data cube based on the analytic hierarchy process.

[0022] Furthermore, the data cube is stored using HBase.

[0023] The present invention also provides a multi-caliber data fusion and analysis system, comprising:

[0024] Data acquisition module: Used to acquire metadata from business data sources using data exploration techniques and establish basic dimension tables;

[0025] Data caliber cube building module: used to construct a data caliber cube based on the basic dimension table and the data calibers of each system;

[0026] Data aggregation module: used to establish a data source layer using data aggregation technology;

[0027] Data cube building module: used to integrate the data source layer and the data aperture cube to build a data cube, and write the data index into the corresponding data aperture cube;

[0028] Data distribution module: used to traverse the data scope cube based on the business data scope to obtain the data index, and use it to query the corresponding data in the data cube.

[0029] The present invention also provides an apparatus comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the novel method for multi-caliber data fusion analysis described above.

[0030] The present invention also provides a computer-readable storage medium comprising a stored computer program that, when the computer program is executed, controls the device containing the computer-readable storage medium to perform the novel method for multi-caliber data fusion analysis described above.

[0031] The beneficial effects of this invention are:

[0032] This invention provides a novel method for multi-caliber data fusion. By employing a dual-cube modeling technique involving data cubes and data caliber cubes, a progressive data caliber standardization process is achieved, reducing the difficulty of data fusion and enabling more flexible use of multi-caliber data, thus maximizing the ability to meet the differentiated development needs of various systems. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a novel method for multi-caliber data fusion according to an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the aperture cube structure according to an embodiment of the present invention. Detailed Implementation

[0035] The present invention will now be further described with reference to the accompanying drawings and embodiments.

[0036] like Figure 1 As shown in the figure, this invention provides a novel method for multi-caliber data fusion, comprising the following steps:

[0037] S101. Use data exploration technology to obtain metadata from business data sources and establish basic dimension tables;

[0038] Data exploration is a database metadata retrieval technique that uses JDBC to directly obtain all metadata from a given database. Based on the specific business characteristics, necessary metadata is extracted and used to construct a star schema through foreign key relationships in a relational database. A star schema is a multidimensional data relationship, consisting of a fact table and dimension tables. Each dimension table has one dimension as its primary key, and the primary keys of all these dimensions combine to form the primary key of the fact table. The non-primary key attributes of the fact table are called facts, which are generally numerical or other computationally achievable data. In this embodiment, the star schema is stored in MySQL, but it can also be stored in other relational database tables.

[0039] S102. Based on the aforementioned basic dimension table and the data scope of each system, construct a data scope cube;

[0040] The structure of the data aperture cube is as follows Figure 2 As shown, the specific steps for constructing a data aperture cube include:

[0041] 1) Establish data standards for each business system

[0042] In a data caliber definition, business system A and business system B can have different data calibers. The data calibers come from the corresponding business systems, and each data caliber is a separate functional module.

[0043] 2) Establish standard data definitions for the analysis platform.

[0044] The standard data caliber can be the same as or different from the data caliber of a certain business system, and it will be continuously improved as the platform is developed.

[0045] 3) Establish a hierarchical mixed data caliber

[0046] Business data caliber and standard data caliber form the basic layer of the data caliber cube. On top of this, hybrid data caliber establishes a higher level of data caliber, which can include business data caliber, standard data caliber, or a hybrid definition of both. And on top of this, higher-level data cube units can be defined using the same method.

[0047] S103. Employ data aggregation technology to establish a data source layer;

[0048] The data aggregation technology uses the open-source DataX to build the source layer.

[0049] S104. Combine the data source layer and the data aperture cube to construct a data cube, and write the data index into the corresponding data aperture cube;

[0050] After the source layer is completed, the cube algorithm module traverses the data aperture cubes, gathers the data corresponding to each data aperture, constructs the data cubes according to the construction method of the data aperture cubes, and writes the data index into the corresponding data aperture cubes for quick data retrieval during reading.

[0051] S105. Based on the business data scope, traverse the data scope cube to obtain the data index, and use it to query the corresponding data in the data cube.

[0052] Data distribution is achieved through multiple channels. During distribution, the data scope can be selected, the data index is obtained by traversing the data scope cube, and the result data is finally retrieved from the data cube.

[0053] The application system selects the data scope through interface parameters, and the analysis platform obtains the corresponding data index through the data scope cube, thereby obtaining the data corresponding to that index.

[0054] Following the data cube construction process, since the data index is written into the corresponding data caliber in the data caliber cube, it can be directly obtained from the data caliber cube.

[0055] According to another embodiment, a multi-caliber data fusion analysis system is also provided, comprising:

[0056] Data acquisition module: Used to acquire metadata from business data sources using data exploration techniques and establish basic dimension tables;

[0057] Data caliber cube building module: used to construct a data caliber cube based on the basic dimension table and the data calibers of each system;

[0058] Data aggregation module: used to establish a data source layer using data aggregation technology;

[0059] Data cube building module: used to integrate the data source layer and the data aperture cube to build a data cube, and write the data index into the corresponding data aperture cube;

[0060] Data distribution module: used to traverse the data scope cube based on the business data scope to obtain the data index, and use it to query the corresponding data in the data cube.

[0061] According to another embodiment, an apparatus is also provided, characterized in that it includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the novel method for multi-caliber data fusion analysis described above.

[0062] According to another embodiment, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the above-described novel method for multi-caliber data fusion analysis.

[0063] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0064] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these embodiments without departing from the principles and essence of the present invention. Therefore, any technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of the prior art should be within the scope of protection defined by the claims.

Claims

1. A new method of multi-aperture data fusion analysis applied to an electronic device, characterized in that, Includes the following steps: (1) Use data exploration technology to obtain metadata of business data sources and establish basic dimension tables; (2) Based on the aforementioned basic dimension table and the data scope of each system, construct a data scope cube; Constructing a data caliber cube involves the following steps: (2.1) Establish business data standards based on each business system, with each business data standard being a separate functional module; (2.2) Establish standard data definitions for the analysis platform based on the analysis platform; (2.3) Based on the business data caliber and the standard data caliber, establish a hierarchical hybrid data caliber and construct a data caliber cube; (3) Use data aggregation technology to establish a data source layer; (4) Combine the data source layer and the data aperture cube to construct a data cube, and write the data index into the corresponding data aperture cube; Building a data cube includes: (4.1) Traverse the data aperture cube and aggregate the data corresponding to each data aperture; (4.2) Based on the analytic hierarchy process and referring to the construction method of the data caliber cube, the data cube is constructed; (5) Based on the business data scope, traverse the data scope cube to obtain the data index, and use it to query the corresponding data in the data cube.

2. The new method of multi aperture data fusion analysis as claimed in claim 1, wherein: Step 1 includes: (1.1) Data exploration is performed using JDBC to obtain all metadata of the corresponding database; (1.2) Based on the characteristics of specific business, extract the necessary metadata, construct a star schema through foreign key associations in relational databases, and establish basic dimension tables.

3. The new method of multi aperture data fusion analysis as claimed in claim 1, wherein: The data aperture cube is stored using a graph database to realize the mapping relationship from the same dimension to multiple data apertures.

4. The new method of multi aperture data fusion analysis as claimed in claim 1, wherein: In step 3, the data aggregation technology uses DataX to construct the source layer.

5. The new method of multi aperture data fusion analysis as claimed in claim 1, wherein: The data cube is stored using HBase.

6. A multi-caliber data fusion and analysis system, characterized in that, include: Data acquisition module: Used to acquire metadata from business data sources using data exploration techniques and establish basic dimension tables; Data caliber cube building module: used to construct a data caliber cube based on the basic dimension table and the data calibers of each system; Constructing a data caliber cube involves the following steps: (2.1) Establish business data standards based on each business system, with each business data standard being a separate functional module; (2.2) Establish standard data definitions for the analysis platform based on the analysis platform; (2.3) Based on the business data caliber and the standard data caliber, establish a hierarchical hybrid data caliber and construct a data caliber cube; Data aggregation module: used to establish a data source layer using data aggregation technology; Data cube building module: used to integrate the data source layer and the data aperture cube to build a data cube, and write the data index into the corresponding data aperture cube; Building a data cube includes: (4.1) Traverse the data aperture cube and aggregate the data corresponding to each data aperture; (4.2) Based on the analytic hierarchy process and referring to the construction method of the data caliber cube, the data cube is constructed; Data distribution module: used to traverse the data scope cube based on the business data scope to obtain the data index, and use it to query the corresponding data in the data cube.

7. An apparatus comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, characterized in that: When the processor executes the computer program, it implements the novel method for multi-caliber data fusion analysis as described in any one of claims 1-5.

8. A computer readable storage medium comprising a stored computer program, characterized in that: When the computer program is running, it controls the device containing the computer-readable storage medium to perform a novel method for multi-caliber data fusion analysis as described in any one of claims 1-5.

Images