A data bus based on data classification

Abstract

The present invention provides a data bus based on data classification, wherein the data bus is divided into a plurality of data spaces according to the type of data structure, each type of data structure corresponds to a data space, each data space corresponds to an interface base class and a plurality of interface subclasses, and each interface subclass performs docking between the data bus and the subclass data structure of the corresponding data structure type by inheriting the interface base class in the corresponding data space. In the data bus, the data space is divided according to the type of data structure, and each data space corresponds to a plurality of subtype data structures. When operating data, the user only needs to distinguish which category the data belongs to, without subdividing it into specific data categories, which is convenient for the user to use, and at the same time, it is convenient for the user to expand new data types without rewriting.

Description

Technical Field

[0001] The invention belongs to the technical field of data interfaces, and in particular relates to a data bus based on data classification. Background Art

[0002] With the rapid development of computer technology and network technology, people's ability to obtain and process spatial data is constantly improving, and the amount of spatial information they have accounts for 80% of the total information. Multi-source heterogeneous spatial data brings inconvenience to data comprehensive utilization and data sharing, and becomes a bottleneck for spatial information sharing. How to effectively, real-time and quickly realize the classification and induction application of multi-source heterogeneous data has become an urgent problem to be solved.

[0003] A data lake is a system that stores data in its natural format. Data lakes are usually used as a single repository for all enterprise data, and can store data in any format and form, including original copies of source system data and transformed data for tasks such as reporting, visualization, analysis, and machine learning. The emergence of data lakes solves the storage layer problem, allowing data warehouses to evolve into data lakes, which do not need to be deployed separately but are directly attached to computing engines. However, the current data lakes do not have a data bus based on data classification to provide data lakes with data interaction management between data lakes and data sources. The data lakes support very limited data types and are not easy to expand. If users want to increase the data types supported by the data lake, they need to rewrite the data lake, and they cannot efficiently reuse the existing data interfaces of the data lake. When adding or reducing data types, it takes a lot of time and energy to rewrite the interfaces of the data lake. Summary of the invention

[0004] The object of the present invention is to provide a data bus based on data classification in view of the above problems.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A data bus based on data classification, the data bus is divided into multiple data spaces according to the type of data structure, each type of data structure corresponds to a data space, each data space corresponds to an interface base class and multiple interface subclasses, and each interface subclass connects the data bus with the subclass data structure of the corresponding data structure type by inheriting the interface base class in the corresponding data space.

[0007] In the above-mentioned data bus based on data classification, the data space includes structured data space, semi-structured data space, unstructured data space and binary data space;

[0008] The data structure type corresponding to the structured data space is the structured data class, the data structure type corresponding to the semi-structured data space is the semi-structured data class, the data structure type corresponding to the unstructured data space is the unstructured data class, and the data structure type corresponding to the binary data space is the binary data class;

[0009] The subclass data structures corresponding to the structured data class are various relational database classes, including MySQL, Oracle, SQLServer, and DB2;

[0010] The subclass data structures corresponding to the semi-structured data class are various non-relational database classes, including Mongo, Neo4j, Hbase, and Redis;

[0011] The subclass data structures corresponding to the unstructured data class include files, text, video, audio, and images;

[0012] The subclass data structures corresponding to the binary data class include binary files and encoded data.

[0013] In the above data classification-based data bus, the structured data space corresponds to a structured data interface base class and multiple relational data interface subclasses, and each relational data interface subclass is connected to the corresponding relational database by inheriting the structured data interface base class;

[0014] The semi-structured data space corresponds to a semi-structured data interface base class and a variety of non-relational data interface subclasses. Each non-relational data interface subclass is connected to the corresponding non-relational data by inheriting the semi-structured data interface base class.

[0015] The unstructured data space corresponds to an unstructured data interface base class and multiple unstructured data interface subclasses. Each unstructured data interface subclass is connected to the corresponding unstructured data by inheriting the unstructured data interface base class.

[0016] The binary data space corresponds to a binary interface base class and multiple binary data interface subclasses. Each binary data interface subclass connects with the corresponding binary data by inheriting the binary interface base class.

[0017] In the above-mentioned data bus based on data classification, the data bus serves as an access layer of the data lake system, and is used to connect the computing engine and the data source to achieve intercommunication between the data lake system and the computing engine and the data source;

[0018] The data bus has a data type configuration file for matching the data space.

[0019] In the above-mentioned data bus based on data classification, the structured data interface base class includes a general field type mapping module, and the structured data interface subclass includes a specific field type mapping module. The general field type mapping module is used to map and convert the common fields of various relational databases, and the specific field type mapping module is used to map and convert the specific fields of the corresponding relational databases.

[0020] In the above data classification-based data bus, the structured data interface base class includes a relational database configuration file for storing basic information of various relational databases; the structured data interface subclass includes basic information of the corresponding relational database in the relational database configuration file.

[0021] In the above-mentioned data bus based on data classification, the semi-structured data interface base class includes a general data format conversion module, and the semi-structured data interface subclass includes a specific data format conversion module, wherein the general data format conversion module is used to convert the common fields of various non-relational databases, and the specific data format conversion module is used to convert the specific fields of the corresponding non-relational databases;

[0022] The semi-structured data interface base class includes a non-relational database configuration file for storing basic information of various non-relational databases; the non-structured data interface subclass includes basic information of the corresponding non-relational database in the non-relational database configuration file.

[0023] In the above-mentioned data bus based on data classification, the data bus has a driver package directory for storing various driver packages, and the driver package directory records the correspondence between the driver package and the driver name. The basic information of the non-relational database and the basic information of the relational database both have the driver names of the drivers supported by the database. The data bus matches the corresponding basic information according to the currently started relational database / non-relational database and loads the corresponding driver package from the driver package directory to perform bidirectional reading and writing with the corresponding relational database / non-relational database.

[0024] In the above-mentioned data bus based on data classification, the unstructured data interface base class includes a general data conversion module, and the unstructured data interface subclass includes a specific data conversion module. The general data conversion module is used to convert general fields of various unstructured data, and the specific data conversion module is used to convert specific fields of corresponding unstructured data.

[0025] The binary interface base class includes a general binary data conversion module, and the binary interface subclass includes a specific binary data conversion module. The general binary data conversion module is used to convert general fields of various binary data, and the specific binary data conversion module is used to convert specific fields of corresponding binary data.

[0026] The advantages of the present invention are:

[0027] In the data bus, the data space is divided according to the type of data structure. Each data space corresponds to multiple sub-type data structures. When users operate data, they only need to distinguish which category the data belongs to, without having to subdivide it into specific data categories, which is convenient for users to use;

[0028] This data bus comes with four commonly used data structure types, and each data type has commonly used corresponding data types, such as binary files and encoded data commonly used in binary data types, which have covered most data types. Therefore, it has the characteristics of universality and can meet the needs of almost all customers, whether it is files, images, audio, structured or unstructured big data or small data.

[0029] Data is classified by data space, and a clear division of labor is achieved in the data bus. On the basis of internally clear functions, the time for adding additional functions is shortened, and rewriting the interface system or complex transplantation is avoided. The efficiency of modifying the data bus interface is improved by adding or reducing interfaces according to categories. If users want to increase the data types supported by the data lake, they do not need to rewrite the data lake, but can directly operate in the corresponding data space according to the data type;

[0030] In addition, each type of data has an interface base class for that type of data structure. The data interfaces of various data types under each type of data structure inherit from the interface base class to implement the interface and methods of the subclass data structure. For specific data structure extensions under each type of data structure, they can inherit the base class to improve the efficiency of the extended interface. Therefore, even if a small number of data types are not currently supported by this system, users can easily and quickly perform custom extensions, thereby indirectly achieving full coverage of data categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the spatial layout of a data bus based on data classification according to the present invention;

[0032] Figure 2 This is a flow chart of multi-source heterogeneous implementation of a data bus based on data classification according to the present invention;

[0033] Figure 3 It is a schematic diagram of the configuration of the structured data space in the data bus based on data classification of the present invention;

[0034] Figure 4 It is a schematic diagram of the configuration of the semi-structured data space in the data bus based on data classification of the present invention;

[0035] Figure 5It is a schematic diagram of the configuration of the unstructured data space in the data bus based on data classification of the present invention;

[0036] Figure 6 It is a schematic diagram of the configuration of binary data space in a data bus based on data classification of the present invention;

[0037] Figure 7 The present invention is a flowchart of the connection of data bus structured data types based on data classification;

[0038] Figure 8 It is a schematic diagram of a relational database configuration file in a data bus structured data type based on data classification of the present invention. DETAILED DESCRIPTION

[0039] The scheme is further described below in conjunction with the accompanying drawings:

[0040] like Figure 1 As shown, this interface system can be installed in the access layer of the data lake system, such as the access layer of the DeltaLake data lake system, to connect the DeltaLake data lake system with computing engines such as the Spark engine and the Hadoop engine, as well as with various data sources. While separating data storage from computing, the computing layer application and the storage layer application can be interconnected. The two parts of the application can be independently expanded, so that the DeltaLake data lake system does not only support the Spark engine, and provides a bus foundation for the expansion of the computing engine of the data lake system.

[0041] Specifically, the data bus based on data classification of the present embodiment is divided into multiple data spaces according to the type of data structure, each type of data structure corresponds to a data space, each data space corresponds to an interface base class and multiple interface subclasses, and each interface subclass provides the connection between the data bus and the subclass data structure of the corresponding data structure type by inheriting the interface base class in the corresponding data space. In some preferred embodiments, the data bus can also include a data base class, and each interface base class inherits the data base class. At this time, the data base class precipitates common functions for all data types as the top-level design of the data bus, and each interface base class precipitates common functions for the corresponding data class as the sub-top-level design of the data bus, and the interface subclass is used as a targeted design to connect to a specific data structure. When it is necessary to expand the function of a specific data type, it is convenient and quick to edit the specific interface subclass directly in the corresponding data space.

[0042] Specifically, the present embodiment divides data into four categories: structured data, semi-structured data, unstructured data and binary data, and each category of data corresponds to a plurality of specific data types. Therefore, the data space divided by the data bus of the present embodiment includes structured data space, semi-structured data space, unstructured data space and binary data space.

[0043] The data structure type corresponding to the structured data space is the structured data class, the data structure type corresponding to the semi-structured data space is the semi-structured data class, the data structure type corresponding to the unstructured data space is the unstructured data class, and the data structure type corresponding to the binary data space is the binary data class.

[0044] The subclass data structures corresponding to the structured data class are various relational database classes, including MySQL, Oracle, SQLServer, DB2, etc.; the subclass data structures corresponding to the semi-structured data class are various non-relational database classes, including Mongo, Neo4j, Hbase and Redis, etc., supporting CSV, XML, JSON, HTML and other formats; the subclass data structures corresponding to the unstructured data class include files, texts, videos, audios, images, etc.; the subclass data structures corresponding to the binary data class include binary files, encoded data, etc.

[0045] Furthermore, the data bus has a data type configuration file for matching the data space, each type of data structure has its own identification name, and the data type configuration file stores the package path containing the data class identification name of each data structure. Figure 2 As shown, when it is put into use, the user selects the data type, and the interface system matches the data class in the data type configuration file according to the identification name of the data type selected by the user. For example, if the user selects structured data, it matches the structured data class, and then further matches the data space based on the matched data class, and performs bidirectional conversion between the corresponding data format and the format supported by the data lake system (usually Parquet format) in the data space. Continuing with structured data as an example, it matches the structured data space, and then performs bidirectional reading and writing of the corresponding data through the structured data space.

[0046] Furthermore, if Figure 3-6 As shown, the structured data space corresponds to a structured data interface base class and a variety of relational data interface subclasses. Each relational data interface subclass is connected to the corresponding relational database by inheriting the structured data interface base class.

[0047] The semi-structured data space corresponds to a semi-structured data interface base class and a variety of non-relational data interface subclasses. Each non-relational data interface subclass is connected to the corresponding non-relational data by inheriting the semi-structured data interface base class.

[0048] The unstructured data space corresponds to an unstructured data interface base class and multiple unstructured data interface subclasses. Each unstructured data interface subclass is connected to the corresponding unstructured data by inheriting the unstructured data interface base class.

[0049] The binary data space corresponds to a binary interface base class and multiple binary data interface subclasses. Each binary data interface subclass connects with the corresponding binary data by inheriting the binary interface base class.

[0050] Specifically, the structured data interface base class includes a general field type mapping module, and the structured data interface subclass includes a specific field type mapping module. The general field type mapping module is used to map and convert the common fields of various relational databases, and the specific field type mapping module is used to map and convert the specific fields of the corresponding relational databases.

[0051] The structured data interface base class includes a relational database configuration file for storing basic information of various relational databases; the structured data interface subclass includes the basic information of the corresponding relational database in the relational database configuration file. The basic information includes the name of the corresponding relational database and the name of the corresponding driver package.

[0052] Similarly, the semi-structured data interface base class includes a general data format conversion module, and the semi-structured data interface subclass includes a specific data format conversion module. The general data format conversion module is used to convert the common fields of various non-relational databases, and the specific data format conversion module is used to convert the specific fields of the corresponding non-relational databases.

[0053] The semi-structured data interface base class includes a non-relational database configuration file for storing basic information of various non-relational databases; the non-structured data interface subclass includes basic information of the corresponding non-relational database in the non-relational database configuration file.

[0054] The data bus has a driver package directory for storing various driver packages, and the driver package directory records the correspondence between the driver package and the driver name. The basic information of the non-relational database and the basic information of the relational database both have the driver name of the driver supported by the database. The data bus matches the corresponding basic information according to the currently started relational database / non-relational database and loads the corresponding driver package from the driver package directory to perform bidirectional reading and writing with the corresponding relational database / non-relational database. When users expand data types, whether it is sub-category data under an existing large category or creating large category data, there is no need to rewrite the driver for the existing driver. The driver name can be directly made to correspond to the existing driver package to achieve driver reuse.

[0055] Take structured data as an example, Figure 7 and Figure 8 As shown, when it is put into use, when the data class of the relational database is needed, the data class specified by the user is matched to the structured data space, and then the basic information of the corresponding relational database is obtained from the relational database configuration file of the structured data space, and the corresponding driver package name is obtained from the basic information, and the driver package corresponding to the driver name is loaded from the driver package directory to connect the corresponding relational database with the interface system through the driver package to perform bidirectional reading and writing. Relationaldb is the identification name of the relational database, and mysql and oracle are the names of two relational database instances respectively.

[0056] Further, the unstructured data interface base class includes a general data conversion module, and the unstructured data interface subclass includes a specific data conversion module, the general data conversion module is used to convert general fields of various unstructured data, and the specific data conversion module is used to convert specific fields of corresponding unstructured data;

[0057] The binary interface base class includes a general binary data conversion module, and the binary interface subclass includes a specific binary data conversion module. The general binary data conversion module is used to convert general fields of various binary data, and the specific binary data conversion module is used to convert specific fields of corresponding binary data.

[0058] Taking the binary data class as an example, when users extend a new type of binary data, they can directly inherit the binary interface base class, and then write a unique interface subclass for the new type, which can greatly reduce the workload of developers and improve expansion efficiency.

[0059] The above data conversions are to convert the corresponding data structure into the Parquet structure supported by the data lake for use by the data lake, or to convert the Parquet structured data captured from the data lake into the corresponding data structure for use by the corresponding computing engine, thereby realizing bidirectional reading and writing between the data lake and various databases / data sources.

[0060] This solution provides a dedicated data bus based on data classification, supports most of the commonly used data types, meets the various needs of various users, and enables the data lake system to support various data types. In addition, this solution summarizes and classifies multi-source heterogeneous data in the interface system, and adopts the idea of ​​base class and subclass inheritance of base class. While clarifying the division of labor in the interface, it provides a clear entry for creating new connections in the future, returns open source, and data type expansion is very convenient. Users can expand data types at any time as needed.

[0061] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0062] Although the terms such as data bus, data space, interface base class, interface subclass, etc. are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A data bus based on data classification, characterized in that: The data bus is divided into multiple data spaces according to the type of data structure. Each type of data structure corresponds to a data space. Each data space corresponds to an interface base class and multiple interface subclasses. Each interface subclass connects the data bus with the subclass data structure of the corresponding data structure type by inheriting the interface base class in the corresponding data space. The data bus serves as the access layer of the data lake system and is used to connect the computing engine and the data source to achieve intercommunication between the data lake system and the computing engine and the data source; The data space includes structured data space, semi-structured data space, unstructured data space and binary data space; The data structure type corresponding to the structured data space is the structured data class, the data structure type corresponding to the semi-structured data space is the semi-structured data class, the data structure type corresponding to the unstructured data space is the unstructured data class, and the data structure type corresponding to the binary data space is the binary data class; The structured data space corresponds to a structured data interface base class and multiple relational data interface subclasses. Each relational data interface subclass is connected to the corresponding relational database by inheriting the structured data interface base class. The semi-structured data space corresponds to a semi-structured data interface base class and a variety of non-relational data interface subclasses. Each non-relational data interface subclass is connected to the corresponding non-relational data by inheriting the semi-structured data interface base class. The unstructured data space corresponds to an unstructured data interface base class and multiple unstructured data interface subclasses. Each unstructured data interface subclass is connected to the corresponding unstructured data by inheriting the unstructured data interface base class. The binary data space corresponds to a binary interface base class and multiple binary data interface subclasses. Each binary data interface subclass connects with the corresponding binary data by inheriting the binary interface base class.

2. The data bus based on data classification according to claim 1, characterized in that: The subclass data structures corresponding to the structured data class are various relational database classes, including MySQL, Oracle, SQLServer, and DB2; The subclass data structures corresponding to the semi-structured data class are various non-relational database classes, including Mongo, Neo4j, Hbase, and Redis; The subclass data structures corresponding to the unstructured data class include text, video, audio, and image; The subclass data structures corresponding to the binary data class include binary files and encoded data.

3. The data bus based on data classification according to claim 2, characterized in that: The data bus has a data type configuration file for matching the data space.

4. The data bus based on data classification according to claim 3, characterized in that: The structured data interface base class includes a general field type mapping module, and the structured data interface subclass includes a specific field type mapping module. The general field type mapping module is used to map and convert the common fields of various relational databases, and the specific field type mapping module is used to map and convert the specific fields of the corresponding relational database.

5. The data bus based on data classification according to claim 4, characterized in that: The structured data interface base class includes a relational database configuration file for storing basic information of various relational databases; the structured data interface subclass includes basic information of the corresponding relational database in the relational database configuration file.

6. The data bus based on data classification according to claim 5, characterized in that: The semi-structured data interface base class includes a general data format conversion module, and the semi-structured data interface subclass includes a specific data format conversion module. The general data format conversion module is used to convert the common fields of various non-relational databases, and the specific data format conversion module is used to convert the specific fields of the corresponding non-relational databases. The semi-structured data interface base class includes a non-relational database configuration file for storing basic information of various non-relational databases; the non-structured data interface subclass includes basic information of the corresponding non-relational database in the non-relational database configuration file.

7. The data bus based on data classification according to claim 6, characterized in that: The data bus has a driver package directory for storing various driver packages, and the driver package directory records the correspondence between the driver package and the driver name. The basic information of the non-relational database and the basic information of the relational database both have the driver names of the drivers supported by the database. The data bus matches the corresponding basic information according to the currently started relational database / non-relational database and loads the corresponding driver package from the driver package directory to perform bidirectional reading and writing with the corresponding relational database / non-relational database.

8. The data bus based on data classification according to claim 7, characterized in that: The unstructured data interface base class includes a general data conversion module, and the unstructured data interface subclass includes a specific data conversion module. The general data conversion module is used to convert the general fields of various unstructured data, and the specific data conversion module is used to convert the specific fields of the corresponding unstructured data. The binary interface base class includes a general binary data conversion module, and the binary interface subclass includes a specific binary data conversion module. The general binary data conversion module is used to convert general fields of various binary data, and the specific binary data conversion module is used to convert specific fields of corresponding binary data.

Images