Big data method and system for multi-data source access based on Java

Abstract

The invention relates to the technical field of big data, in particular to a Java-based multi-data-source access big data method and system, comprising the following steps: constructing a system architecture comprising a data source management module, a connection pool management module, a data query module and a log and monitoring module; the Java-based multi-data source access big data method and system provided by the invention have the beneficial effects that unified access, management and operation of various types of data sources are realized through a standardized interface and a flexible framework design. According to the technology, an adapter mode and a factory mode are adopted, different data source drives can be dynamically loaded and managed, meanwhile, a unified data query interface is provided, developers do not need to care about implementation details of underlying data sources, the development work is greatly simplified, and the flexibility and expandability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a Java-based multi-data source access big data method and system. Background Art

[0002] With the advent of the big data era, the types and number of data sources have increased dramatically. How to efficiently access and process multiple heterogeneous data sources has become an important issue in big data technology.

[0003] Existing big data platforms can usually only process a specific type of data source, and it is difficult to achieve unified access and processing of multiple data sources. At the same time, the use of different programming languages ​​to integrate different data sources also increases the complexity of development and maintenance.

[0004] Therefore, it is particularly important to propose a multi-data source access technology based on Java language, which can simplify the data source access process and improve the efficiency of data processing. Summary of the invention

[0005] The purpose of the present invention is to provide a Java-based big data method and system for accessing multiple data sources to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a big data method for accessing multiple data sources based on Java, the method comprising the following steps:

[0007] Build the system architecture, including data source management module, connection pool management module, data query module, and log and monitoring module;

[0008] The data source management module is responsible for loading and managing different data sources;

[0009] The connection pool management module manages the connection pool of the data source and provides an efficient connection management mechanism;

[0010] The data query module provides a unified data query interface;

[0011] The logging and monitoring module records data source access logs and provides system performance monitoring.

[0012] Preferably, the method further includes the following steps of accessing multiple data sources:

[0013] Users define the data source they need to access through configuration files or program interfaces;

[0014] The data source management module loads the corresponding data source driver according to the user configuration and initializes the connection pool;

[0015] The data query module receives the user's query request and initiates queries to different data sources through a unified query interface;

[0016] The system formats the query results uniformly and returns them to the user.

[0017] Preferably, the following steps are also included to implement the adapter mode:

[0018] Define a unified interface for interacting with upper-layer applications;

[0019] An adapter class is implemented for each type of data source. The adapter class implements the unified interface and converts specific operations of the data source into operations of the unified interface.

[0020] Preferably, application of the adapter pattern enables upper-layer applications to transparently operate different data sources.

[0021] Preferably, the following steps are also included to implement the factory mode:

[0022] Define a factory class to dynamically create data source objects based on user configuration;

[0023] The factory class selects the corresponding data source type according to the user configuration information and creates a data source object;

[0024] Return the created data source object to the data source management module for management and use.

[0025] A Java-based big data system for accessing multiple data sources is applied to a Java-based big data method for accessing multiple data sources. The system includes:

[0026] Data source management module, used to load and manage various types of data sources;

[0027] A connection pool management module, connected to the data source management module, for providing an efficient connection management mechanism for the data source loaded by the data source management module;

[0028] The data query module provides a unified data query interface, allowing users to initiate query requests to different data sources through the interface;

[0029] The log and monitoring module is used to record data source access logs and provide system performance monitoring functions.

[0030] Preferably, the system further includes a multi-data source access process, which includes:

[0031] Users define the data source they need to access through configuration files or program interfaces;

[0032] The data source management module loads the corresponding data source driver according to the user definition and initializes the connection pool;

[0033] The data query module receives the query request initiated by the user through the unified data query interface and initiates a query to the corresponding data source;

[0034] The system formats the query results uniformly and returns them to the user.

[0035] Preferably, the system adopts an adapter mode, which specifically includes:

[0036] Define a unified interface for interacting with upper-layer applications;

[0037] Implement an adapter class for each type of data source. This class implements a unified interface and converts specific operations of the data source into operations of the unified interface, thereby achieving unified access to different types of data sources.

[0038] Preferably, application of the adapter pattern enables upper-layer applications to transparently operate different types of data sources without having to worry about differences in underlying data sources.

[0039] Preferably, application of the adapter pattern enables upper-layer applications to transparently operate different types of data sources without having to worry about differences in underlying data sources.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The Java-based multi-data source access big data method and system proposed in the present invention realizes unified access, management and operation of various types of data sources through standardized interfaces and flexible framework design. The technology adopts the adapter mode and factory mode, which can dynamically load and manage different data source drivers, and provide a unified data query interface, so that developers do not need to care about the implementation details of the underlying data source, which greatly simplifies the development work and improves the flexibility and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1, the present invention provides a technical solution: a big data method for accessing multiple data sources based on Java, the method comprising the following steps:

[0045] Build the system architecture, including data source management module, connection pool management module, data query module and log and monitoring module; the data source management module is responsible for loading and managing different data sources; the connection pool management module manages the connection pool of the data source and provides an efficient connection management mechanism; the data query module provides a unified data query interface; the log and monitoring module records the data source access log and provides system performance monitoring.

[0046] It also includes the following steps for accessing multiple data sources: the user defines the data source to be accessed through a configuration file or a program interface; the data source management module loads the corresponding data source driver according to the user configuration and initializes the connection pool; the data query module receives the user's query request and initiates a query to different data sources through a unified query interface; the system uniformly formats the query results and returns them to the user.

[0047] The following steps are also included to implement the adapter pattern: define a unified interface for interacting with upper-level applications; implement an adapter class for each type of data source, which implements the unified interface and converts specific operations of the data source into operations of the unified interface; the application of the adapter pattern enables upper-level applications to transparently operate different data sources.

[0048] The following steps are also included to implement the factory mode: define a factory class for dynamically creating a data source object based on user configuration; the factory class selects the corresponding data source type and creates a data source object based on the user configuration information; and returns the created data source object to the data source management module for management and use.

[0049] Embodiment 2, refer to the attached Figure 1 As shown, based on the first embodiment, a big data system for accessing multiple data sources based on Java is proposed, which is applied to a big data method for accessing multiple data sources based on Java. The system includes:

[0050] 1. System Architecture

[0051] 1. The system architecture of the present invention includes the following modules:

[0052] i. Data source management module: responsible for loading and managing different data sources.

[0053] ii. Connection pool management module: used to manage the connection pool of the data source and provide an efficient connection management mechanism.

[0054] iii. Data query module: provides a unified data query interface to simplify user query operations.

[0055] iv. Logging and monitoring module: used to record data source access logs and provide system performance monitoring functions.

[0056] 2. Data Source Access Process

[0057] 1. The multi-data source access process of the present invention is as follows:

[0058] i. First, the user defines the data source to be accessed through a configuration file or program interface.

[0059] ii. Then, the data source management module loads the corresponding data source driver and initializes the connection pool according to the user's configuration.

[0060] iii. The data query module provides a unified query interface through which users can initiate query requests to different data sources.

[0061] iv. Finally, the system will uniformly format the query results and return them to the user.

[0062] 3. Application of Adapter Mode and Factory Mode

[0063] In order to achieve unified access to different types of data sources, the present invention adopts the adapter mode and the factory mode. Through the adapter mode, different types of data sources are adapted to a unified interface, so that upper-layer applications can transparently operate different data sources. Through the factory mode, the system can dynamically create different types of data source objects according to the user's configuration, simplifying the coupling of the code.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Java-based big data method for accessing multiple data sources, characterized by: The method comprises the following steps: Build the system architecture, including data source management module, connection pool management module, data query module, and log and monitoring module; The data source management module is responsible for loading and managing different data sources; The connection pool management module manages the connection pool of the data source and provides an efficient connection management mechanism; The data query module provides a unified data query interface; The logging and monitoring module records data source access logs and provides system performance monitoring.

2. According to claim 1, a Java-based big data method for accessing multiple data sources is characterized in that: It also includes the following steps for accessing multiple data sources: Users define the data source they need to access through configuration files or program interfaces; The data source management module loads the corresponding data source driver according to the user configuration and initializes the connection pool; The data query module receives the user's query request and initiates queries to different data sources through a unified query interface; The system formats the query results uniformly and returns them to the user.

3. The big data method for accessing multiple data sources based on Java according to claim 2, characterized in that: The following steps are also included to implement the adapter pattern: Define a unified interface for interacting with upper-layer applications; An adapter class is implemented for each type of data source. The adapter class implements the unified interface and converts specific operations of the data source into operations of the unified interface.

4. The big data method for accessing multiple data sources based on Java according to claim 3, characterized in that: The application of the adapter pattern enables upper-layer applications to transparently operate different data sources.

5. The big data method for accessing multiple data sources based on Java according to claim 1, characterized in that: The following steps are also included to implement the factory pattern: Define a factory class to dynamically create data source objects based on user configuration; The factory class selects the corresponding data source type according to the user configuration information and creates a data source object; Return the created data source object to the data source management module for management and use.

6. A Java-based big data system with multiple data source access, applied to a Java-based big data method with multiple data source access as claimed in any one of claims 1 to 6, characterized in that: The system comprises: Data source management module, used to load and manage various types of data sources; A connection pool management module, connected to the data source management module, for providing an efficient connection management mechanism for the data source loaded by the data source management module; The data query module provides a unified data query interface, allowing users to initiate query requests to different data sources through the interface; The log and monitoring module is used to record data source access logs and provide system performance monitoring functions.

7. The Java-based multi-data source access big data system according to claim 6, characterized in that: The system also includes a multi-data source access process, which includes: Users define the data source they need to access through configuration files or program interfaces; The data source management module loads the corresponding data source driver according to the user definition and initializes the connection pool; The data query module receives the query request initiated by the user through the unified data query interface and initiates a query to the corresponding data source; The system formats the query results uniformly and returns them to the user.

8. The Java-based multi-data source access big data system according to claim 7, characterized in that: The system adopts the adapter mode, which includes: Define a unified interface for interacting with upper-layer applications; Implement an adapter class for each type of data source. This class implements a unified interface and converts specific operations of the data source into operations of the unified interface to achieve unified access to different types of data sources.

9. A Java-based multi-data source access big data system according to claim 8, characterized in that: The application of the adapter pattern enables upper-layer applications to transparently operate different types of data sources without having to worry about the differences in the underlying data sources.

10. The Java-based multi-data source access big data system according to claim 6, characterized in that: The system uses a factory mode to dynamically create different types of data source objects based on user configuration, including: Define a factory class to create a data source object; The factory class selects the corresponding data source type according to the user configuration information and creates a data source object of this type; The factory class returns the created data source object to the data source management module for management and use, thereby simplifying the coupling of the code.

Images