Data intermediate station system

Abstract

The invention discloses a data center system. The system comprises a basic module used for user login, system management, system external management, data configuration and data display; the data source access module is used for data source access, data source state detection and data synchronization; and the data processing module is used for capturing a specified data source, carrying out data processing and outputting a data processing result. According to the application, for different database types, respective dialect implementation classes are provided, and a large amount of flexible configuration is realized in a synchronization process.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a data middle-end system. Background Art

[0002] In the era of big data, the amount of data involved is so huge that it is impossible to capture, manage, process, and organize it into information that helps companies make more proactive business decisions within a reasonable time through traditional tools.

[0003] However, existing big data processing tools have the following problems: 1. High learning cost: For beginners, the configuration files of existing big data processing tools are extremely complex and require a certain amount of learning cost; 2. Unstable performance: When processing large-scale data, the performance of existing big data processing tools may fluctuate and need to be optimized. When extracting other large amounts of data, the memory grows quickly, resulting in frequent GC. 3. No support for real-time processing: Most existing big data processing tools focus on batch processing and are not suitable for scenarios that require high real-time performance. 4. When synchronizing heterogeneous data sources, business coupling occurs. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a data middle-end system which provides simple operation and docking with data sources that are not coupled with the business, realizes the batch migration and synchronization function from the source database to the destination database, supports full and incremental synchronization of data, and solves the problems of high learning cost, unstable performance, lack of support for real-time processing, and business coupling when synchronizing heterogeneous data sources in existing big data processing tools.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a data middle station system, comprising: Basic module: used for user login, system management, system external management, data configuration, data display, and data encryption packaging and storage; Data source access module: used for data source access, data source status detection, and data synchronization; Data processing module: used to capture the specified data source, process the data, and output the data processing results.

[0006] Further: The basic module includes: Login unit: used to log in a user and record the user's subject information as a login log; Operation record storage unit: used to encrypt and store user operation traces; Menu management unit: used to set and manage the menus displayed on the system page; Organizational structure management unit: used to set up and manage the system's organizational structure; Role management unit: used to create and manage roles and set role permissions; User management unit: used to set user account information, and add, modify and delete logged-in users; Data dictionary configuration unit: used to configure the system's general data dictionary; Api open unit: used for authorization management of interface mapping and configuration of external services of the system; And, conduct channel management, manage externally accessible channels; Interface decryption management unit: used to intercept and decrypt interfaces; Data configuration unit: used to configure and associate two tables in the system; Report data export unit: used to export the configured data into reports; Data library unit: used to classify and encode system data, add indexes, and perform data grading, data classification, and data search through fuzzy matching association; Data transaction unit: used to provide a data authentication platform, including data demanders, data providers and third parties; The data provider displays part of the encrypted data online through the data authentication platform and provides it to third parties through standard automated API generation; Data demanders apply for data online through the data authentication platform and obtain APIs through third parties; Data graph unit: used for data configuration unit to display data association through graphics.

[0007] Further: the data source access module includes: Data source access unit: detects the data source and imports the data source that passes the detection into the system; Data source status monitoring unit: used to monitor the push status of the data source and change the link status; Driver management unit: used to manage and use system drivers, and match databases of different data source types according to the drivers; Task execution unit: used to execute tasks according to the data source and generate a list record of the executed tasks.

[0008] Further: the method for the data source access unit to import the data source into the system includes: S101, uploading file data to the system; S102, generating a thread through the system, and using the unique ID of the current thread as the ID of the import task; S103, obtaining file data through the system, creating an object including a task ID, a table name, a table comment, a field list, and a table creation statement, and creating a table according to the object; S104. Insert task data into the created table to complete the data source import.

[0009] Further: the task execution method of the task execution unit includes: S201, creating a task according to the source of the data source; S202, obtaining a task ID, and searching the database for the source and target ends corresponding to the task ID; S203, connecting the source end and the target end corresponding to the task; S204, decoupling through the event bus, and creating a thread pool for executing asynchronous tasks; S205. Publish the task object to the asynchronous time director, and execute the task through the thread pool that executes asynchronous tasks.

[0010] Further: the data processing module includes: Data connection query unit: used to switch the data source connected to the external model, grab the specified data, and perform data connection query with the corresponding database according to the missing discrete values ​​of the data; Central management platform: used to access external models exposed by the network and configure data output direction by role and type through discrete value configuration.

[0011] The beneficial effects of the present invention are: 1. This application is designed to extract file access and data source docking separately as various independent modules, which fundamentally eliminates the situation of business coupling; 2. Considering the scenario of large data volume, if the task is executed immediately, memory leaks and system freezes may occur. Flexible cron expressions are configured on the page, and batch quantity is controlled to make full use of the idle time of the system to perform large data volume synchronization tasks; 3. Support online configuration of various types of data sources on the page, and support online detection of health status; 4. Support page configuration synchronization tasks. On the page, you can select two data sources for configuration. The supported methods are: create a table on the target side and synchronize data, create only a physical table on the target side, and synchronize only the data in the table on the target side. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Schematic diagram of the data middle platform system provided for this application.

[0013] Figure 2 This is a schematic diagram of the basic module.

[0014] Figure 3 The data source access module is a schematic diagram.

[0015] Figure 4 Schematic diagram of the data processing module. DETAILED DESCRIPTION

[0016] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0017] like Figure 1 As shown, in one embodiment of the present invention, a data middle platform system is provided, including: Basic module: used for user login, system management, system external management, data configuration, data display, and data encryption packaging and storage; Data source access module: used for data source access, data source status detection, and data synchronization; Data processing module: used to capture the specified data source, process the data, and output the data processing results.

[0018] like Figure 2 As shown, the basic modules include: Login unit: used to log in a user and record the user's subject information as a login log; Operation record storage unit: used to encrypt and store user operation traces. For important data changes and complex Javabean objects, it is necessary to compare the data before and after the modification, and encrypt and store the comparison results. Menu management unit: used to set and manage the menus displayed on the system page; Organizational structure management unit: used to set up and manage the system's organizational structure; Role management unit: used to create and manage roles and set role permissions; User management unit: used to set user account information, and add, modify and delete logged-in users; Data dictionary configuration unit: used to configure the system's general data dictionary; Api open unit: used for interface mapping and external service configuration of the system; And, conduct channel management, manage externally accessible channels; Interface decryption management unit: used to intercept and decrypt interfaces, specifically: intercept at the controller layer, use ControllerAdvice to intercept RequestBodyAdviceAdapter and ResponseBodyAdvice; Data configuration unit: used to configure and associate two tables in the system; Report data export unit: used to export the configured data into reports; Data library unit: used to classify and encode system data, add indexes, and perform data grading, data classification, and data search through fuzzy matching association; Data transaction unit: used to provide a data authentication platform, including data demanders, data providers and third parties; The data provider displays part of the encrypted data online through the data authentication platform and provides it to third parties through standard automated API generation; Data demanders apply for data online through the data authentication platform and obtain APIs through third parties; Data graph unit: used for data configuration unit to display data association through graphics.

[0019] like Figure 3 As shown, the data source access module includes: Data source access unit: detects the data source and imports the data source that passes the detection into the system; The detection content includes: checking whether the data source is connected normally, and whether there are changes in the fields, names and other dimensions of the data source; Data source status monitoring unit: used to monitor the push status of the data source and change the link status; Driver management unit: used to manage and use system drivers, and match databases of different data source types according to the drivers; Task execution unit: used to execute tasks according to the data source and generate a list record of the executed tasks.

[0020] The data source access module accesses the data through file data, and the method for the data source access unit to import the data source into the system includes: S101, uploading file data to the system; S102, generating a thread through the system, and using the unique ID of the current thread as the ID of the import task; S103, obtaining file data through the system, creating an object including a task ID, a table name, a table comment, a field list, and a table creation statement, and creating a table according to the object; S104. Insert task data into the created table to complete the data source import.

[0021] When importing data, insert the import task in the invoke method, obtain the specific data, use List to receive it, and set the size. When the collected data size meets the designed size, use JdbcTemplate to insert the data. During the insertion process, do not use transactions, use try catch to catch errors, and record the current row number until the storage is completed. In the doAfterAllAnalysed method, it is used to process data that does not meet the size in the invoke method. The logic of inserting into the database is repeated here, and the current time is recorded to update the status of the task table. If there is a record row number with a value, it is a failure, otherwise it is a success. When the user inserts again next time or detects an abnormality in the current task, there is no need to start from the beginning, but to re-import the failed row number. The operator can view the import status on the page.

[0022] In addition, the data center system provided by this application can also import data sources directly instead of through files, which requires the following steps: Data source status detection: When using the direct import function, you need to create the target data source on the page first, fill in the corresponding configuration information, submit the results to the background, and when storing them in the database, the status defaults to normal. Since we cannot control the status of the other party's data source, the product implements the MetadataService interface, dynamically obtains the current data source, and then uses the testQuerySQL method to execute the select 1 statement for data source detection. At this time, the status can be updated based on the results. The front end uses a polling method to query and update the status of the data source regularly. In addition to normal and invalid data source status, there is also a more complex status - push data abnormal status, which mainly refers to when the status of the other party's data source is normal. Since we have configured a scheduled pull of the other party's data, but the synchronization fails, it is in this state. The implementation plan is to start synchronizing data after we successfully connect to the other party's database, and use try catch to capture it. Once the synchronization fails, it will be updated to this status; Driver management: According to the type of data source, the required driver is connected to match the database of different data source types. When the program starts, it will read the driver address in the configuration file, obtain all the drivers in the directory as a file object array, start the loop operation, and determine whether there is a predefined database. If not, add this type to the predefined database driver enumeration, so that it can be added to the driver logic for unified use and management; The task is executed by the task execution unit, and the method includes: S201, creating a task according to the source of the data source; S202, obtaining a task ID, and searching the database for the source and target ends corresponding to the task ID; S203, connecting the source end and the target end corresponding to the task; S204, decoupling through the event bus, and creating a thread pool for executing asynchronous tasks; S205. Publish the task object to the asynchronous time director, and execute the task through the thread pool that executes asynchronous tasks.

[0023] like Figure 4 As shown, the data processing module includes: Data connection query unit: used to switch the data source connected to the external model, grab the specified data, and perform data connection query with the corresponding database according to the missing discrete values ​​of the data; Central management platform: used to access external models exposed by the network and configure data output direction by role and type through discrete value configuration.

[0024] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data middle platform system, characterized in that: include: Basic module: used for user login, system management, system external management, data configuration, data display, and data encryption packaging and storage; Data source access module: used for data source access, data source status detection, and data synchronization; Data processing module: used to capture the specified data source, process the data, and output the data processing results.

2. The data middle platform system according to claim 1 is characterized in that: The basic modules include: Login unit: used to log in a user and record the user's subject information as a login log; Operation record storage unit: used to encrypt and store user operation traces; Menu management unit: used to set and manage the menus displayed on the system page; Organizational structure management unit: used to set up and manage the system's organizational structure; Role management unit: used to create and manage roles and set role permissions; User management unit: used to set user account information, and add, modify and delete logged-in users; Data dictionary configuration unit: used to configure the system's general data dictionary; Api open unit: used for authorization management of interface mapping and configuration of external services of the system; And, conduct channel management, manage externally accessible channels; Interface decryption management unit: used to intercept and decrypt interfaces; Data configuration unit: used to configure and associate two tables in the system; Report data export unit: used to export the configured data into reports; Data library unit: used to classify and encode system data, add indexes, and perform data grading, data classification, and data search through fuzzy matching association; Data transaction unit: used to provide a data authentication platform, including data demanders, data providers and third parties; The data provider displays part of the encrypted data online through the data authentication platform and provides it to third parties through standard automated API generation; Data demanders apply for data online through the data authentication platform and obtain APIs through third parties; Data graph unit: used for data configuration unit to display data association through graphics.

3. The data middle platform system according to claim 1 is characterized in that: The data source access module includes: Data source access unit: detects the data source and imports the data source that passes the detection into the system; Data source status monitoring unit: used to monitor the push status of the data source and change the link status; Driver management unit: used to manage and use system drivers, and match databases of different data source types according to the drivers; Task execution unit: used to execute tasks according to the data source and generate a list record of the executed tasks.

4. The data middle platform system according to claim 3 is characterized in that: The method for the data source access unit to import the data source into the system includes: S101, uploading file data to the system; S102, generating a thread through the system, and using the unique ID of the current thread as the ID of the import task; S103, obtaining file data through the system, creating an object including a task ID, a table name, a table comment, a field list, and a table creation statement, and creating a table according to the object; S104. Insert task data into the created table to complete the data source import.

5. The data middle platform system according to claim 3 is characterized in that: The task execution method of the task execution unit includes: S201, creating a task according to the source of the data source; S202, obtaining a task ID, and searching the database for the source and target ends corresponding to the task ID; S203, connecting the source end and the target end corresponding to the task; S204, decoupling through the event bus, and creating a thread pool for executing asynchronous tasks; S205. Publish the task object to the asynchronous time director, and execute the task through the thread pool that executes asynchronous tasks.

6. The data middle platform system according to claim 1 is characterized in that: The data processing module includes: Data connection query unit: used to switch the data source connected to the external model, grab the specified data, and perform data connection query with the corresponding database according to the missing discrete values ​​of the data; Central management platform: used to access external models exposed by the network and configure data output direction by role and type through discrete value configuration.

Images