A general data extraction system

Through the microservice architecture and REST interface, cross-enterprise data extraction is achieved efficient, secure and easy to operate, solving the problem of data interoperability between enterprises, reducing development and operation and maintenance costs, and simplifying the data integration process.

CN115836284BActive Publication Date: 2024-11-26BOE TECHNOLOGY GROUP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180001459.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-07
Publication Date
2024-11-26
Estimated Expiration
2041-06-07

AI Technical Summary

Technical Problem

In the existing technology, business data between enterprises is not interoperable, which leads to difficulty in data extraction and integration. The cost of using mature tools is high. Open source tools need to intrude into the database, and the development interface is cumbersome, causing data pressure, making it difficult to achieve efficient and secure cross-enterprise. Data extraction.

Method used

A general data extraction system based on microservice architecture is adopted to extract data through REST interface and configuration services and execution services, realize data mapping and synchronization, and use graphical interfaces to configure data sources and target models to reduce development costs and data pressure.

Benefits of technology

It realizes data extraction without intrusion into the data producer database, reduces development and operation and maintenance costs, improves the security and user ease of operation, and simplifies the data integration process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115836284B_ABST
    Figure CN115836284B_ABST
Patent Text Reader

Abstract

A universal data extraction system is applied to data consumers. The system is based on a microservice architecture and includes a service registration center. The system includes: at least one configuration service, which is configured to perform relevant configuration on an extraction task of extracting data from a corresponding data producer; at least one execution service, which is configured to execute the extraction task and map the extracted data to a target location; wherein both the configuration service and the execution service are registered with the microservice architecture, and both use a REST interface to transmit data when communicating with the data producer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data extraction, and in particular to a general data extraction system. Background Art

[0002] With the development of enterprise informatization, information systems are increasingly used in daily work, and different business departments, production lines, enterprises, etc. will form their own business systems.

[0003] As the core asset of the business system, the database has become the core asset of the enterprise and is usually not open to third parties. Even for business departments of the same enterprise, due to historical reasons, such as different programming languages ​​and different confidentiality requirements, the business data between each other cannot be interoperable. However, with the continuous deepening of information construction, a large project usually needs to integrate business data provided by different enterprises or different business departments (data producers), but the processes of these businesses are not interoperable and the data is not shared. This brings great difficulties to users (data consumers) who develop large projects in data analysis and utilization, report development, analysis and mining.

[0004] In the prior art, data warehouse technology is usually used to extract data from databases of different business systems, and then after processing and integration, it is stored in the local database. However, the disadvantage of this solution is that the use of mature data extraction tools requires a high price, which is difficult for general enterprises to afford, and the use of open source tools requires invading the enterprise's database, which is usually not allowed by the enterprise. In addition, the use of the above solution also requires data consumers to develop data synchronization interfaces for different business systems, and the data producers call the interfaces to implement data extraction and data reporting, which results in the need for data consumers to frequently develop interfaces, and at the same time, centralized reporting by data producers will cause greater data pressure on data consumers. Summary of the invention

[0005] The present disclosure provides a general data extraction system to solve the above-mentioned technical problems existing in the prior art.

[0006] In the first aspect, in order to solve the above technical problems, the embodiment of the present disclosure provides a general data extraction system, which is applied to data consumers. The system is based on a microservice architecture and includes a service registration center. The technical solution of the method is as follows:

[0007] At least one configuration service, one of the configuration services being configured to perform relevant configuration on an extraction task of extracting data from a corresponding data producer;

[0008] At least one execution service, one of the execution services is configured to execute the extraction task and map the extracted data to a target location; wherein both the configuration service and the execution service are registered with the microservice architecture, and both use the REST interface to transmit data when communicating with the data producer.

[0009] In a possible implementation manner, the REST interface includes:

[0010] An authentication interface based on the REST and a data extraction interface based on the REST; wherein the authentication interface is configured to obtain authorization information for accessing the data producer, and the data extraction interface is configured to extract data from the data producer using the authorization information.

[0011] In a possible implementation manner, the configuration service is further configured as:

[0012] Sending authorization request information to the data producer through the authentication interface; wherein the authorization request carries the user name and user password required for registration at the data consumer;

[0013] Receive the authorization information returned by the data producer based on the authorization request information; wherein the authorization information is generated based on the user name and the user password.

[0014] In a possible implementation manner, the configuration service is further configured as:

[0015] Obtaining fields contained in the source data table where the required data is located from the data producer;

[0016] The fields contained in the source data table are mapped to the fields of the target data table of the data consumer, and a corresponding extraction task is established.

[0017] In a possible implementation manner, the configuration service is further configured as:

[0018] Sending a data extraction request to the data producer through the data extraction interface; wherein the data extraction request carries the authorization information and relevant information of the data source data table;

[0019] Receiving a piece of sample data in the source data table returned based on the data extraction request;

[0020] The fields and corresponding field types contained in the source data table are obtained based on the sample data.

[0021] In a possible implementation manner, the configuration task is further configured as:

[0022] The data source of the extraction task is defined based on the data producer and the source data table, and a target data model corresponding to the target data table is defined; wherein the target data model includes a method for writing the data extracted from the data producer into the target data table, and a data format adopted by a field type of a field in the target data table;

[0023] Establishing a mapping relationship between fields in the source data table and fields in the target data table and a data format conversion method for the mapping fields;

[0024] The execution cycle and data synchronization method of the extraction task are configured, and the extraction task based on the target data model, the mapping relationship and the corresponding data format conversion method is established.

[0025] In a possible implementation manner, the execution service is configured as follows:

[0026] Generate a corresponding data extraction request according to the extraction task;

[0027] Sending the data extraction request to the data producer through the data extraction interface, and receiving the corresponding data in the source data table returned;

[0028] The returned data is converted into data in the target data table for storage through the extraction task.

[0029] In a possible implementation manner, the REST interface includes:

[0030] User list, used to store the website address of the data producer;

[0031] An interface, used to represent the operations performed on the data producer;

[0032] Request parameters, used to represent the parameters requested from the data producer;

[0033] The return value is used to represent the data returned by the data producer based on the request parameters.

[0034] In a possible implementation manner, the data format used by the request parameter and the return value is JSON format.

[0035] In a possible implementation manner, when the REST interface is the authentication interface, the request parameter carries the authorization request, and the return value carries the authorization information.

[0036] In a possible implementation, when the REST interface is the data extraction interface, the request parameter carries the authorization information and the data extraction request, and the return value carries the data returned by the data producer based on the data extraction request.

[0037] In a possible implementation manner, the definition of the data source, the configuration of the target data model, and the establishment of the mapping relationship are completed using a graphical interface based on user operations.

[0038] In a possible implementation manner, both the configuration service and the execution service use container technology, and each of the configuration service and each of the execution services runs in a corresponding container. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the structure of a general data extraction system provided by an embodiment of the present disclosure;

[0040] Figure 2 A relationship diagram between data consumers and data producers provided in an embodiment of the present disclosure;

[0041] Figure 3 A schematic diagram of the system structure used by the operating platform of the data consumer provided in the embodiment of the present disclosure;

[0042] Figure 4 Configuration diagram of a graphical interface for defining data sources provided in an embodiment of the present disclosure Figure 1 ;

[0043] Figure 5 Schematic diagram of the graphical interface configuration of the data source provided in the embodiment of the present disclosure Figure 2 ;

[0044] Figure 6 A schematic diagram of parameter configuration of a graphical interface of a target data model provided in an embodiment of the present disclosure;

[0045] Figure 7 A schematic diagram of parameter configuration of a graphical interface of a target data table provided in an embodiment of the present disclosure;

[0046] Figure 8 A graphical interface for configuring a mapping relationship between a source data table and a target data table provided in an embodiment of the present disclosure;

[0047] Fig. 9 A graphical interface for configuring extraction tasks provided in an embodiment of the present disclosure;

[0048] Fig.10 A schematic diagram of the relationship between the configuration service and the execution service provided in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] The disclosed embodiments provide a general data extraction system to solve the above-mentioned technical problems existing in the prior art.

[0050] In order to better understand the above technical scheme, the technical scheme of the present disclosure is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical scheme of the present disclosure, rather than limitations on the technical scheme of the present disclosure. In the absence of conflict, the embodiments of the present disclosure and the technical features in the embodiments can be combined with each other.

[0051] Please refer to Figure 1 The embodiment of the present disclosure provides a general data extraction system, which is applied to data consumers. The system is based on a microservice architecture and includes a service registration center 1. The system includes:

[0052] At least one configuration service 2, one configuration service 2 is configured to perform relevant configuration for an extraction task of extracting data from a corresponding data producer; wherein the relevant configuration includes a REST-based data structure;

[0053] At least one execution service 3, one execution service 3 is configured to perform the extraction task and map the extracted data to the target location; wherein both the configuration service and the execution service are registered with the microservice architecture, and both use the REST interface to transmit data when communicating with the data producer.

[0054] The data producer can be the database of an external enterprise, institution, or business department, and the data consumer can be the party that needs to integrate part of the data in the database of the above external enterprise, institution, or business department. Figure 2 A relationship diagram between data consumers and data producers provided in an embodiment of the present disclosure.

[0055] exist Figure 2 Manufacturer C, as a data consumer, needs to establish an operation platform. The data in the database corresponding to the operation platform needs to be extracted from the smart security database of manufacturer A and the smart energy database of manufacturer B, which are data producers.

[0056] See also Figure 3 A schematic diagram of the system structure used by the operating platform of the data consumer provided in the embodiment of the present disclosure. The system is based on a microservice architecture, and the system includes a registration center cluster, which can be composed of multiple service registration centers 1. The service registration center 1 is composed of servers that provide registration services for services in the system, and different service registration centers 1 can provide different types of registration services.

[0057] In the microservice architecture, Spring Eureka (a tool with service registration and discovery capabilities) is usually used as the registration center1. Through Spring Eureka, you can query the services included in the system (including configuration services, execution services, etc.), the registration status, availability, status, etc. of the services, so as to manage and control the registered services.

[0058] The system also includes a business service cluster. According to the service list obtained from the registration center cluster, it can be determined that the business service cluster includes a configuration unit service cluster and an execution unit service set. The configuration unit service cluster includes configuration service A, configuration service B, and other configuration services (corresponding to other business systems of other data consumers). Configuration service A extracts the required data from manufacturer A (data producer) (referred to as extraction task A) and performs relevant configuration. Configuration service B extracts the required data from manufacturer B (data producer) (referred to as extraction task B) and performs relevant configuration. The execution service unit cluster includes execution service A corresponding to configuration service A, execution service B corresponding to configuration service B, and other execution services corresponding to other configuration services. Execution service A is configured to execute extraction task A generated by configuration service A, and map the data extracted from manufacturer A to the database and / or cache (target location) of manufacturer C (data consumer). Execution service B is configured to execute extraction task B generated by configuration service B, and map the data extracted from manufacturer B to the database and / or cache (target location) of manufacturer C (data consumer).

[0059] The relevant parameters required in the above configuration services can be input by the user at the user end, and transmitted to the system through the network, and transmitted to the corresponding configuration service via the load balancing and gateway services in the system.

[0060] When the operation platform extracts data from the smart security, it needs to send a request message to the smart security via the network, so that the smart security will return the extracted data to the operation platform in the response message.

[0061] In order to achieve the above communication, the data consumer defines a REST interface based on Representational State Transfer (REST), and allows the data producer to exchange data with the data consumer through the REST interface.

[0062] REST can be used to regulate how the client interacts with the server's Application Programming Interface (API) at the Hypertext Transfer Protocol (HTTP) layer. REST describes the data interaction rules between the client and the server at the HTTP layer; the client completes an HTTP interaction by sending an HTTP(s) request to the server and receiving the server's response. During this interaction, REST stipulates two important aspects: the method used by the HTTP request and the link of the request.

[0063] In the present disclosure, the data consumer is regarded as the client, the data producer is regarded as the server, and the REST-based network interface uses the URL corresponding to the user list (that is, the URL of the data producer) as the request link, and the operation corresponding to the interface as the method adopted in the HTTP request. The data required by the data consumer or the information provided to the data producer is provided in the request parameters, and the data returned by the data producer based on the request parameters is stored in the return value.

[0064] Specifically, the REST interface includes the following components:

[0065] User list, used to store the URL of the data producer;

[0066] Interface, which is used to represent the operations performed on the data producer;

[0067] Request parameters, used to represent the parameters requested from the data producer;

[0068] The return value is used to represent the data returned by the data producer based on the request parameters.

[0069] The data format used by the request parameters and return values ​​is JSON.

[0070] Please refer to Table 1 for the definition of the REST-based network interface provided for the implementation of the present disclosure.

[0071] Table 1

[0072] name content User List (URL) Interface (Method) Request parameters (parameters) Return Value

[0073] The REST interface includes a REST-based authentication interface and a REST-based data extraction interface; wherein the authentication interface is configured to obtain authorization information for accessing a data producer, and the data extraction interface is configured to extract data from the data producer using the authorization information.

[0074] It should be noted that both the authentication interface and the data extraction interface are REST interfaces and have the same components as the REST interface. The difference is that the authentication interface and the data extraction interface implement different functions.

[0075] In order to legally extract data from the data producer, the data consumer needs to register with the data producer first. At this time, the configuration service 2 is also configured as follows:

[0076] Send authorization request information to the data producer through the authentication interface; wherein the authorization request carries the user name and user password required for registration on the data consumer; receive authorization information returned by the data producer based on the authorization request information; wherein the authorization information is generated based on the user name and user password.

[0077] When the REST interface is an authentication interface, the request parameters carry an authorization request, and the return value carries authorization information.

[0078] Please refer to Table 2 for a code table for registration based on the REST interface provided in an embodiment of the present disclosure.

[0079] Table 2

[0080]

[0081] The data producer can add (POST represents adding) a data producer (the corresponding website is boe.com.cn / va / x1) through the content shown in Table 2 above. The authorization request information carries the registered user name "user" and the password used is "123456". This information is used as a request parameter based on the REST interface. The data producer generates the corresponding authorization information "0a32d8de-4789-49a9-afd7-c5544894fdf5" according to the above information provided in the request parameter. This authorization information is used as the corresponding information and stored in the return value based on the REST interface. After the data consumer obtains this authorization information, it can use the authorization information to send a data extraction request to the data producer to extract the required data. Since the data consumer usually needs to interact with multiple data producers, the data consumer can obtain the authorization information of multiple data producers by using the above method. For the convenience of management, the data consumer can store the authorization information obtained from each data producer in the local database.

[0082] After obtaining the authorization information of the data producer, the data consumer also needs to obtain the fields and field types of the source data table of the data producer to facilitate the establishment of the corresponding extraction task. To achieve this function, the configuration service 2 can also be configured as follows:

[0083] Obtain the fields contained in the source data table where the required data is located from the data producer; map the fields contained in the source data table to the fields of the target data table of the data consumer, and establish the corresponding extraction task.

[0084] Configuration service 2 obtains the fields contained in the source data table where the required data is located from the data producer. This can be achieved in the following ways:

[0085] A data extraction request is sent to the data producer through the data extraction interface; wherein the data extraction request carries authorization information and relevant information of the source data table; a sample data in the source data table returned based on the data extraction request is received; and the fields contained in the source data table and the corresponding field types are obtained based on the sample data.

[0086] When the REST interface is a data extraction interface, the request parameters carry authorization information and a data extraction request, and the return value carries the data returned by the data producer based on the data extraction request.

[0087] Taking Table 2 as an example, after obtaining the authorization information in Table 2, the data consumer sends a data extraction request carrying the authorization information to the data producer in Table 2 through the data extraction interface, and the data producer returns a response message (including a sample data) through the data extraction interface according to the data extraction request. Please refer to Table 3 for a code table for extracting data based on the data extraction interface provided in the embodiment of the present disclosure.

[0088] Table 3

[0089]

[0090] In Table 3, based on the data extraction interface, the data consumer carries the authorization information and the relevant information of the source data table in the request parameters of the REST interface (the range of data to be extracted from the source data table of the data producer is 1 page at a time, and a total of 20 pages of data are obtained). The information in the above request parameters is also the information in the data extraction request. After receiving the above data extraction request, the data producer returns the corresponding response information. The content of the response information is placed in the return value of the REST interface. The return value includes not only the data returned based on the request parameters ("name":"apple", "color":"red"), but also the status of the response message ("ok", "200", 200 indicates a successful response), and the source data table where the returned data is located. Related information ("pageindex": 2, "totalpage": 50, "totalsize": 10000), so that the data producer tells the data consumer through the content in the return value that the source data table currently returns the data of the second page, there are 50 pages in total, 10,000 data, and the extracted data currently returned is "name": "apple", "color": "red" (this data is the sample data, if multiple data are returned, any one of them will be used as sample data). According to the extracted sample data, it can be determined that the source data table includes the two fields of name and color, and according to apple and red, it can be determined that the field types of the two fields of name and color are character types. Through the data returned in the above return value and the related information of the source data table where the data is located, it is possible to determine the fields and field types, data volume, etc. included in the source data table, so as to facilitate the accurate definition of the data source of the extraction task.

[0091] After obtaining the fields and field types contained in the source data table, the fields contained in the source data table can be mapped to the fields of the target data table of the data consumer. Specifically, the configuration service is configured as follows:

[0092] Define the data source of the extraction task based on the data producer and the source data table, and define the target data model corresponding to the target data table; wherein the target data model includes the method of writing the data extracted from the data producer into the target data table, and the data format adopted by the field type of the field in the target data table; establish the mapping relationship between the fields in the source data table and the fields in the target data table and the data format conversion method of the mapping fields; configure the execution cycle and data synchronization method of the extraction task, and establish the extraction task based on the target data model, the mapping relationship and the corresponding data format conversion method.

[0093] The definition of data sources, the configuration of target data models, and the establishment of mapping relationships are completed using a graphical interface based on user operations, which can be voice commands, touch commands, motion recognition, etc.

[0094] See also Figure 4 Configuration diagram of a graphical interface for defining data sources provided in an embodiment of the present disclosure Figure 1 .

[0095] exist Figure 4 In the graphical interface shown, the parameters that need to be configured include: select the data source type (REST), template name (REST01), template description (REST01 description), request protocol (http and https are supported, http is selected), service address (10.10.85.33:7000), Pageindex (1), Pagesize (10), request method (POST), login API (isys / login), user name (user name), password, authentication method (JWT is selected). Users only need to enter the relevant parameters in the above graphical interface.

[0096] according to Figure 4 The content configured in can determine that the parameters (request parameters) required to be requested in the authentication interface include: user name test01, password, and the parameters (request parameters) required to be requested in the data extraction interface include: extracting data from pages 1 to 10 in the source data table.

[0097] See also Figure 5 Schematic diagram of the configuration of the graphical interface for defining the data source provided in the embodiment of the present disclosure Figure 2 .

[0098] exist Figure 5 In the graphical interface shown, the parameters that need to be configured include: data source type (set to REST), data source (set to data-resource-ap), request API (set to / peds:nan / staffs), request method (select GET), request parameters (the parameter name and parameter value of any parameter can be set, such as the parameter value of the parameter named Pageindex is 1, and the parameter value of the parameter named Pagesize is 20. In this item, you can use the provided plus and minus signs to delete the established parameters, or use the plus sign to add a new parameter), parameter preview (generate the corresponding preview data Pageindex=1&Pagesize=20 according to the set parameters).

[0099] The source definition of the data producer can be obtained through Figure 4 and Figure 5The user enters the information through a graphical interface and completes the source definition of the source data table based on the information entered by the user.

[0100] See also Figure 6 A schematic diagram of parameter configuration of a graphical interface of a target data model provided in an embodiment of the present disclosure.

[0101] exist Figure 6 The graphical interface shown includes the following parameters: select data source type (select POSTGRESOL), template name (pg_test), template description (description of pg_test), POSTGRESOL connection parameters (set parameter name to sslmode and parameter value to disabie), POSTGRESOL address (set to 10.10.85.33:5432), authentication method (select default). Users can complete the definition of the target data model by setting relevant parameters in the above graphical interface.

[0102] See also Figure 7 A schematic diagram of parameter configuration of a graphical interface of a target data table provided in an embodiment of the present disclosure.

[0103] exist Figure 7 The graphical interface shown includes the following parameters: data source type (select postgresql), data source (select datasource-pg-dest), database name (i.e. the database name of the target database is set to test), table name (set to tb_staff), write method (set to insert), batch size, etc. By configuring parameters in the graphical interface, users can define which table in the target database the extracted data needs to be written to, whether to write by insert or other methods (such as overwrite), what the batch size is, etc. When writing by insert, only the newly added data can be written, which can reduce the amount of data transmission; when writing by overwrite, the changed data can be updated.

[0104] See also Figure 8 A graphical interface for configuring a mapping relationship between a source data table and a target data table provided in an embodiment of the present disclosure.

[0105] exist Figure 8 The graphical interface shown mainly includes four parts: setting the fields contained in the source data table (called source fields) and field types (referred to as types for short), setting the fields contained in the corresponding source data table (called target fields) and field types (referred to as types for short), and what kind of verification function (what parameters the verification function uses) should be used for verification of the corresponding source fields and target fields, and what kind of conversion function (the parameters used by the conversion function) should be used for data conversion. Figure 8 As shown, the source field (including field 1 to field 3), its corresponding type (type 1 to type 3), the target field (including field a to field c), its corresponding type (type a to type c), Figure 8 The verification function, conversion function and their parameter settings are not specifically shown. The user can directly configure the mapping relationship between each source field and each target field (the same row is a mapping relationship), and their respective field types (i.e. types) through the above graphical interface. If the field types of a source field with a mapping relationship and the corresponding target field are different, a conversion function can be set, and a verification function can also be set for verification.

[0106] The above-mentioned verification function can be a verification rule, such as verifying whether the meaning of the corresponding data before and after the conversion by using the conversion function is the same. Using the verification function to verify the conversion result can prevent the conversion function from making errors during the conversion process, which will cause errors in the data corresponding to the target field, thereby improving the accuracy of data conversion. The conversion function converts the field type corresponding to the source field into the field type corresponding to the target field, which can automatically heterogeneously transform the data extracted from the data producer into the data consumer, thereby improving the efficiency of data heterogeneity.

[0107] See also Fig. 9 A graphical interface for configuring extraction tasks provided in an embodiment of the present disclosure.

[0108] exist Fig. 9 The graphical interface shown includes the following parameters: task name, reminder, timer (e.g., choose to execute the data extraction task every 05 minutes), task description, execution user (e.g., set to admin), execution node (e.g., set to 10.10.85.33.9501), timeout (e.g., set to 43200s), synchronization mode (you can choose full or incremental, Fig. 9 The full amount is selected in the Fig. 9 The graphical interface shown can configure the execution cycle of a certain extraction task, etc.

[0109] In completion Figures 4 to 9 After the configuration, the configuration service 2 creates an extraction task and provides the extraction task to the execution service 3.

[0110] Execution Service 3 is configured as:

[0111] Generate corresponding data extraction request according to the extraction task; send data extraction request to the data producer through the data extraction interface, and receive the corresponding data returned from the source data table; convert the returned data into data in the target data table for storage through the extraction task.

[0112] See also Fig.10 A schematic diagram of the relationship between the configuration service and the execution service provided in the embodiment of the present disclosure.

[0113] Continue with Figure 2 For example, the configuration service can include the following main components: data source definition, target data model definition, data field mapping, and extraction task configuration. The configuration methods of the above parts have been introduced before, so I will not repeat them here. Every time the configuration service creates an extraction task, the corresponding extraction task will be sent to the execution service.

[0114] Execution service 3 may include several main components according to its functions: executor, task scheduling, data mapping conversion, and target data preservation. The extraction task established by configuration service 2 includes the corresponding executor in execution service 3, and the specific executor that controls which extraction task is running is determined by task scheduling. When the execution service receives data from the data producer (assuming that the data producer of the currently extracted data is manufacturer A), task scheduling controls the execution of the executor of the extraction task corresponding to manufacturer A, generates the corresponding data extraction request, and after authorization through the authentication interface, sends the data collection request to the smart security (source data table) of manufacturer A through the data extraction interface, so as to extract data from the smart security of manufacturer A, perform data mapping and data format conversion according to the mapping relationship and field type between the source data table and the target data table configured in the configuration service, obtain the target data table, and maintain it (i.e., save the target data).

[0115] It should be noted that in Fig.10 The configuration service and execution service are divided into Fig.10 The four functions shown in the figure can be divided into one, two, three or even more functions in actual applications. Therefore, it should not be understood that configuration services and execution services are limited to being divided into Fig.10 The 4 functions shown.

[0116] Continue with Figure 2 For example, the current operation analysis platform (manufacturer C) needs the equipment data of smart security (manufacturer A) and the power data of smart energy (manufacturer B). The code diagram of the data exchange between manufacturer C as the data consumer and manufacturer A and manufacturer B as the data producers based on the REST interface is shown in Table 4:

[0117] Table 4

[0118]

[0119]

[0120] According to the information in the REST interface in Table 4 above, manufacturer A and manufacturer B can be defined as the source of metadata in the target database of manufacturer C (such as defined as sourceDs1 and sourceDs2 respectively), and the tables in the target database corresponding to manufacturer A and manufacturer B can be defined (such as defined as destDs1 and destDs2 respectively), sourceDs1 can be mapped to destDs1 and extraction task 1 can be created, sourceDs2 can be mapped to destDs2 and extraction task 2 can be created, and by executing the executors corresponding to extraction task 1 and extraction task 2 respectively, data can be automatically extracted from manufacturers A and B and mapped to the target data table, and then the target data table can be stored in the target location.

[0121] In the embodiment provided by the present disclosure, by making the system of the data consumer include at least one configuration service and at least one execution service, a configuration service is configured to perform relevant configuration on the extraction task of extracting data from a corresponding data producer based on the REST interface; a corresponding execution service is configured to execute the extraction task, and obtain the extracted data through the REST interface, and then map the extracted data to the target location. Since the REST interface defined by the data consumer is adopted, the data consumer can send an http request to the data producer as the server as a client, so that the data producer returns the data required to be extracted by the data consumer according to the REST interface defined by the data consumer, and the data consumer maps the extracted data to the target data table, so that the data consumer can obtain the required extracted data without invading the database of the data producer, thereby improving the security of data extraction, and there is no need to deploy interface programs on the data consumer and the data producer, and develop the interface program separately, thereby reducing the investment of manpower and time costs, and thus reducing the development cost of the enterprise. And because the data consumer configures the relevant parameters through a graphical interface before extracting data, the user's ease of operation is improved, and the operation is simple and convenient.

[0122] Please refer to Table 5 for a comparison table of the advantages and disadvantages of the present disclosure based on the microservice architecture and other systems provided in the embodiments of the present disclosure.

[0123] Table 5

[0124]

[0125]

[0126] In the embodiments provided by the present disclosure, since the system for general data extraction is developed based on the microservice architecture, and the configuration service, execution service, etc. are registered in the service registration center, the configuration service and execution service for different data producers can be quickly launched, which improves the autonomy and independence of each service, and allows the service for new data producers to be quickly released and launched without worrying about the wide-ranging impact and spread on other functions of the system. In addition, the above services can exist in the form of components, so that they can be reused and reorganized to quickly form and release new data extraction applications.

[0127] When more users access this system for data extraction, some services in the data extraction application can be expanded to solve performance bottlenecks. The components corresponding to a service in the microservice can be replaced or restored independently.

[0128] Furthermore, the system using the microservice architecture system has incomparable advantages in development efficiency, stability, and scalability, ensuring high availability and high concurrency of services. It can also enable data extraction applications running on the system to be quickly launched, which means improved speed and efficiency, and can achieve independent expansion and recovery, which means higher security, stability, and scalability of the system.

[0129] Furthermore, since the microservice itself is an independently published, independently deployed, autonomous, tiny service, each of the above services can be independently arranged in a container, and the container is also cross-platform, independently operated, and is a small execution unit. Therefore, in the embodiments of the present disclosure, the deployment of each service adopts a containerized deployment method, which allows the services contained in the entire microservice architecture and the dependent environment to be packaged as a container image for deployment. In this way, the container only needs to encapsulate the service and the dependent files required by the service, thereby realizing a lightweight operating environment and having a higher hardware resource utilization than a virtual machine. Furthermore, different services in applications that rely on the above services can be isolated from each other, realizing one-click deployment of services, and greatly reducing the workload of operation and maintenance personnel.

[0130] Those skilled in the art will appreciate that the disclosed embodiments may be provided as methods, systems, or computer program products. Therefore, the disclosed embodiments may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the disclosed embodiments may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0134] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure.

[0135] Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure is also intended to include these modifications and variations.

Claims

1. A general data extraction system, applied to data consumers, based on a microservice architecture, including a service registration center, wherein: include: At least one configuration service, one of the configuration services is configured to obtain authorization information for legally extracting data from a data producer, and send a data extraction request carrying the authorization information to the data producer; wherein the data extraction request is used to request to extract a sample data from a source data table of the data producer; receive the sample data returned based on the data extraction request; obtain the fields contained in the source data table and the corresponding field types based on the sample data; define the data source of the extraction task based on the data producer and the source data table, and define the target data model corresponding to the target data table; wherein the target data model includes a method for writing the data extracted from the data producer into the target data table, and a data format adopted by the field type of the field in the target data table; establish a mapping relationship between the fields in the source data table and the fields in the target data table, and a data format conversion method for the mapping fields; configure the execution cycle and data synchronization method of the extraction task, and establish the extraction task based on the target data model, the mapping relationship and the corresponding data format conversion method; At least one execution service, one of the execution services is configured to generate a corresponding data extraction request according to the extraction task and send it to the data producer, receive the data extracted from the source data table returned by the data producer, and convert it into data in the target data table for storage; wherein the configuration service and the execution service are both registered with the service registration center.

2. The system of claim 1, wherein: The service registration center communicates with the data producer using a REST interface, and the REST interface includes: A REST-based authentication interface and a REST-based data extraction interface; wherein the authentication interface is configured to obtain the authorization information, and the data extraction interface is configured to extract data from the data producer using the authorization information.

3. The system of claim 2, wherein: The configuration service is also configured to: Sending authorization request information to the data producer through the authentication interface; wherein the authorization request information carries the user name and user password required for the data consumer to register; Receive the authorization information returned by the data producer based on the authorization request information; wherein the authorization information is generated based on the user name and the user password.

4. A system as claimed in claim 2 or 3, wherein: The REST interface further includes: User list, used to store the website address of the data producer; An interface, used to represent the operations performed on the data producer; Request parameters, used to represent the parameters requested from the data producer; The return value is used to represent the data returned by the data producer based on the request parameters.

5. The system of claim 4, wherein: The data format used by the request parameters and the return value is JSON format.

6. The system of claim 5, wherein: When the REST interface is the authentication interface, the request parameter carries the authorization request information, and the return value carries the authorization information.

7. The system of claim 5, wherein: When the REST interface is the data extraction interface, the request parameter carries the authorization information and the data extraction request, and the return value carries the data returned by the data producer based on the data extraction request.

8. The system of claim 1, wherein: The definition of the data source, the configuration of the target data model, and the establishment of the mapping relationship are completed based on user operations using a graphical interface.

9. The system according to any one of claims 1 to 3, wherein: Both the configuration service and the execution service use container technology, and each of the configuration service and each of the execution services runs in a corresponding container.