An ETL data processing system and method combined with a Dubbo service

Abstract

The application provides an ETL data processing system and method combined with a Dubbo service, and the system comprises an ETL data extraction module, a first ETL data cleaning and conversion module, a Dubbo service module, a second ETL data cleaning and conversion module and a data loading module; the ETL data extraction module is used for data extraction; the first ETL data cleaning and conversion module is used for data cleaning and conversion processing of the extracted data; the Dubbo service module is used for calling a corresponding Dubbo service object according to business requirements, processing the data, and storing the processed data in a target database or inputting the data into the second ETL data cleaning and conversion module according to business requirements; the second ETL data cleaning and conversion module is used for data cleaning and conversion processing; and the data loading module is used for loading the processed data into the target database; the Dubbo service module is used for calling a suitable service object to process the data, and the processing capacity for complex data is enhanced.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to an ETL data processing system and method that integrates Dubbo services. Background Technology

[0002] With the rapid development of Internet technology, current business lines are becoming increasingly complex, and various business systems operate independently. The demand for unified extraction, transformation, and final storage of heterogeneous data sources generated from different business sources is increasing, which has led to the emergence of ETL tools.

[0003] Currently, most ETL tools on the market, while having multiple entry points at the data processing source, have relatively "single" data processing methods. They lack the ability to handle complex data, have poor scalability, and cannot flexibly meet complex business needs. This is especially true when there are many data sources and significant data differences, making it difficult to accurately locate the real data.

[0004] Therefore, there is an urgent need to provide an ETL data processing system and method that combines Dubbo services to solve the problems that existing ETL tools have a relatively "single" data processing method, insufficient ability to process complex data, poor scalability, and inability to flexibly meet complex business needs. Summary of the Invention

[0005] In view of this, it is necessary to provide an ETL data processing system and method that combines Dubbo services to solve the problems that existing ETL tools have a relatively "single" data processing method, insufficient ability to process complex data, poor scalability, and inability to flexibly meet complex business needs.

[0006] On one hand, the present invention provides an ETL data processing system that integrates Dubbo services, comprising:

[0007] The system comprises an ETL data extraction module, a first ETL data cleaning and transformation module, a Dubbo service module, a second ETL data cleaning and transformation module, and a data loading module.

[0008] The ETL data extraction module is used to set extraction configuration information according to business needs, and extract data from the business system according to the extraction configuration information to obtain the data to be processed.

[0009] The first ETL data cleaning and transformation module is used to perform data cleaning and transformation on the data to be processed to obtain the first target data;

[0010] The Dubbo service module is used to call the corresponding Dubbo service object according to business needs, process the data to be processed and the first target data to obtain the second target data, and store the second target data in the target database or input the second target data into the second ETL data cleaning and transformation module for processing according to business needs;

[0011] The second ETL data cleaning and transformation module is used to clean and transform the second target data to obtain the third target data;

[0012] The data loading module is used to load the first target data, the second target dataset, and the third target data into the target database.

[0013] In some possible implementations, the ETL data extraction module may also include a homogeneous data extraction module, a heterogeneous data extraction module, and a file type data source extraction module;

[0014] The same-source data extraction module is used to extract data from the business system that is from the same data source as the data warehouse by establishing a link between the data warehouse and the business system.

[0015] The heterogeneous data extraction module is used to extract data from data sources in the business system that are different from the data warehouse, by establishing a link between the data warehouse and the business system in a way that connects to the data source.

[0016] The file type data source extraction module is used to extract data stored in the text database.

[0017] Among the possible implementation methods is: a Dubbo service parameter configuration module;

[0018] The Dubbo service parameter configuration module is used to configure the service parameters for calling the Dubbo service proxy object and the output parameters after data processing.

[0019] In some possible implementations, the Dubbo service module includes: a configuration information invocation module, a service parameter verification module, and a Dubbo service proxy object invocation module;

[0020] The configuration information calling module is used to call the service parameters of the Dubbo service proxy object configured by the Dubbo service parameter configuration module, as well as the output parameters after data processing;

[0021] The service parameter verification module is used to determine whether the configured service parameters for calling the Dubbo service proxy object and the output parameters after processing the data are valid according to the preset service parameter information verification rules.

[0022] The Dubbo proxy object invocation module is used to invoke the matching Dubbo service proxy object to process the data that needs to be processed by the Dubbo service proxy object, based on the service parameters of the Dubbo service proxy object and the output parameters after data processing.

[0023] In some possible implementations, the Dubbo service module includes: a data return module;

[0024] The data return module is used to return the service parameter information verification rule parameters to the second ETL data cleaning and transformation module when the configured service parameters for calling the Dubbo service proxy object and the output parameters after data processing are both valid.

[0025] When the configured service parameters for calling the Dubbo service proxy object and the output parameters after data processing are invalid, the system determines whether to execute the termination exception process and return termination exception information or return information to skip the error parameters and continue executing the task, based on the set custom parameters, and returns the information to the second ETL data cleaning and transformation module.

[0026] In some possible implementations, the Dubbo service module includes: a data inheritance module;

[0027] The data inheritance module is used to obtain the first target data and the data to be processed.

[0028] In some possible implementations, the first ETL data cleaning and transformation module includes a first data cleaning module and a first data transformation module;

[0029] The first data cleaning module is used to clean and filter data that does not conform to the set parameter cleaning rules; the first data conversion module is used to convert the filtered data to obtain the first target data according to the set parameter conversion rules.

[0030] In some possible implementations, the second ETL data cleaning and transformation module includes a second data cleaning module and a second data transformation module;

[0031] The second data cleaning module is used to filter out non-compliant data in the second target data according to the set data cleaning rules;

[0032] The second data conversion module is used to convert the filtered second target data into third target data according to the set data conversion rules.

[0033] In some possible implementations, the ETL data processing system that incorporates Dubbo services also includes: a Dubbo service object construction module;

[0034] The Dubbo service object construction module is used to construct Dubbo service objects based on the configured Dubbo service-related parameters.

[0035] On the other hand, the present invention also provides an ETL data processing method based on Dubbo services, comprising:

[0036] Set the extraction configuration information according to business needs, and extract the data to be processed from the data in the business system according to the extraction configuration information.

[0037] The data to be processed is cleaned and transformed to obtain the first target data;

[0038] According to business requirements, the corresponding Dubbo service object is called to process the data to be processed and the first target data to obtain the second target data. The second target data is then stored in the target database or, according to business requirements, the second target data is input into the second ETL data cleaning and transformation module for processing.

[0039] The second target data is cleaned and transformed to obtain the third target data;

[0040] Load the first target data, the second target dataset, and the third target data into the target database.

[0041] The beneficial effects of adopting the above embodiments are as follows: Compared with the prior art, the ETL data processing system combined with Dubbo service provided by the embodiments of the present invention realizes the ETL processing of complex data through the set ETL data extraction module, first ETL data cleaning and transformation module, Dubbo service module, second ETL data cleaning and transformation module and data loading module. It can access ETL tools through the Dubbo service module to call appropriate proxy objects to process data according to business needs, thereby improving the ETL's ability to process complex data and expanding the ETL's ability to process dynamic data. Attached Figure Description

[0042] Figure 1 A schematic diagram of the system structure of an embodiment of the ETL data processing system combining Dubbo services provided by the present invention;

[0043] Figure 2 A schematic diagram of the structure of an embodiment of the ETL data extraction module provided by the present invention;

[0044] Figure 3 This invention provides another embodiment of an ETL data processing system incorporating Dubbo services, illustrating the system architecture.

[0045] Figure 4This is a schematic diagram of the structure of an embodiment of the Dubbo service module provided by the present invention;

[0046] Figure 5 This is a schematic diagram of another embodiment of the Dubbo service module provided by the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of one embodiment of the first ETL data cleaning and conversion module provided by the present invention;

[0048] Figure 7 This is a schematic diagram of the structure of an embodiment of the second data cleaning and transformation module provided by the present invention;

[0049] Figure 8 This is a schematic diagram of the system structure of another embodiment of the ETL data processing system combined with Dubbo service provided by the present invention;

[0050] Figure 9 This is a flowchart of an embodiment of the ETL data processing method based on Dubbo services provided by the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0052] It should be understood that the illustrative drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0053] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0054] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] This invention provides an ETL data processing system that integrates Dubbo services, which will be described below.

[0056] Figure 1 This is a schematic diagram of the system architecture of an embodiment of the ETL data processing system combining Dubbo services provided by the present invention, as shown below. Figure 1 The ETL data processing system 10, which combines Dubbo services, includes: an ETL data extraction module 100, a first ETL data cleaning and transformation module 200, a Dubbo service module 300, a second ETL data cleaning and transformation module 400, and a data loading module 500.

[0057] The ETL data extraction module 100 is used to set extraction configuration information according to business needs, and extract data from the data in the business system according to the extraction configuration information to obtain the data to be processed.

[0058] The first ETL data cleaning and transformation module 200 is used to perform data cleaning and transformation on the data to be processed to obtain the first target data;

[0059] The Dubbo service module 300 is used to call the corresponding Dubbo service object according to business needs, process the data to be processed and the first target data to obtain the second target data, and store the second target data in the target database or input the second target data into the second ETL data cleaning and transformation module for processing according to business needs;

[0060] The second ETL data cleaning and transformation module 400 is used to clean and transform the second target data to obtain the third target data;

[0061] The data loading module 500 is used to load the first target data, the second target dataset, and the third target data into the target database.

[0062] Compared with the prior art, the ETL data processing system 10 combined with Dubbo service provided in this embodiment of the invention realizes ETL processing of complex data through the set TL data extraction module 100, first ETL data cleaning and transformation module 200, Dubbo service module 300, second ETL data cleaning and transformation module 400 and data loading module 500. It can access ETL through Dubbo service module 300 to call appropriate proxy objects to process data according to business needs, thereby improving the ETL's ability to process complex data and expanding the ETL's ability to process dynamic data.

[0063] In a specific embodiment of the present invention, the Dubbo service module 300 calls the corresponding Dubbo service object according to business needs to process the data that needs to be processed by Dubbo service in the business system extracted by the ETL data extraction module 100, or processes the data processed by the first ETL data processing module 200 again through the Dubbo service. At the same time, it can also input the data processed by the Dubbo service into the second ETL data cleaning and transformation module 400 for data cleaning and transformation again according to business needs; and store the processed data in the target database.

[0064] It should be noted that ETL is an abbreviation for Extract-Transform-Load, which describes the process of extracting, transforming, and loading data from the source to the destination.

[0065] It should be further noted that Dubbo is a service framework that enables applications to implement service output and input functions through high-performance RPC (Remote Procedure Call).

[0066] It should be noted that the first ETL data cleaning and conversion module 200 and the second ETL data cleaning and conversion module 400 are two modules with the same function.

[0067] In some embodiments of the present invention, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of an embodiment of the ETL data extraction module provided by the present invention. The ETL data extraction module 100 further includes a homogeneous data extraction module 201, a heterogeneous data extraction module 202, and a file type data source extraction module 203.

[0068] The same-source data extraction module 201 is used to extract data from the business system that is the same data source as the data warehouse by establishing a link relationship between the data warehouse and the business system.

[0069] The heterogeneous data extraction module 202 is used to extract data from data sources in the business system that are different from the data warehouse data source through the link relationship between the data warehouse and the business system established by connecting the data source.

[0070] The file type data source extraction module 203 is used to extract data stored in the text database.

[0071] In a specific embodiment, the same-source data extraction module 201 extracts data by directly accessing it through a Select statement by establishing a link relationship between the data warehouse and the business system.

[0072] The heterogeneous data extraction module 202 establishes a link between the data warehouse and the business system via ODBC to extract data from different sources in the business system.

[0073] It should be noted that the data warehouse, also known as a data warehouse, can be abbreviated as DW or DWH. A data warehouse is a strategic collection of data that provides support for decision-making processes at all levels of an enterprise. It is a single data store created for analytical reporting and decision support purposes.

[0074] It should be further noted that ODBC stands for Open Database Connectivity, which is used to solve the problem of data sharing between heterogeneous databases.

[0075] In some embodiments of the present invention, such as Figure 3 As shown, Figure 3 The system architecture diagram of another embodiment of the ETL data processing system combined with Dubbo service provided by the present invention is shown below. The ETL data processing system combined with Dubbo service further includes: Dubbo service parameter configuration module 600;

[0076] The Dubbo service parameter configuration module 600 is used to configure the service parameters for calling the Dubbo service proxy object and the output parameters after data processing.

[0077] In some possible ways, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of an embodiment of the Dubbo service module provided by the present invention. The Dubbo service module 300 includes: a configuration information calling module 301, a service parameter verification module 302, and a Dubbo service proxy object calling module 303.

[0078] The configuration information calling module 301 is used to call the service parameters of the Dubbo service proxy object configured by the Dubbo service parameter configuration module, as well as the output parameters after processing the data.

[0079] The service parameter verification module 302 is used to determine whether the configured service parameters for calling the Dubbo service proxy object and the output parameters after processing the data are valid according to the preset service parameter information verification rules.

[0080] The Dubbo proxy object invocation module 303 is used to invoke the matching Dubbo service proxy object to process the data that needs to be processed by the Dubbo service proxy object according to the service parameters of the Dubbo service proxy object and the output parameters after the data is processed.

[0081] The data storage module 304 is used to store the processed data of the Dubbo service proxy object in the database.

[0082] In a specific embodiment, the configuration information calling module 301 accesses the ETL API port by inheriting the data processor, and obtains the service parameters of the Dubbo service proxy object configured in the Dubbo service parameter configuration module 600, as well as the output parameters after data processing.

[0083] The service parameter verification module 302 verifies the service parameters of the configured Dubbo service proxy object and the output parameters after data processing according to the preset parameter verification rules, and determines whether the parameters are valid. If the parameters are valid, execution continues; if they are invalid, invalid information is output to the data return module.

[0084] The Dubbo proxy object invocation module 303 uses the service parameters of the obtained Dubbo service proxy object to invoke the corresponding Dubbo service proxy object at the Zookeeper registry address, thereby enabling the invocation of different Dubbo service proxy objects based on different parameters to obtain different data processing capabilities.

[0085] It should be noted that after the Dubbo proxy object call module 303 obtains the service parameters of the Dubbo service proxy object, the Dubbo service proxy object is called through Java reflection.

[0086] Furthermore, it's important to note that reflection is a feature of the Java programming language, allowing a running Java program to inspect itself. While resources encapsulated as private can only be accessed within a class and not from outside, reflection can directly manipulate private class attributes. Reflection can obtain all information about a class at runtime (including member variables, member methods, constructors, etc.) and can manipulate the class's fields, methods, constructors, and other related parts.

[0087] Furthermore, it should be noted that Zookeeper is a highly available, high-performance distributed coordination service that can be used for service discovery, distributed locks, distributed leader election, configuration management, and more.

[0088] In some possible ways, such as Figure 5 As shown, Figure 5 This is a schematic diagram of another embodiment of the Dubbo service module provided by the present invention. The Dubbo service module 300 further includes: a data return module 305.

[0089] The data return module 305 is used to return the parameters of the service parameter information verification rules to the second ETL data cleaning and conversion module 400 when the configured service parameters for calling the Dubbo service proxy object and the output parameters after data processing are both valid.

[0090] When the configured service parameters for calling the Dubbo service proxy object and the output parameters after data processing are invalid, the system determines whether to execute the termination exception process and return termination exception information or return information to skip the error parameters and continue executing the task, based on the set custom parameters. The information is then returned to the second ETL data cleaning and transformation module 400.

[0091] In some possible ways, such as Figure 5 As shown, Figure 5 This is a schematic diagram of another embodiment of the Dubbo service module provided by the present invention. The Dubbo service module 300 further includes: a data inheritance module 306.

[0092] The data inheritance module 306 is used to acquire the first target data and the data to be processed.

[0093] In a specific embodiment, the data inheritance module obtains the required Dubbo service processing data from the ETL data extraction module through the data inheritance generator.

[0094] In some possible ways, such as Figure 6 As shown, Figure 6This is a schematic diagram of a structure of an embodiment of the first ETL data cleaning and conversion module provided by the present invention. The first ETL data cleaning and conversion module 200 includes a first data cleaning module 201 and a first data conversion module 202.

[0095] The first data cleaning module 201 is used to filter out non-compliant data in the data to be processed according to the set parameter cleaning rules;

[0096] The first data conversion module 202 is used to convert the filtered data to be processed into the first target data according to the set parameter conversion rules.

[0097] In a specific embodiment, the first data cleaning and transformation module cleans the data according to the set parameter cleaning rules;

[0098] It should be noted that the cleaned data includes:

[0099] (1) Incomplete data: This type of data mainly involves missing information, such as supplier names, branch office names, missing customer regional information, and mismatches between the master table and the detailed table in the business system. This type of data is filtered out, and the missing information is written into separate Excel files and submitted to the customer, requiring them to complete the data within a specified time. Only after completion is the data written into the data warehouse.

[0100] (2) Erroneous Data: This type of error occurs because the business system is not robust enough. Input is written directly to the backend database without proper validation. Examples include numeric data being entered as full-width numeric characters, string data ending with a newline character, incorrect date format, and out-of-bounds dates. This type of data also needs to be categorized. For issues like full-width characters or invisible characters before or after the data, the only solution is to use SQL statements to identify them and then require the customer to correct them in the business system before extraction. Errors like incorrect date formats or out-of-bounds dates will cause ETL to fail. These errors need to be identified in the business system database using SQL, submitted to the relevant business department for correction within a specified timeframe, and then extracted.

[0101] (3) Duplicate data: For this type of data—especially in dimension tables—export all fields of the duplicate data records and have the customer confirm and organize them.

[0102] In some possible ways, such as Figure 7 As shown, Figure 7 This is a schematic diagram of an embodiment of the second data cleaning and conversion module provided by the present invention. The second ETL data cleaning and conversion module 400 includes a second data cleaning module 401 and a second data conversion module 402.

[0103] The second data cleaning module 401 is used to filter out non-compliant data in the second target data according to the set data cleaning rules;

[0104] The second data conversion module 402 is used to convert the filtered second target data into third target data according to the set data conversion rules.

[0105] In some possible ways, such as Figure 8 As shown, Figure 8 The schematic diagram of another embodiment of the ETL data processing system combined with Dubbo service provided by the present invention is shown in the present invention. The ETL data processing system 10 combined with Dubbo service also includes: Dubbo service object construction module 600.

[0106] The Dubbo service object construction module 600 is used to construct Dubbo service objects according to the configured Dubbo service-related parameters.

[0107] In a specific implementation, the configured Dubbo service-related parameters are obtained through the data inheritor, a Dubbo service object is constructed, and it is registered in the Zookeeper registry.

[0108] On the other hand, based on the ETL data processing system combined with Dubbo services, this invention also provides an ETL data processing method based on combining Dubbo services, such as... Figure 9 As shown, Figure 9 This is a flowchart of an embodiment of the ETL data processing method based on Dubbo services provided by the present invention. The ETL data processing method based on Dubbo services includes:

[0109] S901. Set the extraction configuration information according to business needs, and extract the data to be processed from the data in the business system according to the extraction configuration information.

[0110] S902. Perform data cleaning and transformation on the data to be processed to obtain the first target data;

[0111] S903. According to business requirements, call the corresponding Dubbo service object to process the data to be processed and the first target data to obtain the second target data, and store the second target data in the target database or input the second target data into the second ETL data cleaning and transformation module for processing according to business requirements.

[0112] S904. Perform data cleaning and transformation on the second target data to obtain the third target data;

[0113] S905. Load the first target data, the second target dataset, and the third target data into the target database.

[0114] It should be noted that the steps in the methods of the above embodiments can be added or extended according to the various modules or units in the ETL data processing system combined with Dubbo services. For details, please refer to the description in the embodiment of the ETL data processing system combined with Dubbo services, which will not be repeated here.

[0115] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0116] The above provides a detailed description of the ETL data processing system and method combined with Dubbo services provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An ETL data processing system integrating Dubbo services, characterized in that, include: The system comprises an ETL data extraction module, a first ETL data cleaning and transformation module, a Dubbo service module, a second ETL data cleaning and transformation module, and a data loading module. The ETL data extraction module is used to set extraction configuration information according to business needs, and extract data from the business system according to the extraction configuration information to obtain the data to be processed. The first ETL data cleaning and transformation module is used to perform data cleaning and transformation on the data to be processed to obtain the first target data; The Dubbo service module is used to call the corresponding Dubbo service object according to business needs, process the data to be processed and the first target data to obtain the second target data, and store the second target data in the target database or input the second target data into the second ETL data cleaning and transformation module for processing according to business needs; The second ETL data cleaning and transformation module is used to clean and transform the second target data to obtain the third target data; The data loading module is used to load the first target data, the second target dataset, and the third target data into the target database; The Dubbo service module includes: a configuration information calling module, a service parameter verification module, a Dubbo service proxy object calling module, and a data return module; The configuration information calling module is used to call the service parameters of the Dubbo service proxy object configured by the Dubbo service parameter configuration module, as well as the output parameters after data processing; The service parameter verification module is used to determine whether the configured service parameters for calling the Dubbo service proxy object and the output parameters after processing the data are valid according to the preset service parameter information verification rules. The Dubbo service proxy object invocation module is used to invoke the matching Dubbo service proxy object to process the data that needs to be processed by the Dubbo service proxy object based on the service parameters of the Dubbo service proxy object and the output parameters after the data is processed. The data return module is used to return the service parameter information verification rule parameters to the second ETL data cleaning and transformation module when the configured service parameters for calling the Dubbo service proxy object and the output parameters after data processing are both valid. When the configured service parameters for calling the Dubbo service proxy object and the output parameters after data processing are invalid, the system determines whether to execute the termination exception process and return termination exception information or return information to skip the error parameters and continue executing the task, based on the set custom parameters, and returns the information to the second ETL data cleaning and transformation module.

2. The ETL data processing system combining Dubbo services according to claim 1, characterized in that, The ETL data extraction module also includes a homogeneous data extraction module, a heterogeneous data extraction module, and a file type data source extraction module; The same-source data extraction module is used to extract data from the business system that is from the same data source as the data warehouse by establishing a link between the data warehouse and the business system. The heterogeneous data extraction module is used to extract data from data sources in the business system that are different from the data warehouse, by establishing a link between the data warehouse and the business system in a way that connects to the data source. The file type data source extraction module is used to extract data stored in the text database.

3. The ETL data processing system combining Dubbo services according to claim 1, characterized in that, It also includes: Dubbo service parameter configuration module; The Dubbo service parameter configuration module is used to configure the service parameters for calling the Dubbo service proxy object and the output parameters after data processing.

4. The ETL data processing system combining Dubbo services according to claim 1, characterized in that, The Dubbo service module includes: a data inheritance module; The data inheritance module is used to obtain the first target data and the data to be processed.

5. The ETL data processing system combining Dubbo services according to claim 1, characterized in that, The first ETL data cleaning and conversion module includes a first data cleaning module and a first data conversion module; The first data cleaning module is used to filter out non-compliant data in the data to be processed according to the set parameter cleaning rules; The first data conversion module is used to convert the filtered data to be processed into the first target data according to the set parameter conversion rules.

6. The ETL data processing system combining Dubbo services according to claim 1, characterized in that, The second ETL data cleaning and conversion module includes a second data cleaning module and a second data conversion module; The second data cleaning module is used to filter out non-compliant data in the second target data according to the set data cleaning rules; The second data conversion module is used to convert the filtered second target data into third target data according to the set data conversion rules.

7. The ETL data processing system combining Dubbo services according to claim 1, characterized in that, The ETL data processing system that integrates Dubbo services also includes: a Dubbo service object construction module; The Dubbo service object construction module is used to construct Dubbo service objects based on the configured Dubbo service-related parameters.

8. A method for ETL data processing based on Dubbo services, comprising the ETL data processing system based on Dubbo services as described in any one of claims 1-7, including: Set the extraction configuration information according to business needs, and extract the data to be processed from the data in the business system according to the extraction configuration information. The data to be processed is cleaned and transformed to obtain the first target data; According to business requirements, the corresponding Dubbo service object is called to process the data to be processed and the first target data to obtain the second target data. The second target data is then stored in the target database or, according to business requirements, the second target data is input into the second ETL data cleaning and transformation module for processing. The second target data is cleaned and transformed to obtain the third target data; Load the first target data, the second target dataset, and the third target data into the target database.

Images