Data extraction method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data extraction method, device and equipment and a computer readable storage medium, and is applied to the technical field of data processing, and the method comprises the steps: obtaining original data of a target data source, and carrying out the initial processing of the original data based on a mapping conversion rule, and obtaining the initial processing data; according to the data source type of the target data source, loading and instantiating an adapter component corresponding to the mapping conversion rule from a pluggable adapter component library; and processing the initial processing data based on the adapter component to obtain target extraction data. Compared with the prior art that a data extraction code needs to be developed for each data source, all data are accessed and analyzed through a uniform process by using a mapping conversion rule and a self-designed adapter component, and extracted data meeting a target is output, so that the quality and consistency of consumption data of a downstream system are ensured, and the data extraction efficiency is improved. And the data extraction efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data extraction method, device, equipment and computer readable storage medium. BACKGROUND

[0002] When constructing a data reconciliation, data center or ETL (Extract, Transform, Load) system, it is usually necessary to extract data from a large number of heterogeneous data sources. These data sources are diverse in type, and their data structures, protocols and access methods differ greatly. The traditional approach is to write special data access and parsing code for each data source, which has the following significant defects: 1. Low development efficiency: each time a new data source is accessed, new code needs to be developed, tested and put into operation, which is time-consuming and costly. 2. Difficult to maintain: when the data source structure changes or needs to be extended, the code must be modified and the system must be redeployed, which is difficult to maintain. 3. Lack of flexibility: it relies heavily on technical personnel, and business personnel cannot quickly respond to changes, so the system is not agile.

[0003] Therefore, how to improve the efficiency and accuracy of data extraction is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a data extraction method, device, equipment and computer readable storage medium, which solves the technical problem of low efficiency and accuracy of data extraction in the prior art.

[0005] To solve the above technical problems, the present application provides a data extraction method, comprising:

[0006] Obtaining original data of a target data source, and performing initial processing on the original data based on a mapping conversion rule to obtain initial processing data;

[0007] According to the data source type of the target data source, loading an adapter component corresponding to the mapping conversion rule from a pluggable adapter component library;

[0008] Processing the initial processing data based on the adapter component to obtain target extraction data.

[0009] Optionally, before obtaining the original data of the target data source, the method further comprises:

[0010] Obtaining data source connection parameters, wherein the data source connection parameters include data source type, host address, port, database name, user name, password, interface address, request method and file path;

[0011] Establishing a connection with the target data source based on the data source connection parameters.

[0012] Optionally, before obtaining original data of a target data source and performing initial processing on the original data based on a mapping conversion rule to obtain initial processing data, the method further comprises:

[0013] The mapping conversion rule from the original data to the standard data is defined through an interface operation, wherein the mapping conversion rule comprises at least one of field mapping, data cleaning and type conversion, and the standard data is a target format of the set data.

[0014] Optionally, before loading the adapter component corresponding to the mapping conversion rule from the pluggable adapter component library according to the data source type of the target data source, the method further comprises:

[0015] When it is determined that the current mapping conversion rule cannot uniformly configure the original data, an adapter component corresponding to the current mapping conversion rule and the data source type is determined to be generated;

[0016] The pluggable adapter component library is constructed based on all the adapter components.

[0017] Optionally, the adapter components in the pluggable adapter component library at least include a database adapter, an interface adapter and a file adapter.

[0018] Optionally, after processing the initial processing data based on the adapter component to obtain target extraction data, the method further comprises:

[0019] The target extraction data is sent to a downstream verification engine or a business system for use.

[0020] Optionally, the target data source is at least one of a MySQL database, an Oracle database, a hypertext transfer protocol interface and a file transfer protocol file.

[0021] The application further provides a data extraction device, comprising:

[0022] An initial processing data acquisition module is configured to acquire original data of a target data source and perform initial processing on the original data based on a mapping conversion rule to obtain initial processing data.

[0023] An adapter component loading module is configured to load an adapter component corresponding to the mapping conversion rule from a pluggable adapter component library according to a data source type of the target data source.

[0024] A target extraction data determination module is configured to process the initial processing data based on the adapter component to obtain target extraction data.

[0025] The application further provides a data extraction device, comprising:

[0026] a memory for storing a computer program;

[0027] a processor for executing the computer program to implement the steps of the data extraction method.

[0028] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the data extraction method.

[0029] The application further provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the data extraction method.

[0030] It can be seen that, by obtaining original data of a target data source, and performing initial processing on the original data based on a mapping conversion rule to obtain initial processing data, and loading an adapter component corresponding to the mapping conversion rule from a pluggable adapter component library according to a data source type of the target data source, and performing processing on the initial processing data based on the adapter component to obtain target extraction data, all data in the application is accessed and parsed through a unified process by using the mapping conversion rule and the adapter component, and standard and clean data is output, so that the quality and consistency of data consumed by a downstream system are ensured, and the efficiency and accuracy of data extraction are improved.

[0031] In addition, the application further provides a data extraction device, equipment and computer readable storage medium, which also have the beneficial effects described above. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute the embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0033] Figure 1 A flowchart of a data extraction method provided by the embodiment of the application;

[0034] Figure 2 A structural framework diagram of a data extraction method provided by the embodiment of the application;

[0035] Figure 3 A structural schematic diagram of a data extraction device provided by the embodiment of the application;

[0036] Figure 4This is a schematic diagram of the structure of a data extraction device provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Please refer to Figure 1 , Figure 1 A flowchart illustrating a data extraction method provided in an embodiment of the present invention. The method may include:

[0039] S101: Obtain the original data from the target data source and perform initial processing on the original data based on the mapping transformation rules to obtain the initial processed data.

[0040] Each step in this embodiment can be executed by a designated electronic device, which can be a server, a portable terminal, or other forms. The raw data in this embodiment is data obtained from the target data source. This embodiment does not limit the specific target data source; for example, the target data source in this embodiment can be a MySQL database (MySQL is the most popular relational database management system), an Oracle database (Oracle database system is currently the world's most popular relational database management system), a relational database such as PostgreSQL (PostgreSQL is a free object-relational database server), an HTTP API (Hypertext Transfer Protocol interface), or a CSV (comma-separated value) / Excel (spreadsheet) file on an SFTP (Secure File Transfer Protocol) file server. The goal of the mapping transformation rules in this embodiment is to transform the messy raw data into clean, uniform, and business-valuable standard data. Standard data is like a pre-defined, perfectly structured format, and all data must conform to its format and specifications. This embodiment does not limit the specific method for setting and issuing mapping transformation rules; for example, rules can be configured through dropdown selection, filling in expressions, or scripts. It should be noted that mapping transformation rules can include field mapping, data cleaning (such as removing spaces and handling invalid values), and type conversion (such as string to number, timestamp to date format), etc. It should be further noted that the target data source is at least one of the following: MySQL database, Oracle database, Hypertext Transfer Protocol interface, and File Transfer Protocol file.

[0041] It should be further explained that, based on any of the above embodiments, before obtaining the raw data from the target data source, the process may further include: obtaining data source connection parameters, wherein the data source connection parameters include data source type, host address, port, database name, username, password, interface address, request method, and file path; and establishing a connection with the target data source based on the data source connection parameters. After determining the data source connection parameters in this embodiment, the corresponding script information, such as SQL (Structured Query Language) / GROOVY (Data Processing Scripting Language), can be configured according to the selected data source type. "Configuring the corresponding script information" indicates that after successfully establishing the connection, the system needs to know what specific operation to perform to obtain or process the data. The data source connection information in this embodiment may include data source type, host address, port, database name, username, password (encrypted storage), API URL interface address, request method, file path, etc.

[0042] It should be further explained that, based on any of the above embodiments, before obtaining the original data from the target data source and performing initial processing on the original data based on the mapping transformation rules to obtain the initially processed data, the method may further include: defining mapping transformation rules from the original data to standard data through a graphical interface; wherein, the mapping transformation rules include at least one of field mapping, data cleaning, and type conversion, and the standard data is the set target format of the data. In this embodiment, the user defines the mapping transformation rules from the original data to the standard data model through a graphical interface. The graphical interface in this embodiment enables zero-coding configuration that can be performed by business personnel. This embodiment provides a user-friendly interface with a low technical threshold, and some configuration work can be completed by business personnel or data analysts, reducing the pressure on the development team.

[0043] It should be further explained that, in order to improve the efficiency of data access, before obtaining the raw data from the target data source, the following steps may also be included: the system automatically loads and instantiates the adapter (as an adapter component) corresponding to the data source from the pluggable adapter component library according to the configured data source type (such as database adapter, API adapter, file adapter). The adapter uses the connection parameters in the configuration to automatically establish a connection with the data source (such as creating a database connection pool, constructing an HTTP request, or logging into the file server), and executes the configured data extraction logic (such as executing SQL, calling API, or downloading files) to obtain the raw data.

[0044] S102, when it is determined that the initial processing data does not meet the extraction target, the adapter component corresponding to the mapping transformation rule is loaded from the pluggable adapter component library according to the data source type of the target data source.

[0045] The adapter component is used in this embodiment because different data sources have fundamental differences, such as differences in protocols, authentication mechanisms, and data acquisition methods. For data that can be processed uniformly, initial processing is performed using mapping transformation rules, and then further processing is performed using the adapter component.

[0046] The adapter is suitable for handling differences between data sources, making data processing and extraction more accurate and comprehensive. In this embodiment, the adapter component is an adapter developed according to a unified interface specification and registered in the component library. Once registered, it can be recognized and called by the system without modifying the core framework code. It is understandable that if the initial data meets the extraction target, the adapter component is not needed for processing.

[0047] It should be further noted that, based on any of the above embodiments, before loading the adapter component corresponding to the mapping transformation rule from the pluggable adapter component library according to the data source type of the target data source, the method may further include: when it is determined that the current mapping transformation rule cannot uniformly configure the original data, determining to generate an adapter component corresponding to the current mapping transformation rule and the data source type; and constructing the pluggable adapter component library based on all the adapter components. In this embodiment, if the mapping transformation rule corresponding to the current target data source cannot achieve mapping transformation, it is necessary to construct an adapter component for that target data source. The adapter components in the pluggable adapter component library in this embodiment include at least a database adapter, an interface adapter, and a file adapter.

[0048] S103, the initial processing data is processed based on the adapter component to obtain the target extraction data.

[0049] Each adapter in this embodiment is specifically designed to handle a particular type of data source. Understandably, adapter components come into play when simple mapping transformation rules fail. This is the core key to handling "different data formats." Different data sources have fundamental technical differences that cannot be resolved with simple mapping transformation rules. Adapter components act as "translators" specifically to solve this problem. For example, an SFTP file adapter parses binary streams of file formats such as CSV / Excel. The core goal of all adapter components is to convert diverse data into a unified data format within the system. When a new data source emerges, simply develop a new adapter according to the unified interface specification and register it in the component library; the system can then recognize and call it without modifying the core framework code.

[0050] It should be further noted that, based on any of the above embodiments, after processing the initial processing data using the adapter component to obtain the target extracted data, the method may further include: sending the target extracted data to the downstream verification engine or business system for use. In this embodiment, data corresponding to all data sources is extracted into unified target extracted data, ensuring the quality and consistency of data consumed by the downstream system (verification engine or business system).

[0051] This invention provides a data extraction method that may include: S101, acquiring raw data from a target data source and performing initial processing on the raw data based on mapping transformation rules to obtain initial processed data; S102, when it is determined that the initial processed data does not meet the extraction target, loading an adapter component corresponding to the mapping transformation rules from a pluggable adapter component library according to the data source type of the target data source; S103, processing the initial processed data based on the adapter component to obtain the target extracted data. Compared with the current method that requires developing data extraction code for each data source, this invention uses mapping transformation rules and adapter components to access and parse all data through a unified process, outputting standard and clean data, ensuring the quality and consistency of data consumed by downstream systems, and improving the efficiency and accuracy of data extraction. Furthermore, new data sources can be quickly accessed through configuration, reducing the response time to business changes from days to hours or even minutes. This avoids the workload of writing repetitive code for each data source, reducing technical complexity and manpower costs; when the data source structure changes, only the configuration needs to be modified.

[0052] When building data reconciliation, data middleware, or ETL (Extract, Transform, Load) systems, it is often necessary to extract data from numerous heterogeneous data sources. These data sources are diverse (such as relational databases like MySQL, Oracle, and PostgreSQL; HTTP API interfaces; CSV / Excel files on SFTP file servers; etc.), with significant differences in their data structures, protocols, and access methods. The traditional approach is to write dedicated data access and parsing code for each data source. This approach has significant drawbacks: 1. Low development efficiency: Each new data source requires the development, testing, and deployment of new code, resulting in long development cycles and high costs. 2. Difficult maintenance: When the data source structure changes or needs to be expanded, the code must be modified and the system redeployed, leading to poor maintainability. 3. Insufficient flexibility: It heavily relies on technical personnel, making it difficult for business personnel to respond quickly to changes, resulting in low system agility.

[0053] To facilitate understanding of this invention, embodiments of the invention provide a process that requires only the adapter component to acquire the raw data, without any subsequent data processing. Please refer to the following for details. Figure 2 , Figure 2A structural framework diagram of a data extraction method provided in an embodiment of the present invention may specifically include:

[0054] I. Configuration Definition Phase.

[0055] (1) Select data source type: In the unified configuration interface, users can select the type of data source to be connected (such as MySQL database, Oracle database, HTTP API hypertext transfer protocol interface, SFTP file transfer protocol file, etc.) through the drop-down menu.

[0056] (2) Fill in the data source connection parameters: Configure the corresponding script information according to the selected data source type, such as SQL / GROOVY, etc.

[0057] (3) Define field mapping and transformation rules: Users define the mapping and transformation rules from raw data to standard data model through the interface operation.

[0058] II. Automatic Execution and Parsing Phase:

[0059] (1) Automatic invocation of adapter components: The system automatically loads and instantiates the adapter (such as database adapter, API adapter, file adapter) corresponding to the parsing rules from the pluggable adapter component library according to the configured data source type.

[0060] (2) Connection and data extraction: The adapter uses the data source connection parameters in the configuration to automatically establish a connection with the data source (such as creating a database connection pool, constructing an HTTP request, or logging into the file server) and execute the configured data extraction logic (such as executing SQL, calling API, or downloading files) to obtain the raw data.

[0061] (3) Rule Engine Parsing: The rule engine automatically processes the extracted raw data according to the mapping and transformation rules defined by S. This includes field mapping, data cleaning (such as removing spaces and handling invalid values), and type conversion (such as converting strings to numbers and timestamps to date formats).

[0062] (4) Output standardized data: After parsing, the original heterogeneous data is converted into a unified standardized data model (such as a standard JSON object or ProtoBuf structure) within the system and sent to the downstream verification engine or business system for use.

[0063] This invention provides a configurable-based automatic access and parsing method for multi-source heterogeneous data. Its core lies in decoupling the data access and parsing logic from hard-coded implementation through a unified configuration center and a pluggable adapter framework, transforming it into a dynamically driven execution based on configuration information. This system is implemented through the collaborative work of the following key modules:

[0064] (1) Provides a graphical user interface (GUI) for centralized management of configuration information for all data sources. It is the "brain" of the entire approach, storing and managing all metadata.

[0065] (2) Data source connection configuration: including data source type, host address, port, database name, username, password (encrypted storage), API URL interface address, request method, file path, etc.

[0066] Same.

[0067] (3) Data extraction configuration: SQL query statement (supports variable substitution, such as ${biz_date}) used to specify the extraction range, API request parameters, and file name pattern to be collected (supports regular expression matching).

[0068] 4. Data Parsing and Mapping Rules: Define the mapping relationship between raw data and the standard model, data cleaning rules (such as removing spaces and filling default values), and format conversion rules (such as converting strings to dates and standardizing currency units). These rules can be configured through drop-down selections, inputting expressions, or scripts.

[0069] Simple rules, common rules.

[0070] (4) Pluggable Adapter Component Library: A repository of pre-built adapters, each designed to handle a specific type of data source. All adapters implement a unified interface standard.

[0071] The following describes a data extraction device provided by an embodiment of the present invention. The data extraction device described below can be referred to in correspondence with the data extraction method described above.

[0072] Please refer to the details. Figure 3 , Figure 3 A schematic diagram of a data extraction device provided in an embodiment of the present invention may include:

[0073] The initial processing data acquisition module 100 is used to acquire the original data from the target data source and perform initial processing on the original data based on the mapping transformation rules to obtain the initial processing data.

[0074] The adapter component loading module 200 is used to load an adapter component corresponding to the mapping transformation rule from the pluggable adapter component library according to the data source type of the target data source when it is determined that the initial processing data does not meet the extraction target.

[0075] The target extraction data determination module 300 is used to process the initial processing data based on the adapter component to obtain target extraction data.

[0076] Furthermore, based on any of the above embodiments, the data extraction device may further include:

[0077] The data source connection parameter determination module is used to obtain data source connection parameters, wherein the data source connection parameters include data source type, host address, port, database name, username, password, interface address, request method, and file path;

[0078] A connection establishment module for the target data source is used to establish a connection with the target data source based on the connection parameters of the data source.

[0079] Furthermore, based on any of the above embodiments, the data extraction device may further include:

[0080] The mapping and transformation rule configuration module is used to define the mapping and transformation rules from raw data to standard data through a graphical interface; wherein, the mapping and transformation rules include at least one of field mapping, data cleaning and type conversion, and the standard data is the target format of the set data.

[0081] Furthermore, based on any of the above embodiments, the data extraction device may further include:

[0082] The adapter component generation module is used to determine the generation of an adapter component corresponding to the current mapping transformation rule and the data source type when it is determined that the current mapping transformation rule cannot uniformly configure the original data.

[0083] A pluggable adapter component library building module is used to build the pluggable adapter component library based on all adapter components.

[0084] Furthermore, based on any of the above embodiments, the adapter components in the pluggable adapter component library include at least a database adapter, an interface adapter, and a file adapter.

[0085] Furthermore, based on any of the above embodiments, the data extraction device may further include:

[0086] The sending module is used to send the target extracted data to the downstream verification engine or business system.

[0087] Furthermore, based on any of the above embodiments, the target data source is at least one of a MySQL database, an Oracle database, a Hypertext Transfer Protocol interface, and a File Transfer Protocol file.

[0088] It should be noted that the order of the modules and units in the above data extraction device can be changed without affecting the logic.

[0089] This invention provides a data extraction device, which may include: an initial processing data acquisition module 100, used to acquire raw data from a target data source and perform initial processing on the raw data based on mapping transformation rules to obtain initial processed data; an adapter component loading module 200, used to load an adapter component corresponding to the mapping transformation rules from a pluggable adapter component library according to the data source type of the target data source when it is determined that the initial processed data does not meet the extraction target; and a target extraction data determination module 300, used to process the initial processed data based on the adapter component to obtain target extraction data. Compared with the current method of developing data extraction code for each data source, this invention uses mapping transformation rules and adapter components to access and parse all data through a unified process, outputting standard and clean data, ensuring the quality and consistency of data consumed by downstream systems, and improving the efficiency and accuracy of data extraction.

[0090] The following describes a data extraction device provided by an embodiment of the present invention. The data extraction device described below can be referred to in correspondence with the data extraction method described above.

[0091] Please refer to Figure 4 , Figure 4 A schematic diagram of a data extraction device provided in an embodiment of the present invention may include:

[0092] Memory 10 is used to store computer programs;

[0093] Processor 20 is used to execute computer programs to implement the data extraction method described above.

[0094] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.

[0095] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:

[0096] Obtain the raw data from the target data source and perform initial processing on the raw data based on the mapping transformation rules to obtain the initial processed data;

[0097] When it is determined that the initial processing data does not meet the extraction target, the adapter component corresponding to the mapping transformation rule is loaded from the pluggable adapter component library according to the data source type of the target data source.

[0098] The initial data is processed using the adapter component to obtain the target extracted data.

[0099] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0100] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0101] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0102] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.

[0103] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the data extraction device in the embodiments of the present invention. In practical applications, the data extraction device may include more than Figure 4 More or fewer components as shown, or combinations of certain components.

[0104] The computer-readable storage medium provided in the embodiments of the present invention is described below. The computer-readable storage medium described below and the data extraction method described above can be referred to in correspondence.

[0105] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data extraction method.

[0106] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0108] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0109] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0110] The above provides a detailed description of a data extraction method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data extraction method, characterized by, The method comprises the following steps: obtaining original data of a target data source, and performing initial processing on the original data based on a mapping conversion rule to obtain initial processing data; when it is determined that the initial processing data does not meet an extraction target, loading an adapter component corresponding to the mapping conversion rule from a pluggable adapter component library according to a data source type of the target data source; performing processing on the initial processing data based on the adapter component to obtain target extraction data.

2. The data extraction method of claim 1, wherein, Before obtaining the original data of the target data source, the method further comprises the following steps: obtaining data source connection parameters, wherein the data source connection parameters comprise a data source type, a host address, a port, a database name, a username, a password, an interface address, a request method and a file path; establishing a connection with the target data source based on the data source connection parameters.

3. The data extraction method of claim 1, wherein, Before obtaining the original data of the target data source and performing initial processing on the original data based on a mapping conversion rule to obtain initial processing data, the method further comprises the following steps: defining the mapping conversion rule from original data to standard data through interface operation, wherein the mapping conversion rule comprises at least one of field mapping, data cleaning and type conversion, and the standard data is a target format of set data.

4. The data extraction method according to any one of claims 1 to 3, characterized in that, Before when it is determined that the initial processing data does not meet an extraction target, loading an adapter component corresponding to the mapping conversion rule from a pluggable adapter component library according to a data source type of the target data source, the method further comprises the following steps: when it is determined that a current mapping conversion rule cannot perform unified configuration on original data, determining to generate an adapter component corresponding to the current mapping conversion rule and a data source type; constructing the pluggable adapter component library based on all adapter components.

5. The data extraction method of claim 4, wherein, The adapter components in the pluggable adapter component library at least comprise a database adapter, an interface adapter and a file adapter.

6. The data extraction method of claim 1, wherein, After performing processing on the initial processing data based on the adapter component to obtain target extraction data, the method further comprises the following steps: sending the target extraction data to a downstream verification engine or a business system for use.

7. The data extraction method of claim 1, wherein, The target data source is at least one of a MySQL database, an Oracle database, a hypertext transfer protocol interface and a file transfer protocol file.

8. A data extraction apparatus characterized by comprising: The method comprises the following steps: an initial processing data obtaining module, configured to obtain original data of a target data source, and perform initial processing on the original data based on a mapping conversion rule to obtain initial processing data; an adapter component loading module, configured to, when it is determined that the initial processing data does not meet an extraction target, load an adapter component corresponding to the mapping conversion rule from a pluggable adapter component library according to a data source type of the target data source; a target extraction data determining module, configured to perform processing on the initial processing data based on the adapter component to obtain target extraction data.

9. A data extraction device, characterized by, The method comprises the following steps: a memory, configured to store a computer program; a processor, configured to execute the computer program to implement the steps of the data extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the data extraction method in any one of claims 1 to 7.

Images