Data extraction method, device and electronic equipment

Abstract

The present application discloses a data extraction method, device and electronic device, which can be applied to the field of big data or the financial field. Parse the data extraction script to obtain the data extraction field and the data extraction rule, obtain the correspondence between the field and the field flow operation process information, query and obtain the field flow operation process information corresponding to the data extraction field from the correspondence, and perform data flow operation processing on the data extraction field in the data extraction script to obtain the target script, run the target script to extract the field extraction result corresponding to the data extraction field in the data lake, and perform data processing operation on the field extraction result according to the data extraction rule. That is, through the present invention, historical business data can be automatically extracted from the data lake, and during the extraction process, the non-standard data extraction field in the data extraction script will be processed by data flow operation to obtain a standardized data extraction field, thereby ensuring the accuracy of data extraction.

Description

Technical Field

[0001] The present invention relates to the field of data extraction, and more specifically, to a data extraction method, device and electronic equipment. Background Art

[0002] When evaluating the performance of a product, it is necessary to obtain more historical business data and perform performance evaluation based on the historical business data. Currently, the business system can only store business data for a short period of time. Therefore, when obtaining historical business data for performance evaluation, it is necessary to extract it from the data lake that stores historical business data according to data extraction requirements.

[0003] Currently, when extracting historical business data from a data lake, data extraction requirements are manually analyzed, and the corresponding historical business data is queried and extracted. This data extraction method has low data extraction accuracy. Summary of the invention

[0004] In view of this, the present invention provides a data extraction method, device and electronic device to solve the problem of low data extraction accuracy in existing data extraction methods.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A data extraction method, comprising:

[0007] Get the data extraction script;

[0008] Parsing the data extraction script to obtain data extraction fields and data extraction rules in the data extraction script;

[0009] Obtaining a correspondence between a pre-generated field and field flow operation process information, wherein the field flow operation process information is data flow operation process information stored from the field to the data lake;

[0010] From the corresponding relationship, query and obtain the field transfer operation process information corresponding to the data extraction field;

[0011] According to the field flow operation process information, the data extraction field in the data extraction script is processed by the data flow operation to obtain the target script;

[0012] The target script is run to extract a field extraction result corresponding to the data extraction field in the data lake, and a data processing operation is performed on the field extraction result according to the data extraction rule.

[0013] Optionally, the process of generating the corresponding relationship includes:

[0014] Get the fields in the visualization interface;

[0015] Determine data flow operation process information stored in the field to the data lake, wherein the data flow operation process information includes a plurality of field flow operation steps, and the field flow operation steps include a data storage location and a data flow operation in the data flow process;

[0016] Generates the correspondence between fields and field transfer operation process information.

[0017] Optionally, determining the data flow operation process information stored in the field to the data lake includes:

[0018] Parse the framework diagram, interface documents and database design documents of the business system to parse out the data flow operation process information stored in the fields to the data lake.

[0019] Optionally, determining the data flow operation process information stored in the field to the data lake includes:

[0020] The data flow operation process information of the input field storage to the data lake is obtained.

[0021] Optionally, according to the field flow operation process information, performing data flow operation processing on the data extraction field in the data extraction script to obtain a target script includes:

[0022] Execute each field transfer operation step in sequence according to the execution order of the field transfer operation steps in the field transfer operation process information, so as to perform field transfer operation on the data extraction field in the data extraction script, and obtain a transfer field corresponding to the data extraction field;

[0023] The target script can be obtained by replacing the data extraction field in the data extraction script with the flow field.

[0024] A data extraction device, comprising:

[0025] A script acquisition module is used to obtain data extraction scripts;

[0026] A script parsing module, used to parse the data extraction script to obtain data extraction fields and data extraction rules in the data extraction script;

[0027] A relationship acquisition module, used to acquire the correspondence between the pre-generated fields and the field flow operation process information, wherein the field flow operation process information is the data flow operation process information stored from the field to the data lake;

[0028] An information query module, used to query and obtain field transfer operation process information corresponding to the data extraction field from the corresponding relationship;

[0029] A script processing module, used for performing data flow operation processing on the data extraction fields in the data extraction script according to the field flow operation process information to obtain a target script;

[0030] The script running module is used to run the target script to extract the field extraction result corresponding to the data extraction field in the data lake, and perform data processing operations on the field extraction result according to the data extraction rule.

[0031] Optionally, a relationship generation module is further included, and the relationship generation module includes:

[0032] The field acquisition submodule is used to obtain the fields in the visualization interface;

[0033] An information determination submodule, used to determine the data flow operation process information stored in the field to the data lake, wherein the data flow operation process information includes a plurality of field flow operation steps, and the field flow operation steps include a data storage location and a data flow operation in the data flow process;

[0034] The relationship generation submodule is used to generate the corresponding relationship between fields and field flow operation process information.

[0035] Optionally, the information determination submodule is specifically used for:

[0036] Parse the framework diagram, interface documents and database design documents of the business system to parse out the data flow operation process information stored in the fields to the data lake.

[0037] Optionally, the information determination submodule is specifically used for:

[0038] The data flow operation process information of the input field storage to the data lake is obtained.

[0039] An electronic device comprising: a memory and a processor;

[0040] Wherein, the memory is used to store programs;

[0041] The processor calls the program and is used to execute the above data extraction method.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention provides a data extraction method, device and electronic device, which obtains a data extraction script, parses the data extraction script, obtains data extraction fields and data extraction rules in the data extraction script, obtains the correspondence between pre-generated fields and field flow operation process information, the field flow operation process information is the data flow operation process information of the field stored in the data lake, and from the correspondence, the field flow operation process information corresponding to the data extraction field is queried, and according to the field flow operation process information, the data extraction field in the data extraction script is processed by data flow operation to obtain a target script, and the target script is run to extract the field extraction result corresponding to the data extraction field in the data lake, and the field extraction result is processed by data processing according to the data extraction rule. That is, through the present invention, historical business data can be automatically extracted from the data lake, and during the extraction process, the non-standard data extraction fields in the data extraction script will be processed by data flow operation to obtain standardized data extraction fields, thereby ensuring the accuracy of data extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A method flow chart of a data extraction method provided by an embodiment of the present invention;

[0046] Figure 2 A method flow chart of another data extraction method provided by an embodiment of the present invention;

[0047] Figure 3 A method flow chart of another data extraction method provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a multi-table association scenario provided by an embodiment of the present invention;

[0049] Figure 5 A structural schematic diagram of a data extraction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] When evaluating the performance of a product, it is necessary to obtain more historical business data and perform performance evaluation based on the historical business data. Currently, the business system can only store business data for a short period of time. Therefore, when obtaining historical business data for performance evaluation, it is necessary to extract it from the data lake that stores historical business data according to data extraction requirements.

[0052] Currently, when extracting historical business data from a data lake, data extraction requirements are manually analyzed, and the corresponding historical business data is queried and extracted. This data extraction method has low data extraction accuracy. In addition, when manually analyzing data extraction requirements, business personnel and data lake operation and maintenance personnel need to communicate and try and error many times, which is complicated.

[0053] In order to solve this technical problem, the present invention provides a data extraction method, device and electronic device, which obtains a data extraction script, parses the data extraction script, obtains the data extraction field and data extraction rules in the data extraction script, obtains the correspondence between the pre-generated field and the field flow operation process information, the field flow operation process information is the data flow operation process information of the field stored in the data lake, and from the correspondence, the field flow operation process information corresponding to the data extraction field is queried, and according to the field flow operation process information, the data extraction field in the data extraction script is processed by data flow operation to obtain a target script, and the target script is run to extract the field extraction result corresponding to the data extraction field in the data lake, and the field extraction result is processed by data processing according to the data extraction rule. That is, through the present invention, historical business data can be automatically extracted from the data lake, and during the extraction process, the non-standard data extraction field in the data extraction script will be processed by data flow operation to obtain a standardized data extraction field, thereby ensuring the accuracy of data extraction.

[0054] It should be noted that the data extraction method, device and electronic device provided by the present invention can be used in the field of big data or finance. The above is only an example and does not limit the application field of the data extraction method, device and electronic device provided by the present invention.

[0055] Based on the above, refer to Figure 1 , the data extraction method in the present invention may include:

[0056] S11. Obtain data extraction script.

[0057] The data extraction script in this embodiment may be a script written by a business person to extract data from a data lake.

[0058] S12: Parse the data extraction script to obtain data extraction fields and data extraction rules in the data extraction script.

[0059] The data extraction script includes data extraction fields and data extraction rules.

[0060] For example, if the content of the data extraction script is to extract data of "transfer amount greater than 100", the data extraction field may be the transfer amount, and the data extraction rule may be "greater than 100".

[0061] It should be noted that a standardized data extraction field is pre-configured. In this embodiment, the data extraction field can be obtained by simply matching the data extraction script with the standardized data extraction field, and the rest is the data extraction rule.

[0062] In actual applications, if the data extraction fields in the data extraction script are not standardized, the non-standardized data extraction fields can be standardized first, such as performing a standardization operation on the non-standardized data extraction fields based on the correspondence between the standardized fields and the non-standardized fields.

[0063] S13. Obtain a correspondence between pre-generated fields and field transfer operation process information, where the field transfer operation process information is data transfer operation process information stored from the field to the data lake.

[0064] Specifically, in this embodiment, a correspondence between fields and field flow operation process information is pre-generated, and the field flow operation process information is data flow operation process information stored from the fields to the data lake.

[0065] For example: field A in the business system --> system S1 (table T1, new field name S1A, operation Op1 such as type conversion, mathematical calculation, etc.) --> system S2 (table T2, new field name S2A, operation Op2)... --> data lake (table Tn, field name SnA, operation Opn).

[0066] Field A can be a date, such as 2022 / 1 / 1. It is first transferred to table T1 of system S1, with the new field name S1A being 2020-1-1, and operation Op1 being format conversion. It is then transferred to table T2 of system S2, with the new field name S1A being 2020.1.1, and operation Op2 being format conversion. Finally, it is transferred to table Tn of the data lake, with the field name SnA being 20200101, and operation Opn being format conversion. In other words, the date 2022 / 1 / 1 is ultimately stored in the format of 20200101.

[0067] In addition, if field A is a number, such as 20, the addition operation is performed when it flows through each system, and the number of times it flows through the system is 4 times. Finally, the addition operation is performed when it reaches the data lake. Finally, the data stored in the data lake is 25.

[0068] The field flow operation process information in this embodiment can also be called the blood relationship of the field. Each field is sorted to obtain the corresponding blood relationship, and all blood relationships constitute a blood network.

[0069] In practical applications, refer to Figure 2 , the generation process of the corresponding relationship may include:

[0070] S21. Get fields in the visualization interface.

[0071] In this embodiment, the fields involved in the data extraction script provided by the business personnel often come from the operation interface of the business system, such as the transfer amount statistics. For mobile banking, it can be locked to the [Transfer Amount] field of the transfer and remittance interface. For the business system, all visual interface information is collected, and fields in the visual interface, such as transfer amount, transfer time, remittance amount, etc., are extracted. Then, a blood relationship process is established for each field, recording the various systems and corresponding operations that the field passes through, and finally to the data lake.

[0072] S22. Determine the data flow operation process information stored in the field to the data lake.

[0073] The data flow operation process information includes multiple field flow operation steps, and the field flow operation steps include the data storage location and data flow operation of the data flow process.

[0074] Among them, multiple field transfer operation steps can be field A of the above-mentioned business system-->system S1 (table T1, new field name S1A, operation Op1 such as type conversion, mathematical calculation, etc.)-->system S2 (table T2, new field name S2A, operation Op2)...-->data lake (table Tn, field name SnA, operation Opn).

[0075] From field A in the business system --> system S1, this is a field transfer operation step; system S1 (table T1, new field name S1A, operation Op1 such as type conversion, mathematical calculation, etc.) --> system S2 (table T2, new field name S2A, operation Op2) is a field transfer operation step, and so on, until it finally reaches the data lake.

[0076] The field transfer operation step includes the data storage location and data transfer operation of the data transfer process. The data storage location can be a new field name S1A in table T1 in system S1, and the data transfer operation can be operation Op1, operation Op2, etc.

[0077] In practical applications, there are two ways to determine the data flow operation process information stored in the field to the data lake. They are introduced below.

[0078] 1. The first implementation method:

[0079] Parse the framework diagram, interface documents and database design documents of the business system to parse out the data flow operation process information stored in the fields to the data lake.

[0080] This step is an automatic process. The framework diagram, interface documents, and database design documents of the business system describe the format of the fields in each storage location and the data flow process. Therefore, by directly parsing the framework diagram, interface documents, and database design documents of the business system, you can analyze the data flow operation process information of the fields stored in the data lake based on the data flow process and the format of the fields in each storage location.

[0081] 2. The second implementation method:

[0082] The data flow operation process information of the input field storage to the data lake is obtained.

[0083] In this step, the data flow operation process information stored in the field to the data lake can be manually sorted out, and then the data flow operation process information stored in the field to the data lake can be output by writing code or inputting into a visual interface.

[0084] S23. Generate a correspondence between fields and field transfer operation process information.

[0085] In this step, after the field transfer operation process information corresponding to each field is determined, the corresponding relationship between the field and the field transfer operation process information can be generated.

[0086] S14. From the corresponding relationship, query and obtain the field transfer operation process information corresponding to the data extraction field.

[0087] In this embodiment, by querying the corresponding relationship, the field transfer operation process information corresponding to the data extraction field can be obtained.

[0088] S15. According to the field flow operation process information, perform data flow operation processing on the data extraction fields in the data extraction script to obtain a target script.

[0089] Specifically, refer to Figure 3 , step S15 may include:

[0090] S31. Execute each field transfer operation step in sequence according to the execution order of the field transfer operation steps in the field transfer operation process information, so as to perform field transfer operation on the data extraction field in the data extraction script and obtain the transfer field corresponding to the data extraction field.

[0091] Taking the above field A as an example, according to the steps of field A of the business system --> system S1 (table T1, new field name S1A, operation Op1 such as type conversion, mathematical calculation, etc.) --> system S2 (table T2, new field name S2A, operation Op2)... --> data lake (table Tn, field name SnA, operation Opn), the data flow operation is processed to obtain the flow field corresponding to the data extraction field, such as field A corresponding to the new field name S2A.

[0092] It should be noted that in addition to determining the transfer field corresponding to the data extraction field by superimposing the above operations (executing each field transfer operation step in sequence), you can also directly analyze the field transfer operation process information to establish the correspondence between the field A of the business system and the data lake (table Tn, field name SnA, operation Opn), thereby directly determining the transfer field corresponding to the data extraction field based on the correspondence. However, generally speaking, if you encounter a scenario where the field transfer operation steps are changed, this solution is not applicable, and you can still process it according to the field transfer operation steps in the above field transfer operation process information.

[0093] S32. Replace the data extraction field in the data extraction script with the flow field to obtain the target script.

[0094] In this embodiment, the target script can be obtained by performing field replacement.

[0095] S16. Run the target script to extract a field extraction result corresponding to the data extraction field in the data lake, and perform a data processing operation on the field extraction result according to the data extraction rule.

[0096] Specifically, after the target script is determined, the target script can be run to extract the field extraction result corresponding to the data extraction field in the data lake, and perform data processing operations on the field extraction result according to the data extraction rule.

[0097] It should be noted that there may be a many-to-many relationship between the business fields in the bloodline network and the fields in the data lake. In this case, after the field replacement in step S32, the same field existing in multiple tables can be used as the primary key for associating and merging multiple tables. All fields involved in multiple tables are summarized to generate the target script. Figure 4 , Figure 4 The intermediate system is omitted. The business rules involve fields A, B, and C, which are traced back to tables T1, T2, and T3 in the data lake. If field B exists in three tables, B is automatically used for multi-table association. The target script is select t1.F11, t2.F22, t3.F22 from T1 t1 left join T2 t2 on t1.F12=t2.F22 left join T3 t3 on t1.F12=t3.F32.

[0098] In this embodiment, a data extraction script is obtained, the data extraction script is parsed, the data extraction fields and data extraction rules in the data extraction script are obtained, the correspondence between the pre-generated fields and the field flow operation process information is obtained, the field flow operation process information is the data flow operation process information of the fields stored in the data lake, the field flow operation process information corresponding to the data extraction field is queried from the correspondence, the data extraction field in the data extraction script is processed for data flow operation according to the field flow operation process information, the target script is obtained, the target script is run to extract the field extraction result corresponding to the data extraction field in the data lake, and the field extraction result is processed for data processing according to the data extraction rules. That is, through the present invention, historical business data can be automatically extracted from the data lake, and during the extraction process, the non-standard data extraction fields in the data extraction script will be processed for data flow operation to obtain standardized data extraction fields, thereby ensuring the accuracy of data extraction.

[0099] In addition, in the present invention, by combing the complete lineage relationship of fields from the business system to the data lake, a complete field lineage network of the bank is constructed, and with the help of tracing the lineage relationship, the data extraction script is automatically converted into a target script that can extract data in the data lake, thereby greatly shortening the data extraction task cycle and improving the accuracy and efficiency of data extraction.

[0100] Optionally, based on the above data extraction method embodiment, another embodiment of the present invention provides a data extraction device, referring to Figure 5 , which may include:

[0101] A script acquisition module 11 is used to acquire a data extraction script;

[0102] A script parsing module 12, used to parse the data extraction script to obtain data extraction fields and data extraction rules in the data extraction script;

[0103] A relationship acquisition module 13 is used to acquire a correspondence between a pre-generated field and field flow operation process information, wherein the field flow operation process information is data flow operation process information stored from the field to the data lake;

[0104] An information query module 14 is used to query and obtain field transfer operation process information corresponding to the data extraction field from the corresponding relationship;

[0105] A script processing module 15, configured to perform data flow operation processing on the data extraction fields in the data extraction script according to the field flow operation process information to obtain a target script;

[0106] The script running module 16 is used to run the target script to extract the field extraction result corresponding to the data extraction field in the data lake, and perform data processing operations on the field extraction result according to the data extraction rule.

[0107] Furthermore, a relationship generation module is included, and the relationship generation module includes:

[0108] The field acquisition submodule is used to obtain the fields in the visualization interface;

[0109] An information determination submodule, used to determine the data flow operation process information stored in the field to the data lake, wherein the data flow operation process information includes a plurality of field flow operation steps, and the field flow operation steps include a data storage location and a data flow operation in the data flow process;

[0110] The relationship generation submodule is used to generate the corresponding relationship between fields and field flow operation process information.

[0111] Furthermore, the information determination submodule is specifically used for:

[0112] Parse the framework diagram, interface documents and database design documents of the business system to parse out the data flow operation process information stored in the fields to the data lake.

[0113] Furthermore, the information determination submodule is specifically used for:

[0114] The data flow operation process information of the input field storage to the data lake is obtained.

[0115] Further, the script processing module 15 includes:

[0116] A field determination submodule, for executing each field transfer operation step in sequence according to the execution order of the field transfer operation steps in the field transfer operation process information, so as to perform a field transfer operation on the data extraction field in the data extraction script, and obtain a transfer field corresponding to the data extraction field;

[0117] The field replacement submodule is used to replace the data extraction field in the data extraction script with the flow field to obtain the target script.

[0118] In this embodiment, a data extraction script is obtained, the data extraction script is parsed, the data extraction fields and data extraction rules in the data extraction script are obtained, the correspondence between the pre-generated fields and the field flow operation process information is obtained, the field flow operation process information is the data flow operation process information of the fields stored in the data lake, the field flow operation process information corresponding to the data extraction field is queried from the correspondence, the data extraction field in the data extraction script is processed for data flow operation according to the field flow operation process information, the target script is obtained, the target script is run to extract the field extraction result corresponding to the data extraction field in the data lake, and the field extraction result is processed for data processing according to the data extraction rules. That is, through the present invention, historical business data can be automatically extracted from the data lake, and during the extraction process, the non-standard data extraction fields in the data extraction script will be processed for data flow operation to obtain standardized data extraction fields, thereby ensuring the accuracy of data extraction.

[0119] In addition, in the present invention, by combing the complete lineage relationship of fields from the business system to the data lake, a complete field lineage network of the bank is constructed, and with the help of tracing the lineage relationship, the data extraction script is automatically converted into a target script that can extract data in the data lake, thereby greatly shortening the data extraction task cycle and improving the accuracy and efficiency of data extraction.

[0120] It should be noted that, for the working process of each module and sub-module in this embodiment, please refer to the corresponding description in the above embodiment, which will not be repeated here.

[0121] Optionally, based on the above data extraction method and device embodiments, another embodiment of the present invention provides an electronic device, including: a memory and a processor;

[0122] Wherein, the memory is used to store programs;

[0123] The processor calls the program and is used to execute the above data extraction method.

[0124] In this embodiment, a data extraction script is obtained, the data extraction script is parsed, the data extraction fields and data extraction rules in the data extraction script are obtained, the correspondence between the pre-generated fields and the field flow operation process information is obtained, the field flow operation process information is the data flow operation process information of the fields stored in the data lake, the field flow operation process information corresponding to the data extraction field is queried from the correspondence, the data extraction field in the data extraction script is processed for data flow operation according to the field flow operation process information, the target script is obtained, the target script is run to extract the field extraction result corresponding to the data extraction field in the data lake, and the field extraction result is processed for data processing according to the data extraction rules. That is, through the present invention, historical business data can be automatically extracted from the data lake, and during the extraction process, the non-standard data extraction fields in the data extraction script will be processed for data flow operation to obtain standardized data extraction fields, thereby ensuring the accuracy of data extraction.

[0125] In addition, in the present invention, by combing the complete lineage relationship of fields from the business system to the data lake, a complete field lineage network of the bank is constructed, and with the help of tracing the lineage relationship, the data extraction script is automatically converted into a target script that can extract data in the data lake, thereby greatly shortening the data extraction task cycle and improving the accuracy and efficiency of data extraction.

[0126] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data extraction method, characterized in that: include: Get the data extraction script; Parsing the data extraction script to obtain data extraction fields and data extraction rules in the data extraction script; Obtaining a correspondence between a pre-generated field and field flow operation process information, wherein the field flow operation process information is data flow operation process information stored from the field to the data lake; From the corresponding relationship, query and obtain the field transfer operation process information corresponding to the data extraction field; According to the field flow operation process information, the data extraction field in the data extraction script is processed by the data flow operation to obtain the target script; Running the target script to extract a field extraction result corresponding to the data extraction field in the data lake, and performing a data processing operation on the field extraction result according to the data extraction rule; The generation process of the corresponding relationship includes: Get the fields in the visualization interface; Determine data flow operation process information stored in the field to the data lake, wherein the data flow operation process information includes a plurality of field flow operation steps, and the field flow operation steps include a data storage location and a data flow operation in the data flow process; Generate the correspondence between fields and field transfer operation process information; According to the field flow operation process information, the data extraction field in the data extraction script is subjected to data flow operation processing to obtain a target script, including: Execute each field transfer operation step in sequence according to the execution order of the field transfer operation steps in the field transfer operation process information, so as to perform field transfer operation on the data extraction field in the data extraction script, and obtain a transfer field corresponding to the data extraction field; The data extraction field in the data extraction script is replaced with the flow field. For the same field existing in multiple tables, the field is used as the primary key for associating and merging multiple tables. After summarizing all the fields involved in multiple tables, a target script is generated.

2. The data extraction method according to claim 1, characterized in that: Determine the data flow operation process information of the field stored in the data lake, including: Parse the framework diagram, interface documents and database design documents of the business system to parse out the data flow operation process information stored in the fields to the data lake.

3. The data extraction method according to claim 1, characterized in that: Determine the data flow operation process information of the field stored in the data lake, including: The data flow operation process information of the input field storage to the data lake is obtained.

4. A data extraction device, characterized in that: include: A script acquisition module is used to obtain data extraction scripts; A script parsing module, used to parse the data extraction script to obtain data extraction fields and data extraction rules in the data extraction script; A relationship acquisition module, used to acquire the correspondence between the pre-generated fields and the field flow operation process information, wherein the field flow operation process information is the data flow operation process information stored from the field to the data lake; An information query module, used to query and obtain field transfer operation process information corresponding to the data extraction field from the corresponding relationship; A script processing module, used for performing data flow operation processing on the data extraction fields in the data extraction script according to the field flow operation process information to obtain a target script; A script running module, used to run the target script to extract a field extraction result corresponding to the data extraction field in the data lake, and perform a data processing operation on the field extraction result according to the data extraction rule; Wherein, it also includes a relationship generation module, and the relationship generation module includes: The field acquisition submodule is used to obtain the fields in the visualization interface; An information determination submodule, used to determine the data flow operation process information stored in the field to the data lake, wherein the data flow operation process information includes a plurality of field flow operation steps, and the field flow operation steps include a data storage location and a data flow operation in the data flow process; The relationship generation submodule is used to generate the corresponding relationship between fields and field transfer operation process information; According to the field flow operation process information, the data extraction field in the data extraction script is subjected to data flow operation processing to obtain a target script, including: Execute each field transfer operation step in sequence according to the execution order of the field transfer operation steps in the field transfer operation process information, so as to perform field transfer operation on the data extraction field in the data extraction script, and obtain a transfer field corresponding to the data extraction field; The data extraction field in the data extraction script is replaced with the flow field. For the same field existing in multiple tables, the field is used as the primary key for associating and merging multiple tables. After summarizing all the fields involved in multiple tables, a target script is generated.

5. The data extraction device according to claim 4, characterized in that: The information determination submodule is specifically used for: Parse the framework diagram, interface documents and database design documents of the business system to parse out the data flow operation process information stored in the fields to the data lake.

6. The data extraction device according to claim 4, characterized in that: The information determination submodule is specifically used for: The data flow operation process information of the input field storage to the data lake is obtained.

7. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor calls the program and is used to execute the data extraction method according to any one of claims 1 to 3.

Images