Method and system for unloading data from configured cloud data lake to application database applied to bank

Abstract

The invention relates to a method and a system for unloading data from a configured cloud data lake to an application database applied to a bank. The method comprises the following steps: importing source data of a source system into a data lake platform for preprocessing to form a data table; configuring corresponding unloading information for the data table, unloading the data table into a data file, and generating a corresponding verification file; transmitting the data file and the verification file to a specified client through a file transmission platform, querying corresponding unloading strategy information in a strategy table by a unified unloading service component according to a data table name in the verification file, and carrying out database connection; and the unified data unloading service component reads the verification file, analyzes the data file, converts the data file into a data format matched with the database and then executes database insertion operation. Therefore, the data in the data lake can be automatically identified and converted, the data can be efficiently and flexibly migrated to the application database of the source system, the complexity of data unloading is reduced, and the flexibility and adaptability of data processing are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, in particular to the financial industry, and more particularly to a method and system for unloading data from a configured cloud data lake to an application database for use in a bank. Background Art

[0002] With the acceleration of the digital transformation of the financial industry, the amount of data within banks has increased dramatically. As an efficient data storage and processing platform, cloud data lakes are widely used in various banking business systems. With its trillion-level computing power, data lakes have greatly reduced the data computing pressure of various source systems. By integrating the data of various source systems, data barriers between systems have been broken down, giving full play to the maximum value of data.

[0003] For example, the Chinese invention patent publication CN115185608A discloses a visual data unloading method and system based on batch process service, including: configuring a data unloading control table, a server information table and an FTP server information table for data unloading in the database table; using Java and SQLULDR2 to build a common component to implement the data unloading function in the batch service application; registering the data in the data unloading control table to generate the unloading process of each table in the data unloading control table into a data unloading process; configuring a data unloading process group based on the generated data unloading process, associating all processes with the process group, and displaying the data unloading process group to perform unloading and log viewing through the interface. However, this method does not have the functions of automatic identification and conversion, is not adaptable to data migration, and cannot flexibly unload the data in the data lake directly to the application database of the source system to meet the source system's ability to provide data statistical analysis in multiple dimensions and different perspectives.

[0004] Therefore, the present invention proposes a configurable cloud data lake to application database data unloading method and system for banks, aiming to automatically identify and transform data in the data lake, and realize efficient and flexible migration of data to the application database of the source system, thereby reducing the complexity of data unloading and improving the flexibility and adaptability of data processing, so that banks can make full use of the powerful computing power and data storage capabilities of the data lake, while meeting the source system's needs for data statistical analysis in multiple dimensions and from different perspectives. Summary of the invention

[0005] In order to solve the deficiencies of the prior art, the present invention proposes a method and system for unloading data from a configurable cloud data lake to an application database for use in a bank.

[0006] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0007] A configurable cloud data lake to application database data unloading method applied to a bank, characterized by comprising:

[0008] S1. Import the source data of the source system into the data lake platform, and pre-process the source data to form a data table;

[0009] S2, after configuring the data table with corresponding uninstall information, uninstall it into a data file, and generate a corresponding verification file;

[0010] S3, transmitting the data file and the verification file to a designated client through a file transmission platform;

[0011] S4, the unified data offloading service component in the client queries the corresponding offloading policy information in the policy table according to the data table name in the verification file and connects to the database, wherein the policy table is preset;

[0012] S5, reading the verification file and parsing the data file through the unified data unloading service component, converting the data file into a data format that matches the database, and then performing a database insertion operation;

[0013] S6. Generate a log result table according to relevant data in the data insertion process, wherein the log result table is used to query execution records or perform error log tracking.

[0014] Further, the S1 specifically includes:

[0015] S11. Import the source data in the source system into the cloud data lake platform through the data transmission service;

[0016] S12, preprocessing the source data based on the data lake platform, wherein the preprocessing includes removing duplicate data, correcting erroneous data, formatting data, and performing data standardization;

[0017] S13. The preprocessed data forms a data table based on the data model.

[0018] Further, the S2 specifically includes:

[0019] S21, configuring corresponding uninstallation information for the data table, wherein the uninstallation information includes uninstallation type, uninstallation frequency, and uninstallation time;

[0020] S22. Unload the data table into a data file according to the unloading information, and simultaneously generate a verification file corresponding to the data file, wherein the verification file contains meta-information about the data file, and the meta-information includes file size, number of data items, and field verification information for data integrity and accuracy verification.

[0021] Further, the S3 specifically includes:

[0022] S31, defining uninstall interface information in the file transfer platform, wherein the uninstall interface information includes uninstall ID, uninstall type, file type, encoding format, and file landing path;

[0023] S32: Synchronize the uninstall interface information to the file transmission platform, and the file transmission platform transmits the data file and the verification file to the designated client.

[0024] Further, the S4 specifically includes:

[0025] S41, comparing the verification file with the data to verify integrity and accuracy through the unified data offloading service component;

[0026] S42: The unified data unloading service component in the client queries the corresponding unloading policy information in the preset policy table according to the data table name in the verification file, wherein the unloading policy information includes the landing path of the data file, the target database, the target data table, and the unloading method corresponding to the unloading type;

[0027] S43, the unified offloading service component connects to the corresponding database according to the offloading policy information, and checks whether the target data table exists;

[0028] S44. If the target data table already exists, the unified unloading service component performs subsequent operations according to the unloading policy information. If the target data table does not exist and the unloading policy information allows the creation of a new table, the unified unloading service component automatically generates and executes a table creation SQL statement according to the unloading policy information and the table structure information in the verification file to create the corresponding data table.

[0029] Further, the S5 specifically includes:

[0030] S51, the unified data unloading service component reads the verification file and constructs a field mapping table, wherein the field mapping table is used for data parsing and conversion;

[0031] S52, the unified data unloading service component reads the data file byte by byte, and uses a separator to distinguish different fields to obtain original data;

[0032] S53, the unified data unloading service component converts the format of the original data according to the field mapping table;

[0033] S54, updating the converted original data to the target data table and inserting it into the database.

[0034] In addition, the present invention also relates to a configurable cloud data lake to application database data unloading system applied to a bank, characterized in that it includes:

[0035] The data import module imports the source data of the source system into the data lake platform and pre-processes the source data to form a data table;

[0036] An uninstallation module, configured to uninstall the data table into a data file after configuring the corresponding uninstallation information, and generate a corresponding verification file;

[0037] A transmission module, used to transmit the data file and the verification file to a designated client via a file transmission platform;

[0038] A policy execution module, used to query the corresponding uninstallation policy information in the policy table according to the data table name in the verification file through the unified unloading service component and connect to the database, wherein the policy table is preset;

[0039] A data insertion module, used to read the verification file and parse the data file through a unified data unloading service component, convert the data file into a data format that matches the database, and then perform a database insertion operation;

[0040] The log module is used to generate a log result table according to relevant data in the data insertion process, and the log result table is used to query execution records or perform error log tracking.

[0041] In addition, the present invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

[0042] The memory is used to store operation instructions;

[0043] The processor is used to execute the above method by calling the operation instruction.

[0044] In addition, the present invention also relates to a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0045] Adopt the scheme provided by this application, import the source data of the source system into the data lake platform, and pre-process the source data to form a data table; configure the corresponding unloading information for the data table and unload it into a data file, and generate a corresponding verification file; transfer the data file and the verification file to the specified client through the file transfer platform; in the client, the unified unloading service component queries the corresponding unloading policy information in the policy table according to the data table name in the verification file and connects to the database, wherein the policy table is pre-set; the unified unloading service component reads the verification file and parses the data file, converts the data file into a data format that matches the database, and then performs a database insertion operation; generates a log result table according to the relevant data in the data insertion process, and the log result table is used to query the execution record or track the error log. Therefore, through data analysis and transformation, the data in the data lake can be automatically identified and transformed, and the data can be efficiently and flexibly migrated to the application database of the source system, which reduces the complexity of data unloading, improves the flexibility and adaptability of data processing, and enables banks to make full use of the powerful computing power and data storage capacity of the data lake, while meeting the source system's needs for data statistical analysis in multiple dimensions and different perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a method for unloading data from a configured cloud data lake to an application database applied to a bank provided in an embodiment of the present application;

[0047] Figure 2 A structural diagram of a configurable cloud data lake to application database data unloading system for a bank provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.

[0050] Those skilled in the art can understand that, unless otherwise stated, the singular forms "one", "an", "said" and "the" used herein may also include plural forms, and the "first", "second", etc., are only used to distinguish and define objects for the purpose of introducing the scheme clearly, and do not limit the objects themselves. Of course, the objects defined by "first" and "second" may be the same terminal, device, user, etc., or the same type of terminal, device and user. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0051] In order to more clearly introduce the technical solution of the present application, some concepts, terms or devices that may be involved in the following embodiments are introduced below to help understand the technical solution disclosed in the present application:

[0052] Data Lake: A system or storage that stores data in its natural / original format, usually as blocks of objects or files. A data lake is usually a single store for all the data in an enterprise, including raw copies of data generated by the original system and transformed data generated for various tasks, such as reporting, visualization, advanced analytics, and machine learning. There are many types of data in a data lake, mainly structured data (rows and columns) from relational databases, semi-structured data (such as CSV, logs, XML, JSON), unstructured data (such as email, documents, PDF, etc.), and binary data (such as images, audio, video).

[0053] File transfer platform (FEX): It is an in-house file transfer platform with the ability to transfer large volumes of files. By defining the IP, port, and path addresses of the producer and consumer servers, it provides scheduled file transfer services between servers with FEX clients installed.

[0054] At present, the cloud data lake in the bank carries data from all aspects of the source layer, common layer, domain market, application layer, etc. of each system. It is a bridge for data exchange between various systems and an important data asset in the bank. However, the field attributes, length, number and other factors of the processing tables in the lake are different and uncontrollable.

[0055] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0056] Figure 1 The flowchart of the method for unloading data from a configured cloud data lake to an application database applied to a bank is shown in an embodiment of the present application, which specifically includes:

[0057] S1. Import the source data of the source system into the data lake platform, and pre-process the source data to form a data table;

[0058] In this implementation, S1 specifically includes:

[0059] S11. Import the source data in the source system into the cloud data lake platform through the data transmission service;

[0060] S12, preprocessing the source data based on the data lake platform, wherein the preprocessing includes removing duplicate data, correcting erroneous data, formatting data, and performing data standardization;

[0061] S13. The preprocessed data forms a data table based on the data model.

[0062] S2, after configuring the data table with corresponding uninstall information, uninstall it into a data file, and generate a corresponding verification file;

[0063] In this implementation, S2 specifically includes:

[0064] S21, configuring corresponding uninstallation information for the data table, wherein the uninstallation information includes uninstallation type, uninstallation frequency, and uninstallation time;

[0065] Among them, the uninstallation type is divided into incremental type, full type, and update type, and the uninstallation frequency is daily / weekly / monthly / quarterly / yearly / according to other cron expressions, and the uninstallation time is the specific time of uninstalling the file.

[0066] S22. According to the uninstallation information, the data table is uninstalled into a .dtf format data file, and a .ok format verification file corresponding to the data file is generated at the same time, wherein the verification file contains meta-information about the data file, and the meta-information includes file size, number of data items, and field verification information for data integrity and accuracy verification.

[0067] S3, transmitting the data file and the verification file to a designated FEX client through the FEX file transfer platform;

[0068] In this implementation, S3 specifically includes:

[0069] S31. Define uninstall interface information in the FEX file transfer platform, where the uninstall interface information includes uninstall ID, uninstall type, file type, encoding format, and file landing path;

[0070] The file landing path is the absolute path in the installed fex client server.

[0071] S32: Synchronize the unloading interface information to the FEX file transmission platform, and the FEX file transmission platform transmits the data file and the verification file to the designated FEX client.

[0072] S4, the unified data offloading service component in the FEX client queries the corresponding offloading policy information in the policy table according to the data table name in the verification file and connects to the database, wherein the policy table is pre-set;

[0073] In this implementation, S4 specifically includes:

[0074] S41, comparing the verification file with the data to verify integrity and accuracy through the unified data offloading service component;

[0075] Specifically, use an XML parser to read the .ok verification file, extract key verification information from the verification file, such as the size of the data file, the number of records, and the field checksum, etc. According to the data file in the .dtf format, use byte-by-byte reading / delimiter splitting to parse the data file;

[0076] Compare the actual size of the data file on disk with the size recorded in the verification file to see if they are consistent, and compare the number of data records obtained after parsing with the number of records recorded in the verification file to verify the integrity and accuracy;

[0077] If a mismatch is found during the verification process, the error information is recorded, an error report is generated, and the error information is inserted into the log table for subsequent tracking and problem location.

[0078] S42: The unified data unloading service component in the client queries the corresponding unloading policy information in the preset policy table according to the data table name in the verification file, wherein the unloading policy information includes the landing path of the data file, the target database, the target data table, and the unloading method corresponding to the unloading type;

[0079] Specifically, the preset policy table records the uninstallation method corresponding to the uninstallation type. If the uninstallation type is a full type, the data added on the previous day will be uninstalled (filtered according to the incremental field); if it is a full type, all full table data will be uninstalled; if it is an update type, the data updated on the previous day will be uninstalled (filtered according to the update field).

[0080] S43, the unified offloading service component connects to the corresponding database according to the offloading policy information, and checks whether the target data table exists;

[0081] S44. If the target data table already exists, the unified unloading service component performs subsequent operations according to the unloading policy information. If the target data table does not exist and the unloading policy information allows the creation of a new table, the unified unloading service component automatically generates and executes a table creation SQL statement according to the unloading policy information and the table structure information in the verification file to create the corresponding data table.

[0082] S5, reading the verification file and parsing the data file through the unified data unloading service component, converting the data file into a data format that matches the database, and then performing a database insertion operation;

[0083] In this implementation, S5 specifically includes:

[0084] S51, the unified data unloading service component reads the verification file and constructs a field mapping table, wherein the field mapping table is used for data parsing and conversion;

[0085] S52, the unified data unloading service component reads the data file byte by byte, and uses a separator (0x03) to distinguish different fields to obtain original data;

[0086] S53, the unified data unloading service component converts the format of the original data according to the field mapping table;

[0087] S54, updating the converted original data to the target data table and inserting it into the database.

[0088] S6. Generate a log result table according to relevant data in the data insertion process, wherein the log result table is used to query execution records or perform error log tracking.

[0089] Specifically, the relevant data in the data insertion process includes the execution result (success / failure), the business date of execution, the number of data items executed, the start and end time of execution, error information, etc.

[0090] In addition, the execution log can be queried through the visual interface to facilitate error log tracking.

[0091] By adopting the above technical solution, the data in the data lake can be automatically identified and transformed through data analysis and transformation, and the data can be efficiently and flexibly migrated to the application database of the source system, which reduces the complexity of data unloading and improves the flexibility and adaptability of data processing. It enables banks to fully utilize the powerful computing power and data storage capabilities of the data lake, while meeting the source system's needs for data statistical analysis in multiple dimensions and from different perspectives.

[0092] based on Figure 1The configuration cloud data lake to application database data unloading method for bank is shown, and on the other hand, it also relates to a configuration cloud data lake to application database data unloading system for bank, and its structure is as follows Figure 2 As shown, including:

[0093] The data import module 201 imports the source data of the source system into the data lake platform, and pre-processes the source data to form a data table;

[0094] The unloading module 202 is used to unload the data table into a data file after configuring the corresponding unloading information, and generate a corresponding verification file;

[0095] The transmission module 203 is used to transmit the data file and the verification file to a designated client via a file transmission platform;

[0096] The policy execution module 204 is used to query the corresponding uninstallation policy information in the policy table according to the data table name in the verification file through the unified uninstallation service component and perform database connection, wherein the policy table is preset;

[0097] The data insertion module 205 is used to read the verification file and parse the data file through the unified data unloading service component, convert the data file into a data format that matches the database, and then perform a database insertion operation;

[0098] The log module 206 is used to generate a log result table according to relevant data in the data insertion process, and the log result table is used to query execution records or perform error log tracking.

[0099] Further, the data import module 201 is specifically used for:

[0100] Import the source data in the source system to the cloud-based data lake platform through the data transfer service;

[0101] Preprocessing the source data based on the data lake platform, wherein the preprocessing includes removing duplicate data, correcting erroneous data, formatting data, and performing data standardization;

[0102] The preprocessed data forms a data table based on the data model.

[0103] Further, the uninstall module 202 is specifically used for:

[0104] Configure corresponding uninstall information for the data table, the uninstall information including uninstall type, uninstall frequency, and uninstall time;

[0105] According to the uninstall information, the data table is uninstalled into a data file, and a verification file corresponding to the data file is generated at the same time, wherein the verification file contains meta-information about the data file, and the meta-information includes file size, number of data items, and field verification information, which is used for data integrity and accuracy verification.

[0106] Further, the transmission module 203 is specifically used for:

[0107] Defining uninstall interface information in the file transfer platform, wherein the uninstall interface information includes uninstall ID, uninstall type, file type, encoding format, and file landing path;

[0108] The uninstall interface information is synchronized to the file transmission platform, and the file transmission platform transmits the data file and the verification file to the designated client.

[0109] Furthermore, the policy execution module 204 is specifically used for:

[0110] Comparing the verification file with the data to verify the integrity and accuracy by the unified data unloading service component;

[0111] The unified data unloading service component in the client queries the corresponding unloading policy information in the preset policy table according to the data table name in the verification file, wherein the unloading policy information includes the landing path of the data file, the target database, the target data table, and the unloading method corresponding to the unloading type;

[0112] The unified data offloading service component connects to the corresponding database according to the offloading policy information, and checks whether the target data table exists;

[0113] If the target data table already exists, the unified unloading service component performs subsequent operations according to the unloading policy information. If the target data table does not exist and the unloading policy information allows the creation of a new table, the unified unloading service component automatically generates and executes a table creation SQL statement according to the unloading policy information and the table structure information in the verification file to create the corresponding data table.

[0114] Further, the data insertion module 205 is specifically used for:

[0115] The unified data unloading service component reads the verification file and constructs a field mapping table, wherein the field mapping table is used for data parsing and conversion;

[0116] The unified data unloading service component reads the data file byte by byte, and uses separators to distinguish different fields to obtain original data;

[0117] The unified data unloading service component converts the format of the original data according to the field mapping table;

[0118] The converted original data is updated to the target data table and inserted into the database.

[0119] An embodiment of the present invention further provides an electronic device for executing the above method, as an implementation device of the method, comprising a processor and a memory;

[0120] A memory, used for storing operation instructions;

[0121] The processor is used to execute the configurable cloud data lake to application database data unloading method provided in any embodiment of the present application by calling an operation instruction.

[0122] As an example, Figure 3 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present application is shown, and the electronic device 300 includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as being connected through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one. It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the specific structure of the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Optionally, the electronic device may also include a display screen 305 for displaying images, or receiving user operation instructions when necessary.

[0123] The processor 301 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 304 may include a receiver and a transmitter. The transceiver 304 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0124] Processor 301 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0125] The processor 301 may also include one or more processing units, for example, the processor 301 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device 300. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be provided in the processor 301 for storing instructions and data. In some embodiments, the memory in the processor 301 is a high-speed cache memory. The memory may store instructions or data that the processor 301 has just used or circulated. If the processor 301 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 301 is reduced, and the efficiency of the system is improved.

[0126] The processor 301 can run a configurable cloud data lake to application database data unloading method for banks provided in an embodiment of the present application. The processor 301 may include different devices. For example, when a CPU and a GPU are integrated, the CPU and the GPU can cooperate to execute a configurable cloud data lake to application database data unloading method for banks provided in an embodiment of the present application. Part of the algorithm is executed by the CPU, and the other part of the algorithm is executed by the GPU to obtain faster processing efficiency.

[0127] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0128] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or may include a high-speed random access memory, or may include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0129] Optionally, the memory 303 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the configurable cloud data lake to application database data unloading method provided in any embodiment of the present application.

[0130] The memory 303 can be used to store computer executable program codes, which include instructions. The processor 301 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the memory 303. The memory 303 may include a program storage area and a data storage area. Among them, the program storage area may store the code of the operating system, application program, etc. The data storage area may store data created during the use of the electronic device 300 (such as images, videos, etc. collected by the camera application), etc.

[0131] The memory 303 may also store one or more computer programs corresponding to the method for unloading data from the configured cloud data lake to the application database provided in the embodiment of the present application. The one or more computer programs are stored in the above-mentioned memory 303 and are configured to be executed by the one or more processors 301. The one or more computer programs include instructions, and the above-mentioned instructions can be used to execute each step in the above-mentioned corresponding embodiment.

[0132] Of course, the code of the configurable cloud data lake to application database data unloading method provided in the embodiment of the present application can also be stored in an external memory.

[0133] The display screen 305 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include 1 or N display screens 305, where N is a positive integer greater than 1. The display screen 305 can be used to display information input by a user or information provided to a user and various graphical user interfaces (GUI). For example, the display screen 305 can display photos, videos, web pages, or files.

[0134] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0135] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0136] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0138] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0140] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A configurable cloud data lake to application database data unloading method for banks, characterized in that: The following steps are involved: S1. Import the source data of the source system into the data lake platform, and pre-process the source data to form a data table; S2, after configuring the data table with corresponding uninstall information, uninstall it into a data file, and generate a corresponding verification file; S3, transmitting the data file and the verification file to a designated client through a file transmission platform; S4, the unified data offloading service component in the client queries the corresponding offloading policy information in the policy table according to the data table name in the verification file and connects to the database, wherein the policy table is preset; S5. Read the verification file and parse the data file through the unified data unloading service component, convert the data file into a data format that matches the database, and then perform a database insertion operation.

2. The method for unloading data from a configured cloud data lake to an application database for a bank according to claim 1, characterized in that: The S1 specifically includes: S11. Import the source data in the source system into the cloud data lake platform through the data transmission service; S12, preprocessing the source data based on the data lake platform, wherein the preprocessing includes removing duplicate data, correcting erroneous data, formatting data, and performing data standardization; S13. The preprocessed data forms a data table based on the data model.

3. The method for unloading data from a configured cloud data lake to an application database for a bank according to claim 1, characterized in that: The S2 specifically includes: S21, configuring corresponding uninstallation information for the data table, wherein the uninstallation information includes uninstallation type, uninstallation frequency, and uninstallation time; S22. Unload the data table into a data file according to the unloading information, and simultaneously generate a verification file corresponding to the data file, wherein the verification file contains meta-information about the data file, and the meta-information includes file size, number of data items, and field verification information for data integrity and accuracy verification.

4. The method for unloading data from a configured cloud data lake to an application database for a bank according to claim 1, characterized in that: The S3 specifically includes: S31, defining uninstall interface information in the file transfer platform, wherein the uninstall interface information includes uninstall ID, uninstall type, file type, encoding format, and file landing path; S32: Synchronize the uninstall interface information to the file transmission platform, and the file transmission platform transmits the data file and the verification file to the designated client.

5. The method for unloading data from a configured cloud data lake to an application database for a bank according to claim 1, characterized in that: The S4 specifically includes: S41, comparing the verification file with the data to verify integrity and accuracy through the unified data offloading service component; S42: The unified data unloading service component in the client queries the corresponding unloading policy information in the preset policy table according to the data table name in the verification file, wherein the unloading policy information includes the landing path of the data file, the target database, the target data table, and the unloading method corresponding to the unloading type; S43, the unified offloading service component connects to the corresponding database according to the offloading policy information, and checks whether the target data table exists; S44. If the target data table already exists, the unified unloading service component performs subsequent operations according to the unloading policy information. If the target data table does not exist and the unloading policy information allows the creation of a new table, the unified unloading service component automatically generates and executes a table creation SQL statement according to the unloading policy information and the table structure information in the verification file to create the corresponding data table.

6. The method for unloading data from a configured cloud data lake to an application database for use in a bank according to claim 5, characterized in that: The S5 specifically includes: S51, the unified data unloading service component reads the verification file and constructs a field mapping table, wherein the field mapping table is used for data parsing and conversion; S52, the unified data unloading service component reads the data file byte by byte, and uses a separator to distinguish different fields to obtain original data; S53, the unified data unloading service component converts the format of the original data according to the field mapping table; S54, updating the converted original data to the target data table and inserting it into the database.

7. The method for unloading data from a configured cloud data lake to an application database for a bank according to claim 1, characterized in that: The method further comprises: S6. Generate a log result table according to relevant data in the data insertion process, wherein the log result table is used to query execution records or perform error log tracking.

8. A configurable cloud data lake to application database data unloading system for banks, characterized in that: include: The data import module imports the source data of the source system into the data lake platform and pre-processes the source data to form a data table; An uninstallation module, configured to uninstall the data table into a data file after configuring the corresponding uninstallation information, and generate a corresponding verification file; A transmission module, used to transmit the data file and the verification file to a designated client via a file transmission platform; A policy execution module, used to query the corresponding uninstallation policy information in the policy table and perform database connection according to the data table name in the verification file through the unified uninstallation service component, wherein the policy table is preset; A data insertion module, used to read the verification file and parse the data file through a unified data unloading service component, convert the data file into a data format that matches the database, and then perform a database insertion operation; The log module is used to generate a log result table according to relevant data in the data insertion process, and the log result table is used to query execution records or perform error log tracking.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store operation instructions; The processor is used to execute the method according to any one of claims 1 to 7 by calling the operation instruction.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images