Method and system for transmitting system data to data lake based on configuration

Abstract

The invention relates to a method and a system for transmitting system data to a data lake based on configuration. The method comprises the following steps of: extracting data from each micro-service database in a configuration manner by utilizing a unified service component; generating a lake entering file according to the configuration, and transmitting the lake entering file to a data lake platform by using an FEX file transmission platform; analyzing and storing the lake entering file on the data lake platform, and executing data cleaning and processing; and unloading the in-lake data file to a target system according to a service requirement for further analysis and application. The scheme is remarkable in flexibility and universality, supports cross-service data extraction, optimizes resource task scheduling, and solves the problem of performance security.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for transmitting system data to a data lake based on configuration. Background Art

[0002] In today's digital age, the rapid development of banking business has created a huge demand for data processing and analysis. Cloud data lake technology has emerged as the times require, and with its powerful computing and storage capabilities, it has become the backbone of banking data management. The implementation of the data lake not only greatly alleviates the data computing pressure of the source system, but also maximizes the value of data by breaking down the data barriers between systems.

[0003] However, existing technologies have limitations in efficiently and stably offloading source system database data to the data lake. Especially with the widespread use of microservice architecture today, how to ensure the real-time and accuracy of data and the stability of system performance has become an urgent problem to be solved.

[0004] This invention was born in such a context. It innovatively integrates the microservice architecture, cloud data lake and FEX file transfer platform, opens up the link of data flow, and proposes a method for transferring system data to the data lake based on configuration. This method not only realizes the efficient transmission of source system database data to the data lake, but also ensures the performance stability and accuracy of the data entry process by reasonably allocating microservice application resources. Summary of the invention

[0005] In order to address the shortcomings of the prior art, the present invention realizes the operations of adding, modifying, and deleting data lake tables and unloading data to the application database by flexibly configuring library table data. There is no need to develop and launch it, and the flexible configuration can take effect in real time. It has flexibility and versatility, and can realize cross-service data acquisition between microservices and send files into the lake, solving the problems of resource task scheduling and allocation and resource idleness among the microservices of the system, and preventing performance and safety problems caused by too many data forms entering the lake or too much use of application resources by individual microservices.

[0006] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0007] A method for transferring system data to a data lake based on configuration, comprising:

[0008] S1. Use unified service components to extract data from each microservice database in a configurable manner;

[0009] S2. Generate a lake entry file according to the configuration, and use the FEX file transfer platform to transfer the lake entry file to the data lake platform;

[0010] S3, parsing and storing the data lake entry files on the data lake platform, and performing data cleaning and processing;

[0011] S4. According to business needs, unload the data files in the lake to the target system for further analysis and application.

[0012] Furthermore, the step S1 further comprises:

[0013] Define the data entry strategy, record the configured strategy information in the corresponding scheduled task, and set the execution period of the scheduled task.

[0014] Furthermore, the information of the configuration policy includes but is not limited to: uninstall type, lake entry frequency, file type, encoding format and delivery path name.

[0015] Furthermore, the step S3 also includes: cleaning and processing the files entering the lake through the processing operators in the lake for use by other systems.

[0016] Furthermore, the unified service component is also used to manage the storage, archiving, and cleaning of files entering the lake, as well as define an interface for unloading files.

[0017] Furthermore, the integrated design between the data lake platform and the FEX file transfer platform enables the FEX client to directly accept lake-entry files from the FEX file transfer platform, ensuring that the process of transferring files to the data lake client is seamless and efficient.

[0018] A system for transferring system data to a data lake based on configuration, characterized by comprising:

[0019] Microservice architecture module, used to support data extraction from multiple microservice application databases;

[0020] The data lake platform module is used to parse and store the files entering the lake and perform data cleaning and processing;

[0021] FEX file transfer platform module, used to transfer lake-entry files from the microservice architecture to the data lake platform;

[0022] The unified service component module is used to configure and define the strategy for retrieving data and sending files into the lake, as well as to execute the tasks of entering the lake at a scheduled time.

[0023] Specifically, the unified service component is the core of the entire solution. It is responsible for executing the lake entry task, fetching data from each microservice database and generating the corresponding lake entry file, and then uploading the lake entry file to the fixed platform for FEX to move into the lake. It also takes care of storing files, archiving and cleaning files, defining the interface for uninstalling files, parsing files, verifying files, policy task configuration, and health checks for microservice lake entry tasks. The data lake platform is responsible for the entry of source data from each database into the lake, cleaning and processing data in the lake, and uninstalling data files in the lake. The FEX file transfer platform is responsible for transferring files.

[0024] The present invention also provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

[0025] The present invention also provides an electronic device, characterized in that it includes a processor and a memory;

[0026] The memory is used to store computer programs;

[0027] The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

[0028] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement any of the methods of claims 1 to 6 when executed by a processor.

[0029] The present invention also provides a unified service component for transmitting system data to a data lake, wherein the unified service component comprises:

[0030] The policy definition module is used to define the policy for retrieving data into the lake in a configurable manner;

[0031] Task configuration module, used to record policy information in the corresponding scheduled tasks;

[0032] The data processing module is used to access the corresponding microservice database and retrieve data based on the policy information, and then upload the generated entry file to the lake for the FEX client to handle;

[0033] The task execution module is used to execute the entry-to-lake strategy and issue the entry-to-lake file.

[0034] The beneficial effects of the present invention are as follows: the present invention rationally allocates microservice application resources, cleverly combines the microservice architecture, data lake platform and FEX file transfer platform. The core advantage of the solution lies in the ability to extract data across services and send files to the lake, effectively solving the problem of resource task scheduling and allocation between microservices, and avoiding performance and safety risks caused by idle or overloaded resources. Through minute-level time control, the efficiency of developers responding to data retrieval needs is greatly improved, productivity is significantly enhanced, and the configuration can take effect in real time without the need for traditional development, showing extremely high flexibility and versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a flowchart of a method for transferring data to a data lake based on configuration according to the present invention applied to a bank.

[0036] Figure 2 This is a schematic diagram of the system structure for transmitting data to a data lake based on configuration in the present invention.

[0037] Figure 3 The figure is a logical flow chart of the execution method of the present invention for transferring data to a data lake based on configuration.

[0038] Figure 4 This is a functional schematic diagram of the present invention based on configuration to transfer data to the data lake.

[0039] Figure 5 This is a schematic diagram of the unified service component structure of the system for transmitting data to a data lake based on configuration in the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0042] Embodiment 1

[0043] The first aspect of the present invention relates to a process of steps as follows Figure 1 and 3 A configuration-based method for transferring system data to a data lake is shown, comprising:

[0044] S1. Use unified service components to extract data from each microservice database in a configurable manner;

[0045] S2. Generate a lake entry file according to the configuration, and use the FEX file transfer platform to transfer the lake entry file to the data lake platform;

[0046] S3, parsing and storing the data lake entry files on the data lake platform, and performing data cleaning and processing;

[0047] S4. According to business needs, unload the data files in the lake to the target system for further analysis and application.

[0048] Specifically, open the lake entry strategy configuration page in a separate menu bar on the system configuration page. This page can be used to add, delete, modify, and query the lake entry table task strategy. At the same time, you can flexibly configure data retrieval conditions, fields, sequence, attributes and other information. After completion, you can view the configured policy information and table field definitions in the configuration table.

[0049] like Figure 4 As shown, the created policy is configured as a scheduled task, and the execution time and period can be defined. At the same time, health checks and regular space cleanup tasks can be configured to ensure stable component functions and save system space.

[0050] The entry-lake strategy table for task execution is in the req microservice database, but the task executor is executed by the res microservice. The reason is that the req microservice undertakes many other application functions and its own service resources are limited. In addition, there are many tasks in the entry-lake table, and other microservices are needed to share the execution. Therefore, the task is transferred to the res microservice for execution.

[0051] The specific task execution process includes: After the scheduled task is configured, it will be executed regularly according to the configured time (data acquisition, health check, file cleanup). Here we mainly introduce the data acquisition task:

[0052] According to the policy ID configured in the task, query the policy information stored in the table, including database, table name, field attributes, where conditions, lake entry strategy (full, incremental or historical, etc.), file naming, file distribution path, etc. Execute the select statement in the corresponding microservice database to obtain data based on the obtained policy information. The backend of this step has been configured with paging query to prevent the failure of reading and distributing files at one time due to the large amount of data.

[0053] Create a def file with the corresponding name and date in the path of the policy configuration, write the data obtained by paging into the file, and record the data information (number of data items, file size, etc.) after completion. Create an ok file with the corresponding name and date in the path, and write the recorded data information, field attributes in the policy configuration table, and other information into the file.

[0054] Health check: After configuring the scheduled tasks, the health scan check of the inbound strategy tasks will be performed according to the configured time. The number of tasks, time, data volume, number of fields and other characteristics of each microservice will be compared horizontally to prevent individual microservices from having peak concurrent tasks or excessive execution, which will cause excessive service resources to be occupied and affect system performance and the use of other functions.

[0055] File cleaning function: After configuring the corresponding task and setting the time range, path, execution time and frequency, the files under the path will be filtered and cleared within the date range at a fixed time point to prevent the generation of a large number of files due to too many data entry tasks and occupying too much server space, thus ensuring the security and stability of the system.

[0056] The above-mentioned functional tasks can be triggered or modified manually when the task execution deviates and needs to be corrected. For example, the original file cleanup was set to be done once a month, and only files from three months ago were cleaned up. However, when it was found that the system memory space was insufficient, the files entered in the lake for only two months occupied most of the storage space. At this time, the cleanup time range of the scheduled task can be modified, and the files can be cleaned up immediately manually. This ensures the flexibility and scalability of the overall solution function and ensures the performance and safety of the corresponding system.

[0057] Embodiment 2

[0058] Step 1: According to the configured policies and scheduled tasks, the required data is extracted from the microservice database and the lake entry file is generated. The data extracted from the database is converted into a lake entry file in a predetermined format, such as CSV, JSON, or XML.

[0059] Step 2: The extracted data is processed and converted to generate standardized lake-entry files, which are uploaded to the FEX file transfer platform and sent to the data lake platform through a stable and reliable transmission service;

[0060] Step 3: The data lake platform parses the received files, stores the parsed data in the data lake, and performs data cleaning and processing operations;

[0061] Step 4: Store the processed data in the data lake and unload the data from the data lake to the target system or for further analysis and application based on business needs;

[0062] Step 5: Unify the service component execution management work including file storage, archiving, and cleanup, as well as manual execution strategy management work such as defining the uninstall file interface and viewing log table information;

[0063] Step 6: During this process, the unified service component is also responsible for monitoring the execution status of the lake entry task, performing health checks, and ensuring the stability and security of data transmission and processing processes.

[0064] Embodiment 3

[0065] Another aspect of the present invention also relates to a system for transferring system data to a data lake based on configuration, wherein the structure is as follows: Figure 2 As shown, including:

[0066] Microservice architecture module, used to support data extraction from multiple microservice application databases;

[0067] The data lake platform module is used to parse and store the files entering the lake and perform data cleaning and processing;

[0068] FEX file transfer platform module, used to transfer lake-entry files from the microservice architecture to the data lake platform;

[0069] The unified service component module is used to configure and define the strategy for retrieving data and sending files into the lake, as well as to execute the tasks of entering the lake at a scheduled time.

[0070] Specifically, the data lake platform is used to accept the source data from various source systems into the lake. FEX moves the lake-entering files of each system to the data lake FEX client, and then the data lake performs file parsing and data storage. At the same time, it provides processing operators in the lake for each application system to perform common layer data cleaning and application layer data processing.

[0071] The FEX file transfer platform is used to provide stable and reliable file transfer services. After integrating it with the data lake platform, it can obtain the lake-entry files sent by the designated FEX client of each system, and transfer the lake-entry files to the data lake client for use by various application systems.

[0072] The unified service component is used to configure and define the strategy for retrieving data and sending files into the lake, as well as to execute the tasks of entering the lake at a scheduled time. At the same time, it also provides the reading of microservice database data and the generation of files into the lake; supports microservice architectures to access each other's databases and execute the strategies for entering the lake; regularly cleans up overdue files; performs health checks on tasks entering the lake; tracks error logs and queries execution records; and provides a visual configuration interface for manual execution strategies and component services.

[0073] By using this system, the above-mentioned operation processing method can be executed and the corresponding technical effects can be achieved.

[0074] Embodiment 4

[0075] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0076] Embodiment 5

[0077] An embodiment of the present invention also provides an electronic device for executing the above method. As an implementation device of the method, the electronic device has at least a processor and a memory, and in particular, the memory stores data and related computer programs required for executing the method, and the processor calls the data and programs in the memory to execute all steps of the implementation method and obtains the corresponding technical effect.

[0078] Preferably, the electronic device may include a bus architecture, the bus may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are all well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same component, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor when performing operations.

[0079] In addition, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operation control) used may include a microprocessor or other processor device and / or a logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices, which may store the above-mentioned related data information, and may also store a program for executing related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it may be a key or a touch input device; the power supply is used to provide power to the electronic device; the display is used to display display objects such as images and text, for example, it may be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processor, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0080] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0082] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0084] Embodiment 6

[0085] The embodiment of the present invention also provides a unified service component for transmitting system data to a data lake, such as Figure 5 As shown, the unified service component includes:

[0086] The policy definition module is used to define the policy for retrieving data into the lake in a configurable manner;

[0087] Task configuration module, used to record policy information in the corresponding scheduled tasks;

[0088] The data processing module is used to access the corresponding microservice database and retrieve data based on the policy information, and then upload the generated entry file to the lake for the FEX client to handle;

[0089] The task execution module is used to execute the entry-to-lake strategy and issue the entry-to-lake file.

[0090] Exemplarily, the policy definition module includes but is not limited to: i. defining the database and form content that you want to enter the lake; ii. configuring field attributes; iii. configuring the unloading type, such as incremental, full, update, delete, add + modify, add + modify + delete; iv. configuring the frequency of entering the lake; v. configuring the file type, encoding format, and file distribution path name; vi. synchronizing interface information to the FEX file transfer platform; The policy definition module further includes: i. providing add, delete, modify, and query functions, allowing users to flexibly configure and modify the lake entry policy.

[0091] The task configuration module further includes: i. providing monitoring and management of task execution to ensure smooth progress and timely response of tasks; ii. configuring execution strategy information and execution time cycle.

[0092] The data processing module, i. provides paging processing for data reading to meet the needs of reading large amounts of data; ii. provides log records of file generation to record key information in the file generation process.

[0093] The task execution module further includes: i. providing access control and authority management to ensure the security of data access; ii. providing exception handling and recovery mechanism for task execution to improve the robustness of the system.

[0094] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for transferring system data to a data lake based on configuration, characterized in that: The following steps are involved: S1. Use unified service components to extract data from each microservice database in a configurable manner; S2. Generate a lake entry file according to the configuration, and use the FEX file transfer platform to transfer the lake entry file to the data lake platform; S3, parsing and storing the data lake entry files on the data lake platform, and performing data cleaning and processing; S4. According to business needs, unload the data files in the lake to the target system for further analysis and application.

2. The method according to claim 1, characterized in that The step S1 further comprises: Define the data entry strategy, record the configured strategy information in the corresponding scheduled task, and set the execution period of the scheduled task.

3. The method according to claim 2, characterized in that The information of the configuration policy includes but is not limited to: uninstall type, lake entry frequency, file type, encoding format and delivery path name.

4. The method according to claim 3, characterized in that The step S3 also includes: performing data cleaning and processing on the files entering the lake through the processing operators within the lake for use by other systems.

5. The method according to claim 1, characterized in that The unified service component is also used to manage the storage, archiving, and cleaning of files entering the lake, and to define an interface for unloading files.

6. The method according to claim 4, characterized in that The integrated design between the data lake platform and the FEX file transfer platform enables the FEX client to directly accept lake-entry files from the FEX file transfer platform, ensuring that the process of file transfer to the data lake client is seamless and efficient.

7. A system for transferring system data to a data lake based on configuration, characterized in that: include: Microservice architecture module, used to support data extraction from multiple microservice application databases; The data lake platform module is used to parse and store the files entering the lake and perform data cleaning and processing; FEX file transfer platform module, used to transfer lake-entry files from the microservice architecture to the data lake platform; The unified service component module is used to configure and define the strategy for retrieving data and sending files into the lake, as well as to execute the tasks of entering the lake at a scheduled time.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

11. A unified service component for transferring system data to a data lake, characterized in that: The unified service components include: The policy definition module is used to define the policy for retrieving data into the lake in a configurable manner; Task configuration module, used to record policy information in the corresponding scheduled tasks; The data processing module is used to access the corresponding microservice database and retrieve data based on the policy information, and then upload the generated entry file to the lake for the FEX client to handle; The task execution module is used to execute the entry-to-lake strategy and issue the entry-to-lake file.

Images