Business data extraction method and device, storage medium and computer equipment

Abstract

The invention relates to the technical field of financial science and technology and the technical field of computers, and discloses a business data extraction method and device, a storage medium and computer equipment, and the method comprises the steps: collecting business data and metadata corresponding to the business data for the business data generated by a target business platform in a preset collection time period, constructing an extraction data table of a preset acquisition time period, and storing the extraction data table in a distributed file system of a data extraction platform; and receiving a business data extraction instruction, obtaining demand business data in the distributed file system based on a demand acquisition time period, a demand business database name and a demand business data table name corresponding to the business data extraction instruction, and returning the demand business data to a demand receiving terminal corresponding to the business data extraction instruction, the business data extraction instruction is triggered based on the data extraction platform. The service data can be obtained only by inputting the service data table name and the service database name by a developer, the operation is simple and convenient, and the efficiency is higher.

Description

Technical Field

[0001] The present application relates to the fields of financial technology and computer technology, and in particular to a business data extraction method and device, storage medium, and computer equipment. Background Art

[0002] At present, when users handle various financial services (such as insurance claims or auto financing payments) on financial platforms, a large amount of business data is usually generated. Developers need to regularly obtain these business data to maintain the basic database of the financial platform, such as maintaining the label system, contract center, customer complaint files, etc. in the financial platform. When developers extract these business data, in the current data management and processing field, there are a variety of technologies and tools to support data extraction and transmission from operational data storage (ODS) to big data storage systems (such as HDFS, Hadoop Distributed File System). These technologies and tools include but are not limited to ETL (Extract Ransform Loader), data integration platform, custom scripts and automated workflows. Although these solutions meet the needs of data migration and processing to a certain extent, they still have some defects and shortcomings. For example, this traditional data extraction mode requires developers to provide detailed table fields and other contents, build tables first and then extract business data, which increases the complexity of personnel operations to a certain extent and has a greater probability of error. Summary of the invention

[0003] In view of this, the present application provides a business data extraction method and device, storage medium, and computer equipment, which collect business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, construct an extraction data table for the preset collection period, and store the extraction data table in the distributed file system of the data extraction platform; receive a business data extraction instruction, obtain the required business data in the distributed file system based on the required collection period, required business database name, and required business data table name corresponding to the business data extraction instruction, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform. The developer only needs to enter the business data table name and the business database name to obtain the business data, which is easy to operate and more efficient.

[0004] According to one aspect of the present application, a business data extraction method is provided, which is applied to a data extraction platform, and the method includes:

[0005] For the business data generated by the target business platform within a preset collection period, the business data and metadata corresponding to the business data are collected, wherein the business data is stored in a business database corresponding to the target business platform in the form of a business data table, and the metadata includes a business data table name and a business database name;

[0006] Based on the business database name, the business data table name and the business data, construct an extracted data table for the preset collection period, and store the extracted data table in a distributed file system of the data extraction platform;

[0007] Receive a business data extraction instruction, obtain the demand business data in the distributed file system based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction, and return the demand business data to the demand receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on a data extraction platform.

[0008] Optionally, constructing the extracted data table of the preset collection period based on the business database name, the business data table name and the business data includes:

[0009] Based on the preset collection period, the business database name, the business data table name and the business data, a table creation statement is concatenated, and an extracted data table of the preset collection period is obtained based on the table creation statement.

[0010] Optionally, the acquiring the demand business data in the distributed file system based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction includes:

[0011] Based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction, splicing data query statements;

[0012] In the distributed file system, required business data is acquired based on the data query statement.

[0013] Optionally, the distributed file system includes a buffer layer and an operational data storage layer; the distributed file system storing the extracted data table in the data extraction platform includes:

[0014] The extracted data table of the latest preset collection period is stored in the buffer layer of the distributed file system, and the extracted data table of the historical preset collection period is stored in the operational data storage layer of the distributed file system;

[0015] Accordingly, the step of acquiring the demand service data in the distributed file system based on the demand collection period, the demand service database name, and the demand service data table name corresponding to the service data extraction instruction includes:

[0016] When the demand preset collection period corresponding to the business data extraction instruction is the latest preset collection period, based on the demand collection period corresponding to the business data extraction instruction, the name of the demand business database and the name of the demand business data table, the demand business data is obtained in the buffer layer of the distributed file system;

[0017] When the demand preset collection period corresponding to the business data extraction instruction is a historical preset collection period, the demand business data is obtained in the operational data storage layer of the distributed file system based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction.

[0018] Optionally, the method further comprises:

[0019] In the operational data storage layer of the distributed file system, the extracted data table within the preset data cleaning period is cleared at every preset time interval, wherein the preset data cleaning period is greater than the preset collection period.

[0020] Optionally, the collecting of the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period includes:

[0021] For each business database corresponding to the target business platform, based on the business database address and business database type of the business database, splice the data collection statement;

[0022] Based on the data collection statement, the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period are collected.

[0023] Optionally, after receiving the service data extraction instruction, the method further includes:

[0024] In the distributed file system, for the demand collection time period and the demand business database name corresponding to the business data extraction instruction, when the demand business database name corresponding to the demand collection time period is not found to have the demand business data table name, a data extraction error prompt message is generated based on the demand collection time period, the demand business database name and the demand extraction business data table name, and the data extraction error prompt message is displayed based on the data extraction platform.

[0025] According to another aspect of the present application, a service data extraction device is provided, the device comprising:

[0026] A metadata collection module, for collecting the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, wherein the business data is stored in a business database corresponding to the target business platform in the form of a business data table, and the metadata includes a business data table name and a business database name;

[0027] An extracted data table construction module, used to construct an extracted data table for the preset collection period based on the business database name, the business data table name and the business data, and store the extracted data table in a distributed file system of the data extraction platform;

[0028] The business data extraction module is used to receive business data extraction instructions, obtain the required business data in the distributed file system based on the required collection period, required business database name and required business data table name corresponding to the business data extraction instruction, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform.

[0029] Optionally, the extracted data table construction module is further used to:

[0030] Based on the preset collection period, the business database name, the business data table name and the business data, a table creation statement is concatenated, and an extracted data table of the preset collection period is obtained based on the table creation statement.

[0031] Optionally, the business data extraction module is further used to:

[0032] Based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction, splicing data query statements;

[0033] In the distributed file system, required business data is acquired based on the data query statement.

[0034] Optionally, the distributed file system includes a buffer layer and an operational data storage layer; the device further includes: an extraction data table storage module, which is used to:

[0035] The extracted data table of the latest preset collection period is stored in the buffer layer of the distributed file system, and the extracted data table of the historical preset collection period is stored in the operational data storage layer of the distributed file system.

[0036] Optionally, the business data extraction module is further used to:

[0037] When the demand preset collection period corresponding to the business data extraction instruction is the latest preset collection period, based on the demand collection period corresponding to the business data extraction instruction, the name of the demand business database and the name of the demand business data table, the demand business data is obtained in the buffer layer of the distributed file system;

[0038] When the demand preset collection period corresponding to the business data extraction instruction is a historical preset collection period, the demand business data is obtained in the operational data storage layer of the distributed file system based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction.

[0039] Optionally, the device further comprises: an extraction data table cleaning module, configured to:

[0040] In the operational data storage layer of the distributed file system, the extracted data table within the preset data cleaning period is cleared at every preset time interval, wherein the preset data cleaning period is greater than the preset collection period.

[0041] Optionally, the metadata collection module is further used to:

[0042] For each business database corresponding to the target business platform, based on the business database address and business database type of the business database, splice the data collection statement;

[0043] Based on the data collection statement, the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period are collected.

[0044] Optionally, the device further comprises: a data extraction error prompt module, configured to:

[0045] In the distributed file system, for the demand collection time period and the demand business database name corresponding to the business data extraction instruction, when the demand business database name corresponding to the demand collection time period is not found to have the demand business data table name, a data extraction error prompt message is generated based on the demand collection time period, the demand business database name and the demand extraction business data table name, and the data extraction error prompt message is displayed based on the data extraction platform.

[0046] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned business data extraction method is implemented.

[0047] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned business data extraction method when executing the program.

[0048] By means of the above technical scheme, the present application provides a business data extraction method and device, storage medium, and computer equipment, which collect business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, construct an extraction data table for the preset collection period, and store the extraction data table in the distributed file system of the data extraction platform; receive a business data extraction instruction, obtain the required business data in the distributed file system based on the required collection period, required business database name, and required business data table name corresponding to the business data extraction instruction, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform. The developer only needs to enter the business data table name and the business database name to obtain the business data, which is easy to operate and more efficient.

[0049] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0051] Figure 1 A schematic diagram of a process flow of a business data extraction method provided in an embodiment of the present application is shown;

[0052] Figure 2 A schematic diagram of a business data requirement registration page provided in an embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of a process flow of another method for extracting business data provided by an embodiment of the present application is shown;

[0054] Figure 4 A schematic diagram of a process flow of another method for extracting business data provided by an embodiment of the present application is shown;

[0055] Figure 5 A schematic diagram of the structure of a business data extraction device provided in an embodiment of the present application is shown;

[0056] Figure 6 A schematic diagram of the structure of another business data extraction device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0058] In this embodiment, a method for extracting business data is provided. Figure 1 As shown, the method includes:

[0059] Step 101, collect the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, wherein the business data is stored in the business database corresponding to the target business platform in the form of a business data table, and the metadata includes the business data table name and the business database name.

[0060] At present, in the current data management and processing field, the maintenance cost of existing technologies is relatively high, and the technical staff's skills are required to be high, because they need to be continuously monitored and maintained by professionals. Most existing solutions focus on the single link of data extraction, and cannot provide full-process optimization from demand registration to data landing. With the growth of data volume and the diversification of data sources, the scalability of existing systems has become a bottleneck, making it difficult to cope with the challenges of large-scale data processing.

[0061] In the above embodiments of the present application, it can be applied to the field of financial technology. The target business platform is, for example, a financial platform. When a user logs in to the financial platform to handle financial business, business data will be generated. For example, when a user logs in to the financial platform to handle auto finance payment business, the processing time and payment amount and other data will be recorded. The preset collection period can be set to one day, that is, one day is used as a cycle to collect metadata of daily business data. In particular, it can be set to start collection at 10 o'clock every night, that is, to collect business data generated by the financial platform from 10 o'clock last night to 10 o'clock today. Specifically, for the business data generated by the target business platform within the preset collection period, the business data and the metadata corresponding to the business data are collected. The business data is stored in the business database corresponding to the target business platform in the form of a business data table. The metadata is used to describe the structure and fields of the business data table and other information, including the name of the business data table and the name of the business database stored in the business data table.

[0062] Step 102: construct an extracted data table for the preset collection period based on the business database name, the business data table name and the business data, and store the extracted data table in a distributed file system of a data extraction platform.

[0063] Next, based on the metadata (business database name, business data table name) and business data, a data extraction table for the preset collection period is constructed, and the extracted data table is stored in the distributed file system of the data extraction platform so that subsequent developers can directly extract business data based on the extracted data table in the distributed file system.

[0064] Step 103, receive a business data extraction instruction, obtain the required business data in the distributed file system based on the required collection period, required business database name and required business data table name corresponding to the business data extraction instruction, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform.

[0065] Next, when the developer triggers the business data extraction instruction based on the data extraction platform, for example, the developer logs in to the data extraction platform, enters (or selects) the name of the business database and the name of the business data table on the business data extraction requirement registration page of the data extraction platform, and then generates the business data extraction instruction. The business data extraction requirement registration page is as follows: Figure 2 As shown, in particular, when developers extract business data, they can also add an audit link, that is, when users submit business data extraction requirements on the business data extraction requirements registration page of the data extraction platform, they can first submit it to a specially set auditor, who will review it through the data extraction platform. After the review is passed, the auditor will return the required business data to the receiving terminal corresponding to the business data extraction instruction, or the data extraction platform will automatically return the required business data. At the same time, the data extraction requirement registration page can add options at any time according to the needs of data extraction. For example, in addition to registering the demand collection period, the name of the demand business database and the name of the demand business data table, you can also register the project team where the developer is located, the demand description and notes of this data extraction, etc., so that the auditor can review it. Then, based on the demand collection period, the name of the demand business database and the name of the demand business data table corresponding to the business data extraction instruction, the required business data is obtained in the distributed file system, and the required business data is returned to the demand receiving terminal corresponding to the business data extraction instruction. Compared with the traditional data extraction mode, which requires developers to provide detailed field content information, this application establishes an "extraction data table" so that developers only need to provide the required business database name and the required business data table name to obtain the required business data. The process is simple, highly operational, and also improves data extraction efficiency.

[0066] By applying the technical solution of this embodiment, for the business data generated by the target business platform within the preset collection period, the business data and the metadata corresponding to the business data are collected, the extraction data table of the preset collection period is constructed, and the extracted data table is stored in the distributed file system of the data extraction platform; the business data extraction instruction is received, and the required business data is obtained in the distributed file system based on the required collection period, the required business database name and the required business data table name corresponding to the business data extraction instruction, and the required business data is returned to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform. The developer only needs to enter the business data table name and the business database name to obtain the business data, which is simple to operate and more efficient, and realizes the automation of the entire process from demand registration to data extraction, transmission and landing, which significantly reduces manual intervention and operational errors.

[0067] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another business data extraction method is provided, such as Figure 3 As shown, the method includes:

[0068] Step 201 , for each business database corresponding to the target business platform, based on the business database address and business database type of the business database, splice data collection statements.

[0069] Step 202, based on the data collection statement, collect the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, wherein the business data is stored in the business database corresponding to the target business platform in the form of a business data table, and the metadata includes the business data table name and the business database name.

[0070] In the above embodiment of the present application, for each business database corresponding to the target business platform, the target business platform is such as a financial platform, business; such as auto finance payment, etc., and the business database is such as an auto finance contract center library, an auto finance report library, etc. Based on the preset collection time period (for example, the time period between 10 pm on the current day and 10 pm on the previous day), the business database address of the business database (for example, the address of the auto finance contract center library and the address of the auto finance report library) and the business database type (Oracle library, PG library and My SQL library), the data collection statement is spliced. Then, based on the data collection statement, the business data generated by the target business platform within the preset collection period and the metadata corresponding to the business data are collected. In particular, the business data is stored in the business database corresponding to the target business platform in the form of a business data table. The metadata includes the business data table name and the business database name, in preparation for the subsequent extraction of the data table establishment.

[0071] Specifically, first configure the collected business database information, the configured information includes the type and address of the business database, so that the collection program can automatically identify and use the appropriate collection template. Then, configure the SQL template of the collection database, execute the collection task, and collect different types of databases separately, that is, collect Oracle library metadata, collect PG library metadata, and collect MySQL library metadata. At this time, the main information collected is the library name, table name, column name, table comment, column comment, field type, field length, etc. of the business database. The collection frequency can be set to automatically collect at 10 o'clock every night, and the collection program can also be manually triggered. The collected metadata is the business data between 10 o'clock in the evening of the day and 10 o'clock in the evening of the previous day, corresponding to a preset collection period, and the length of the preset collection period can also be changed according to different application scenarios. By collecting metadata, table creation statements, query statements, insert statements, etc. can be automatically generated later, and finally the extracted data table is obtained.

[0072] Step 203, based on the preset collection period, the business database name, the business data table name and the business data, splice the table creation statement, obtain the extracted data table of the preset collection period based on the table creation statement, and store the extracted data table in the distributed file system of the data extraction platform.

[0073] Next, the collected metadata can be stored in the PG library of the data extraction platform, and then based on the preset collection period, business database name, business data table name and business data, the table building statement is spliced, and the extracted data table of the preset collection period is obtained based on the table building statement, and the extracted data table is stored in the distributed file system of the data extraction platform. Specifically, according to the business database name and business data table name passed in by the front end, the function or method is called to generate the table building statement, query statement and insert statement, and the extracted data table is completed into a file through the query statement, and then the built extracted data table can be automatically pushed to hdfs (distributed file system) through the kit l eETL tool. In particular, in the process of establishing the extracted data table, since the distributed file system address is specified, after the extracted data table is pushed to the specified address (distributed file system), it can be accessed on the page side through tools such as HUE, which is convenient for developers to register business data extraction requirements at any time and obtain required business data.

[0074] Step 204, receiving a business data extraction instruction, and splicing a data query statement based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction.

[0075] Next, receive the business data extraction instruction, and splice the data query statement based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction. Specifically, the demand registration can be developed into a registerable web page through Django technology, such as Figure 2 As shown, developers or demand analysts only need to enter the business system tables involved in their needs (demand collection period, demand business database name and demand business data table name) on this page, without paying attention to the specific table structure information. After registration is completed, if an audit link is added, the ODS person in charge will be automatically notified by email. After the ODS person in charge receives the email, he will complete the processing (review) and modify the status on the page. The registrant will receive an email indicating that the processing is completed. In the above process, the front-end page only requires developers to enter: database / table / project team (project group) / brief description of the purpose of the table / remarks and other information, which is convenient for developers to the greatest extent. The traditional model may require developers to provide detailed information such as table fields to build the table first, and then extract it. The process is cumbersome, inefficient, and prone to errors.

[0076] Step 205, in the distributed file system, the required business data is obtained based on the data query statement, and the required business data is returned to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform.

[0077] Next, in the distributed file system, the required business data is obtained based on the data query statement, and the required business data is returned to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform. Considering that data needs to be used in a timely manner in the daily development process, a temporary database can be created in the production environment for developers to temporarily synchronize data on the data extraction platform to quickly support the data needs of developers. Formal data can be configured to be extracted in the scheduling task to achieve logical isolation of data and avoid affecting the production environment. At the same time, the temporary database data can meet the user's data verification and testing and solve the user's number needs. Specifically, considering that the urgency of table data is high in the development process of some project teams, if the data is extracted by relying on the person in charge of ods, there is a large time uncertainty. In this case, after the page end table is entered, the person who registers the table can execute it with one click, and the table data can be automatically synchronized to the temporary library environment for temporary use by the development teacher. After completing the table registration, the person in charge of ods has completed the review and can publish it to the scheduling task with one click. Since it may be necessary to access thousands of tables from the target business platform, it is impossible to extract each table manually. It is necessary to automatically publish the demand table registered by the developer to the scheduling tool platform to achieve automatic synchronization of data tasks to the data center hi ve at the specified time every day. To this end, the automation and optimization design reduces the dependence on professional technicians and reduces the maintenance cost of the system. The extraction personnel only need to have basic sql knowledge, which is reflected in:

[0078] 1. In the process of data synchronization, the code is encapsulated and only three parameters, namely database name, table name and flag bit, are exposed to the outside. The registrant only needs to select the correct database and table on the registration page, and the ODS person in charge only needs to be concerned with the review process without manually obtaining relevant information from the business system table.

[0079] 2. When logging into a table, there will be verification and prompts for registered tables or registered tables that do not exist in the business system.

[0080] 3. Easy to expand: If you want to change the full amount to incremental amount, you only need to modify the configuration on the page.

[0081] 4. Full process optimization: The full process automation from demand registration to release and launch is realized.

[0082] Step 206, in the distributed file system, for the demand collection time period and the demand business database name corresponding to the business data extraction instruction, when the demand business database name for the demand collection time period does not correspond to the demand business data table name, a data extraction error prompt message is generated based on the demand collection time period, the demand business database name and the demand extraction business data table name, and the data extraction error prompt message is displayed based on the data extraction platform.

[0083] Next, in the distributed file system, for the demand collection period and demand business database name corresponding to the business data extraction instruction, when the demand business database name corresponding to the demand collection period is not found to have a demand business data table name, based on the demand collection period, demand business database name and demand extraction business data table name, data extraction error prompt information is generated, and the data extraction error prompt information is displayed based on the data extraction platform. To this end, by adding error prompts, when developers register errors on the registration page, they can know and correct them immediately, which is conducive to improving data extraction efficiency.

[0084] By applying the technical solution of this embodiment, for each business database corresponding to the target business platform, based on the preset collection period, the business database address and the business database type of the business database, the data collection statements are spliced, based on the data collection statements, the business data and metadata corresponding to the business data generated by the target business platform within the preset collection period are collected, based on the preset collection period, the business database name, the business data table name and the business data, the table building statements are spliced, based on the preset collection period, the business database name, the business data table name and the business data, the extracted data table of the preset collection period is obtained based on the table building statements, and the extracted data table is stored in the distributed file system of the data extraction platform. The business data extraction instruction is received, and the data query statement is spliced ​​based on the required collection period, the required business database name and the required business data table name corresponding to the business data extraction instruction. In the distributed file system, the required business data is obtained based on the data query statement, and the required business data is returned to the required receiving terminal corresponding to the business data extraction instruction. When the required business database name corresponding to the required collection period and the required business database name in the distributed file system is not found to correspond to the required business data table name, based on the required collection period, the required business database name and the required extracted business data table name, a data extraction error prompt message is generated, and the data extraction error prompt message is displayed based on the data extraction platform. The developer only needs to enter the business data table name and the business database name to obtain the business data, which is easy to operate and efficient.

[0085] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another business data extraction method is provided, such as Figure 4As shown, the method includes:

[0086] Step 301, collect the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, wherein the business data is stored in the business database corresponding to the target business platform in the form of a business data table, and the metadata includes the business data table name and the business database name.

[0087] In the above embodiment of the present application, for the business data generated by the target business platform within a preset collection period, the business data and the metadata corresponding to the business data are collected. For example, it can be set to 10 o'clock every night to collect the metadata corresponding to the business data generated by the target business platform from 10 o'clock the previous night to 10 o'clock this evening. In particular, the business data is stored in the business database corresponding to the target business platform in the form of a business data table, and the metadata includes the business data table name and the business database name.

[0088] Step 302: construct an extracted data table for the preset collection period based on the business database name, the business data table name and the business data.

[0089] Step 303, stores the extracted data table of the latest preset collection period in the buffer layer of the distributed file system, and stores the extracted data table of the historical preset collection period in the operational data storage layer of the distributed file system, wherein the distributed file system includes a buffer layer and an operational data storage layer.

[0090] Next, based on the business database name, business data table name and business data, construct the extracted data table of the preset collection period. The extracted data table of the latest preset collection period is stored in the buffer layer of the distributed file system, and the extracted data table of the historical preset collection period is stored in the operational data storage layer of the distributed file system. Specifically, after collecting metadata, a query statement is automatically generated to realize the subsequent storage of business database data on the local disk, and then the local file is pushed to the specified directory of the distributed storage system. Then, the buffer layer (stage layer) and operational data storage layer (ods layer) table building statements are automatically generated, and then the buffer layer data is automatically generated. Insert statements are inserted into the partition table of the operational data storage layer every day (every preset collection period). In particular, the extracted data table of the latest preset collection period is stored in the buffer layer of the distributed file system, and the extracted data table of the historical preset collection period is stored in the operational data storage layer of the distributed file system.

[0091] Step 304, receiving a business data extraction instruction. When the preset collection period corresponding to the business data extraction instruction is the latest preset collection period, based on the required collection period, required business database name and required business data table name corresponding to the business data extraction instruction, the required business data is obtained in the buffer layer of the distributed file system.

[0092] Step 305, when the demand preset collection period corresponding to the business data extraction instruction is a historical preset collection period, based on the demand collection period corresponding to the business data extraction instruction, the demand business database name and the demand business data table name, the demand business data is obtained in the operational data storage layer of the distributed file system.

[0093] Step 306: Return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform.

[0094] Step 307: In the operational data storage layer of the distributed file system, clear the extracted data table within a preset data cleaning period at every preset time interval, wherein the preset data cleaning period is greater than the preset collection period.

[0095] Next, a business data extraction instruction is received. When the preset demand collection period corresponding to the business data extraction instruction is the latest preset collection period, based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction, the demand business data is obtained in the buffer layer of the distributed file system. In particular, when extracting business data, when the preset demand collection period corresponding to the business data extraction instruction is the historical preset collection period, based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction, the demand business data is obtained in the operational data storage layer of the distributed file system. The demand business data is returned to the demand receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform. In particular, in order to regularly clean up space, the extracted data table in the preset data cleaning period can also be cleared at every preset time interval in the operational data storage layer of the distributed file system, wherein the preset data cleaning period is greater than the preset collection period. For example, after storing the extracted data table in the operational data storage layer every day, the historical extracted data table within the previous three months can be cleared in a cycle of three months.

[0096] To this end, the speed and efficiency of data processing are improved by automating the entire process of demand registration, data extraction, transmission and landing. This means that data can be transferred from the source system to the target system more quickly, thereby speeding up the process of data analysis and decision-making. Automated processes reduce the need for manual operations, thereby reducing errors and inconsistencies caused by manual intervention. This not only improves the accuracy of data processing, but also reduces the workload of developers and data engineers. Automated and optimized designs reduce dependence on professional technicians, thereby reducing the long-term maintenance costs of the system. Enterprises can reallocate resources to other key business areas and improve overall operational efficiency. Ultimately, these technical improvements can translate into a better user experience, whether it is the data platform used by internal employees or customer-oriented services, which can be improved due to the efficient processing of data. As data management and processing become more efficient and reliable, enterprises have more opportunities to explore new business models and innovations, driving the development of the entire industry.

[0097] By applying the technical solution of this embodiment, for the business data generated by the target business platform within the preset collection period, the business data and the metadata corresponding to the business data are collected, and the extracted data table of the preset collection period is constructed based on the business database name, the business data table name and the business data. The extracted data table of the latest preset collection period is stored in the buffer layer of the distributed file system, and the extracted data table of the historical preset collection period is stored in the operational data storage layer of the distributed file system, and the business data extraction instruction is received. When the required preset collection period corresponding to the business data extraction instruction is the latest preset collection period, the required business data is obtained in the buffer layer of the distributed file system based on the required collection period, the required business database name and the required business data table name corresponding to the business data extraction instruction. When the required preset collection period corresponding to the business data extraction instruction is the historical preset collection period, the required business data is obtained in the operational data storage layer of the distributed file system based on the required collection period, the required business database name and the required business data table name corresponding to the business data extraction instruction. The required business data is returned to the required receiving terminal corresponding to the business data extraction instruction. In the operational data storage layer of the distributed file system, the extracted data table in the preset data cleaning period is cleared at preset time intervals, wherein the preset data cleaning period is greater than the preset collection period. The developer only needs to enter the business data table name and business database name to obtain the business data, which is easy to operate and efficient.

[0098] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a business data extraction device, such as Figure 5 As shown, the device comprises:

[0099] The metadata collection module 401 is used to collect the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, wherein the business data is stored in the business database corresponding to the target business platform in the form of a business data table, and the metadata includes the business data table name and the business database name;

[0100] An extracted data table construction module 402 is used to construct an extracted data table for the preset collection period based on the business database name, the business data table name and the business data, and store the extracted data table in a distributed file system of a data extraction platform;

[0101] The business data extraction module 403 is used to receive business data extraction instructions, obtain the required business data in the distributed file system based on the required collection period, required business database name and required business data table name corresponding to the business data extraction instruction, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform.

[0102] Optionally, the extracted data table construction module 402 is further used to:

[0103] Based on the preset collection period, the business database name, the business data table name and the business data, a table creation statement is concatenated, and an extracted data table of the preset collection period is obtained based on the table creation statement.

[0104] Optionally, the business data extraction module 403 is further used to:

[0105] Based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction, splicing data query statements;

[0106] In the distributed file system, required business data is acquired based on the data query statement.

[0107] Optionally, the business data extraction module 403 is further used to:

[0108] When the demand preset collection period corresponding to the business data extraction instruction is the latest preset collection period, based on the demand collection period corresponding to the business data extraction instruction, the name of the demand business database and the name of the demand business data table, the demand business data is obtained in the buffer layer of the distributed file system;

[0109] When the demand preset collection period corresponding to the business data extraction instruction is a historical preset collection period, the demand business data is obtained in the operational data storage layer of the distributed file system based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction.

[0110] Optionally, the metadata collection module 401 is further used to:

[0111] For each business database corresponding to the target business platform, based on the business database address and business database type of the business database, splice the data collection statement;

[0112] Based on the data collection statement, the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period are collected.

[0113] Furthermore, the present application embodiment provides another service data extraction device, such as Figure 6 As shown, the device comprises:

[0114] The metadata collection module 501 is used to collect the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, wherein the business data is stored in the business database corresponding to the target business platform in the form of a business data table, and the metadata includes the business data table name and the business database name;

[0115] An extracted data table construction module 502 is used to construct an extracted data table for the preset collection period based on the business database name, the business data table name and the business data, and store the extracted data table in a distributed file system of a data extraction platform;

[0116] The business data extraction module 503 is used to receive a business data extraction instruction, obtain the required business data in the distributed file system based on the required collection period, the required business database name and the required business data table name corresponding to the business data extraction instruction, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform;

[0117] The extracted data table storage module 504 is used to store the extracted data table of the latest preset collection period in the buffer layer of the distributed file system, and store the extracted data table of the historical preset collection period in the operational data storage layer of the distributed file system, wherein the distributed file system includes the buffer layer and the operational data storage layer;

[0118] The extracted data table cleaning module 505 is used to clean up the extracted data table in a preset data cleaning period at a preset time interval in the operational data storage layer of the distributed file system, wherein the preset data cleaning period is greater than the preset collection period;

[0119] The data extraction error prompt module 506 is used to generate data extraction error prompt information in the distributed file system for the demand collection time period and the demand business database name corresponding to the business data extraction instruction, when the demand business database name of the demand collection time period does not correspond to the demand business data table name, based on the demand collection time period, the demand business database name and the demand extraction business data table name, and display the data extraction error prompt information based on the data extraction platform.

[0120] Optionally, the extracted data table construction module 502 is further used to:

[0121] Based on the preset collection period, the business database name, the business data table name and the business data, a table creation statement is concatenated, and an extracted data table of the preset collection period is obtained based on the table creation statement.

[0122] Optionally, the business data extraction module 503 is further used to:

[0123] Based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction, splicing data query statements;

[0124] In the distributed file system, required business data is acquired based on the data query statement.

[0125] Optionally, the business data extraction module 503 is further used to:

[0126] When the demand preset collection period corresponding to the business data extraction instruction is the latest preset collection period, based on the demand collection period corresponding to the business data extraction instruction, the name of the demand business database and the name of the demand business data table, the demand business data is obtained in the buffer layer of the distributed file system;

[0127] When the demand preset collection period corresponding to the business data extraction instruction is a historical preset collection period, the demand business data is obtained in the operational data storage layer of the distributed file system based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction.

[0128] Optionally, the metadata collection module 501 is further used to:

[0129] For each business database corresponding to the target business platform, based on the business database address and business database type of the business database, splice the data collection statement;

[0130] Based on the data collection statement, the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period are collected.

[0131] It should be noted that for other corresponding descriptions of the functional units involved in the business data extraction device provided in the embodiment of the present application, reference can be made to Figure 1 , Figure 3 and Figure 4 The corresponding description in the method will not be repeated here.

[0132] Based on the above Figure 1 , Figure 3 and Figure 4 The method shown in the embodiment of the present application is accordingly provided with a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned Figure 1 , Figure 3 and Figure 4 The business data extraction method shown.

[0133] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0134] Based on the above Figure 1 , Figure 3 and Figure 4 The method shown, and Figure 5 and Figure 6 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 , Figure 3 and Figure 4 The business data extraction method shown.

[0135] Optionally, the computer device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display screen (Displ ay), an input unit such as a keyboard (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0136] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not limit the computer device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0137] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and saves the hardware and software resources of the computer device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the physical device.

[0138] Through the description of the above implementation methods, the technical personnel in this field can clearly understand that the present application can be implemented by means of software plus necessary general hardware platforms, or can be implemented by hardware for the business data generated by the target business platform within the preset collection period, collect business data and metadata corresponding to the business data, construct an extraction data table for the preset collection period, and store the extracted data table in the distributed file system of the data extraction platform; receive a business data extraction instruction, based on the required collection period, required business database name and required business data table name corresponding to the business data extraction instruction, obtain the required business data in the distributed file system, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform. The developer only needs to enter the business data table name and the business database name to obtain the business data, which is easy to operate and efficient.

[0139] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0140] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A method for extracting business data, characterized in that: Applied to a data extraction platform, the method comprises: For the business data generated by the target business platform within a preset collection period, the business data and metadata corresponding to the business data are collected, wherein the business data is stored in a business database corresponding to the target business platform in the form of a business data table, and the metadata includes a business data table name and a business database name; Based on the business database name, the business data table name and the business data, construct an extracted data table for the preset collection period, and store the extracted data table in a distributed file system of the data extraction platform; Receive a business data extraction instruction, obtain the demand business data in the distributed file system based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction, and return the demand business data to the demand receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on a data extraction platform.

2. The method according to claim 1, characterized in that: The step of constructing the extracted data table for the preset collection period based on the business database name, the business data table name and the business data includes: Based on the preset collection period, the business database name, the business data table name and the business data, a table creation statement is concatenated, and an extracted data table of the preset collection period is obtained based on the table creation statement.

3. The method according to claim 1, characterized in that The acquiring the demand business data in the distributed file system based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction includes: Based on the demand collection period, demand business database name and demand business data table name corresponding to the business data extraction instruction, splicing data query statements; In the distributed file system, required business data is acquired based on the data query statement.

4. The method according to claim 1, characterized in that: The distributed file system includes a buffer layer and an operational data storage layer; the distributed file system storing the extracted data table in the data extraction platform includes: The extracted data table of the latest preset collection period is stored in the buffer layer of the distributed file system, and the extracted data table of the historical preset collection period is stored in the operational data storage layer of the distributed file system; Accordingly, the step of acquiring the demand service data in the distributed file system based on the demand collection period, the demand service database name, and the demand service data table name corresponding to the service data extraction instruction includes: When the demand preset collection period corresponding to the business data extraction instruction is the latest preset collection period, based on the demand collection period corresponding to the business data extraction instruction, the name of the demand business database and the name of the demand business data table, the demand business data is obtained in the buffer layer of the distributed file system; When the demand preset collection period corresponding to the business data extraction instruction is a historical preset collection period, the demand business data is obtained in the operational data storage layer of the distributed file system based on the demand collection period, the demand business database name and the demand business data table name corresponding to the business data extraction instruction.

5. The method according to claim 4, characterized in that The method further comprises: In the operational data storage layer of the distributed file system, the extracted data table within the preset data cleaning period is cleared at every preset time interval, wherein the preset data cleaning period is greater than the preset collection period.

6. The method according to claim 1, characterized in that The collecting of the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period includes: For each business database corresponding to the target business platform, based on the business database address and business database type of the business database, splice the data collection statement; Based on the data collection statement, the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period are collected.

7. The method according to any one of claims 1 to 6, characterized in that After receiving the service data extraction instruction, the method further includes: In the distributed file system, for the demand collection time period and the demand business database name corresponding to the business data extraction instruction, when the demand business database name corresponding to the demand collection time period is not found to have the demand business data table name, a data extraction error prompt message is generated based on the demand collection time period, the demand business database name and the demand extraction business data table name, and the data extraction error prompt message is displayed based on the data extraction platform.

8. A business data extraction device, characterized in that: The device comprises: A metadata collection module, for collecting the business data and metadata corresponding to the business data generated by the target business platform within a preset collection period, wherein the business data is stored in a business database corresponding to the target business platform in the form of a business data table, and the metadata includes a business data table name and a business database name; An extracted data table construction module, used to construct an extracted data table for the preset collection period based on the business database name, the business data table name and the business data, and store the extracted data table in a distributed file system of the data extraction platform; The business data extraction module is used to receive business data extraction instructions, obtain the required business data in the distributed file system based on the required collection period, required business database name and required business data table name corresponding to the business data extraction instruction, and return the required business data to the required receiving terminal corresponding to the business data extraction instruction, wherein the business data extraction instruction is triggered based on the data extraction platform.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting business data described in any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method for extracting business data described in any one of claims 1 to 7 is implemented.

Images