Bill data processing method, device and system

Abstract

The application discloses a kind of bill data processing method, device and system, it is related to big data field.The method comprises: obtaining bill data from bill management system, and write bill data into first relational database;Periodically determine the first incremental data in first relational database, and first incremental data is imported into data lake public zone;First incremental data is loaded to the source data table in data lake public zone, wherein the source data table includes the bill information table of multiple dimensions;Second incremental data in each bill information table is processed to the bill information wide table corresponding to the dimension of bill information table in data lake private zone;In response to the query operation of terminal, according to multiple bill information wide table, the target data report corresponding to the query operation is generated, and the target data report is fed back to terminal.By the present application, the technical problem that bill data processing efficiency is lower in related art, it is difficult to meet the technical problem of user demand.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a method, apparatus, and system for processing invoice data. Background Technology

[0002] Financial institutions' bill management systems typically require extensive statistical query and analysis functions. These requirements necessitate a project plan from developers, with each requirement requiring the development of a corresponding query function. This consumes significant development resources, and the time required for project implementation is substantial, hindering timely fulfillment of business needs. Furthermore, the increasing volume of online bill transactions year by year makes complex online analysis relatively time-consuming.

[0003] Meanwhile, in the bill business management system of related technologies, when salespersons or analysts create corresponding reports, they need to deal with hundreds of bill business tables from the source application and figure out the complex relationship logic, resulting in low data processing efficiency. Although some technologies also provide relatively automated bill data processing methods, they often process bill data on a daily basis, which is not only inefficient but also has high latency. Users can often only query the bill business data of the previous day, which is difficult to meet application needs.

[0004] There is currently no effective solution to the technical problem of low efficiency in processing invoice data, which makes it difficult to meet user needs. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, and system for processing invoice data, in order to solve the technical problem that the efficiency of invoice data processing in related technologies is low and it is difficult to meet user needs.

[0006] According to one aspect of this application, a method for processing invoice data is provided. The method includes: acquiring invoice data from an invoice management system and writing the invoice data into a first relational database; periodically determining first incremental data in the first relational database and importing the first incremental data into a public area of ​​a data lake; loading the first incremental data into a source data table within the public area of ​​the data lake, wherein the source data table includes multiple dimensions of invoice information tables; processing second incremental data from each invoice information table into a wide invoice information table corresponding to the dimensions of the invoice information tables within a private area of ​​the data lake; and, in response to a query operation from a terminal, generating a target data report corresponding to the query operation based on the multiple wide invoice information tables, and feeding the target data report back to the terminal.

[0007] Optionally, periodically determining the first incremental data in the first relational database and importing the first incremental data into the public area of ​​the data lake includes: reading the archived logs of the first relational database according to a preset time period; capturing change data in the archived logs to obtain the first incremental data, wherein the first incremental data is the incremental data between the previous reading time and the current reading time, and the first incremental data includes incremental files of multiple dimensions; synchronizing the first incremental data to the Kafka message queue, wherein each incremental file corresponds to a topic in the Kafka message queue; and importing the first incremental data into the public area of ​​the data lake through the Kafka message queue.

[0008] Optionally, after determining the first incremental data in the first relational database, for each incremental file in the first incremental data, a verification file corresponding to the incremental file is generated, wherein the verification file includes the file attribute information of the incremental file; and the verification file is sent to the public area of ​​the data lake.

[0009] Optionally, the first incremental data is loaded into the source data table within the public area of ​​the data lake, including: for each incremental file in the first incremental data, the incremental file is verified according to the verification file corresponding to the incremental file; when the incremental file passes the verification, the incremental file is added to the invoice information table corresponding to the dimension of the incremental file.

[0010] Optionally, within the private area of ​​the data lake, the second incremental data from each invoice information table is processed into a wide invoice information table corresponding to the dimension of the invoice information table. This includes: within the private area of ​​the data lake, using the Spark computing engine to process the second incremental data from each invoice information table into a wide invoice information table corresponding to the dimension of the invoice information table; wherein, when processing each wide invoice information table, the amount of system resources allocated to the thread processing the wide invoice information table is determined based on the total data capacity of the second incremental data in the invoice information table corresponding to the dimension of the wide invoice information table.

[0011] Optionally, within the private area of ​​the data lake, the second incremental data from each bill information table is processed into a bill information wide table corresponding to the dimensions of the bill information table. This includes: for each bill information wide table to be processed, determining the type of the bill information wide table; when the bill information wide table is a theme aggregation wide table, merging the second incremental data from the bill information table corresponding to the dimensions of the bill information wide table with the existing data in the bill information wide table to obtain the full bill information wide table; when the bill information wide table is an indicator aggregation wide table, calculating incremental indicators based on the second incremental data from the bill information table corresponding to the dimensions of the bill information wide table, and merging the incremental indicators with the existing indicators in the bill information wide table to obtain the full bill information wide table.

[0012] Optionally, within the private area of ​​the data lake, the second incremental data from each bill information table is processed into a bill information wide table corresponding to the dimension of the bill information table. This includes: classifying the second incremental data according to the institution to which it belongs, resulting in third incremental data belonging to the target institution and fourth incremental data not belonging to the target institution; and processing the third and fourth incremental data into the bill information wide table respectively, wherein the system resources allocated for processing the third incremental data are greater than the system resources allocated for processing the fourth incremental data.

[0013] Optionally, after determining the first incremental data in the first relational database, the fifth incremental data in the second relational database associated with the first relational database is determined, and the fifth incremental data is imported into the public area of ​​the data lake; the fifth incremental data is then loaded into the source data table within the public area of ​​the data lake.

[0014] Optionally, the dimensions of the bill information table include: basic bill information table, bill endorsement information table, bill acceptance information table, bill purchase information table, bill sale information table, bill payment reminder table, and corporate customer information table; the types of bill information wide tables include: thematic aggregation wide tables and indicator aggregation wide tables. The dimensions of thematic aggregation wide tables include: customer summary information wide table, institution summary information wide table, bill purchase transaction details wide table, bill sale transaction details wide table, bill inventory details wide table, bill pool business details wide table, bill endorsement transaction details wide table, bill circulation details wide table, bill acceptance details wide table, bill payment reminder wide table, bill pledge details wide table, bill recovery processing details wide table, and bill recovery processing details wide table. The dimensions of indicator aggregation wide tables include: bill customer indicator wide table and bill institution indicator wide table.

[0015] According to another aspect of this application, a bill data processing apparatus is also provided, comprising: an acquisition module for acquiring bill data from a bill management system and writing the bill data into a first relational database; a determination module for periodically determining first incremental data in the first relational database and importing the first incremental data into a public area of ​​a data lake; a loading module for loading the first incremental data into a source data table in the public area of ​​the data lake, wherein the source data table includes multiple dimensions of bill information tables; a processing module for processing second incremental data from each bill information table into a bill information wide table corresponding to the dimensions of the bill information table in a private area of ​​the data lake; and a query module for responding to a query operation from a terminal, generating a target data report corresponding to the query operation based on multiple bill information wide tables, and feeding back the target data report to the terminal.

[0016] According to another aspect of this application, a bill data processing system is also provided, the system comprising: a bill management system, a bill data platform, and a terminal; wherein, the bill data platform comprises: a first relational database and a distributed storage cluster, the distributed storage cluster comprising: a public area of ​​a data lake and a private area of ​​a data lake; the bill data platform is used to execute the above-described bill data processing method.

[0017] According to another aspect of this application, a processor is also provided for running a program, wherein the program executes the above-described ticket data processing method during runtime.

[0018] According to another aspect of this application, an electronic device is also provided, which includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the above-described ticket data processing method.

[0019] In this application, invoice data is first obtained from the invoice management system and written into a first relational database. Then, the first incremental data in the first relational database is periodically determined and imported into the public area of ​​the data lake. Within the public area of ​​the data lake, the first incremental data is loaded into a source data table, which includes multiple dimensions of invoice information tables. In the private area of ​​the data lake, the second incremental data from each invoice information table is processed into a wide invoice information table corresponding to the dimensions of the invoice information table. Finally, in response to terminal query operations, a target data report corresponding to the query operation is generated based on multiple wide invoice information tables, and the target data report is fed back to the terminal. By periodically importing incremental invoice data from the invoice management system into the source data table in the public area of ​​the data lake, and processing the data in the private area of ​​the data lake using a combination of existing and incremental data, the processing efficiency of invoice data can be greatly improved, thus solving the technical problem of low invoice data processing efficiency in related technologies, which makes it difficult to meet user needs. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram of the structure of an optional invoice data processing system provided according to an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating an optional invoice data processing method according to an embodiment of this application;

[0023] Figure 3This is a schematic diagram of an optional data ingestion process provided according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of an inventory analysis report generated based on a wide table of invoice inventory details, according to an embodiment of this application.

[0025] Figure 5 This is an interactive schematic diagram of an optional invoice data processing system provided according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of an optional invoice data processing device according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of an optional electronic device provided according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. Where there is no conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0031] Kafka is a distributed message queue that categorizes messages by topic. Message senders are called producers, and message receivers are called consumers. In addition, a Kafka cluster consists of multiple Kafka instances, and each instance server is called a broker.

[0032] Change Data Capture (CDC) is the process of identifying and capturing changes made to data in a database (including insertions, updates, and deletions of data or tables), then recording these changes in chronological order and transmitting them in real time to downstream processes or systems via message brokers. In this way, CDC provides efficient, low-latency data transmission to the data warehouse, enabling information to be promptly transformed and delivered to applications dedicated to analysis.

[0033] Spark: A fast and versatile computing engine designed for large-scale data processing. Its intermediate outputs can be stored in memory, eliminating the need to read distributed file systems, resulting in higher data processing efficiency and making it better suited for iterative distributed computing algorithms such as data mining and machine learning.

[0034] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, if there is an interface between this system and the relevant user or organization, before obtaining the relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information from the aforementioned user or organization.

[0035] Figure 1 This is a schematic diagram of an optional invoice data processing system provided according to an embodiment of this application, such as... Figure 1 As shown, the system includes: a bill management system 11, a bill data platform 12, and a terminal 13. The bill data platform 12 includes: a first relational database 121 and a distributed storage cluster 122. The distributed storage cluster 122 includes: a public area of ​​the data lake 1221 and a private area of ​​the data lake 1222.

[0036] Among them, the bill management system 11 can be a system in various financial institutions that is responsible for bill transactions and circulation, and it stores a large amount of bill data information.

[0037] The bill data platform 12 mainly processes, organizes, and stores bill data information to facilitate subsequent bill business queries and calls. The first relational database 121 can be an Oracle database corresponding to the bill management system 11, and the distributed storage cluster 122 can be a Hadoop cluster. Optionally, the bill data platform 12 may also include a second relational database corresponding to other business systems associated with the bill management system 11, such as the Oracle databases corresponding to the customer management system and the credit management system.

[0038] Terminal 13 can be a mobile phone, tablet, computer or other terminal device, which can query relevant bill business information through the provided interactive interface.

[0039] In related technologies, invoice data is usually processed on a daily basis. For example, all invoice data for the day is imported into a data lake for processing every night. Due to the large amount of data, the processing efficiency is low and the data latency is high. Users can often only query the invoice business data of the previous day, which is difficult to meet application needs.

[0040] To address the aforementioned issues, this application introduces a first relational database. By writing the invoice data from the invoice management system into the first relational database, periodically determining the incremental data within it, and importing it into the public area of ​​the data lake, near real-time data import into the data lake can be achieved. Subsequently, data processing is performed in the private area of ​​the data lake using a combination of existing and incremental data, which can reduce the amount of data that needs to be processed each time, thereby greatly improving the efficiency of invoice data processing.

[0041] The following section provides a detailed explanation of the process executed by the invoice data platform, using the preferred implementation steps as an example.

[0042] Figure 2 This is a flowchart illustrating an optional invoice data processing method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0043] Step S201: Obtain invoice data from the invoice management system and write the invoice data into the first relational database;

[0044] Step S202: Periodically determine the first incremental data in the first relational database and import the first incremental data into the public area of ​​the data lake;

[0045] Step S203: Load the first incremental data into the source data table in the public area of ​​the data lake. The source data table includes multiple dimensions of invoice information tables.

[0046] Step S204: In the private area of ​​the data lake, process the second incremental data of each bill information table into the bill information wide table corresponding to the dimension of the bill information table;

[0047] Step S205: In response to the terminal's query operation, generate a target data report corresponding to the query operation based on multiple wide tables of invoice information, and feed the target data report back to the terminal.

[0048] As an optional implementation, when periodically determining the first incremental data in the first relational database and importing the first incremental data into the public area of ​​the data lake, it can be done as follows: read the archived logs of the first relational database according to a preset time period; capture change data in the archived logs to obtain the first incremental data, wherein the first incremental data is the incremental data between the previous reading time and the current reading time, and the first incremental data includes incremental files of multiple dimensions; synchronize the first incremental data to the Kafka message queue, wherein each incremental file corresponds to a topic in the Kafka message queue; and import the first incremental data into the public area of ​​the data lake through the Kafka message queue.

[0049] In order to improve the real-time performance of the data, the preset time period is usually short, such as 10 minutes or 15 minutes. It can be set according to the needs, and no specific limitation is made here.

[0050] The aforementioned incremental files typically include, according to their dimensions, the following: incremental files for basic information on bills, incremental files for endorsement information on bills, incremental files for acceptance information on bills, incremental files for purchase information on bills, incremental files for sale information on bills, incremental files for payment presentation information on bills, and incremental files for corporate customer information, etc.

[0051] Optionally, this application can also combine the above-mentioned near real-time data entry method with the daily data entry method in related technologies. Figure 3 A schematic diagram of an optional data ingestion process is shown, including the following steps:

[0052] S1 reads the Oracle archive logs every 15 minutes.

[0053] S2, determine the first incremental data by replicating CDC data;

[0054] S3, synchronize the first incremental data to the Kafka message queue;

[0055] S4, import the first incremental data into the public area of ​​the data lake in batches;

[0056] S5 will import all invoice data for the day into the public area of ​​the data lake in batches at 21:00 every day.

[0057] Step S5 can be considered as a fallback solution for steps S1-S4.

[0058] As an optional implementation, in order to improve the accuracy of the data entering the data lake, after determining the first incremental data in the first relational database, for each incremental file in the first incremental data, a verification file corresponding to the incremental file can also be generated, wherein the verification file includes the file attribute information of the incremental file; and then the verification file is sent to the public area of ​​the data lake.

[0059] Specifically, for each incremental file, a check file with the same name ending in CHK can be generated based on the file name, file size, number of records, and other information of the incremental file. Then, the incremental file and the check file are imported into the public area of ​​the data lake.

[0060] Subsequently, when loading the first incremental data into the source data table within the data lake's public area, for each incremental file in the first incremental data, the incremental file can first be validated against its corresponding validation file; only when the incremental file passes validation will it be added to the invoice information table corresponding to the dimension of the incremental file. The primary key can be used as a reference when adding data.

[0061] Optionally, the dimensions of the bill information table in the source data table typically include: basic bill information table, bill endorsement information table, bill acceptance information table, bill purchase information table, bill sale information table, bill payment reminder table, and corporate customer information table.

[0062] As an optional implementation, in order to improve the richness of the data entering the lake, after determining the first incremental data in the first relational database, the fifth incremental data in the second relational database associated with the first relational database can also be determined and imported into the public area of ​​the data lake; then the fifth incremental data is loaded into the source data table in the public area of ​​the data lake.

[0063] The second relational database can be an Oracle database for the customer management system, and its corresponding fifth incremental data includes: incremental files of legal person customer certificates, incremental files of legal person customer information, incremental files of customer system dictionary, etc.; the second relational database can also be an Oracle database for the credit management system, and its corresponding fifth incremental data includes: incremental files of teller management, incremental files of loan contract ledgers, incremental files of loan receipt ledgers, incremental files of statistical bill issuance, incremental files of online guarantee agreement ledgers, etc.

[0064] Optionally, the bill information wide tables in the private area of ​​the data lake can be divided into two types: thematic aggregation wide tables and indicator aggregation wide tables. The dimensions of thematic aggregation wide tables include: customer summary information wide table, institution summary information wide table, bill purchase transaction details wide table, bill sale transaction details wide table, bill inventory details wide table, bill pool business details wide table, bill endorsement transaction details wide table, bill circulation details wide table, bill acceptance details wide table, bill payment reminder wide table, bill pledge details wide table, bill recovery processing details wide table, and bill recovery processing details wide table. The dimensions of indicator aggregation wide tables include: bill customer indicator wide table and bill institution indicator wide table.

[0065] Taking the customer summary information wide table as an example, it is a customer-dimensional indicator wide table that needs to include customer-related indicators, including: whether there are loan accounts, the amount issued (invoices), the amount of bill acceptance processed, the amount received (invoices), the amount endorsed, the amount endorsed, the amount of bills in the pool for custody, the amount of bills in the pool for pledge, the balance of bills in the pool for custody, the amount of bills in the pool for pledge financing, the amount discounted to this institution, the amount discounted to other institutions, and the amount of payment presented.

[0066] Taking the wide table of institutional summary information as an example, it is a wide table of indicators at the institutional level. It requires the inclusion of institutional-related indicators, including: summary date, institutional code, affiliated secondary institution, affiliated primary institution, affiliated branch, discount balance, rediscount balance, buy-back balance, sell-repurchase balance, rediscount balance, discount transaction volume, rediscount outright transaction volume, rediscount outright transaction volume (outside the system), buy-back transaction volume, rediscount outright transaction volume, rediscount outright transaction volume (outside the system), sell-repurchase transaction volume, rediscount transaction volume, number of discount clients, number of rediscount outright transaction clients, number of rediscount buy-back transaction clients (outside the system), number of buy-back transaction clients, number of rediscount outright transaction clients, number of rediscount outright transaction clients (outside the system), number of sell-repurchase transaction clients. The indicators include: number of discount transactions, number of rediscount buyout transactions, number of rediscount buyout transactions (outside the system), number of buy-back transactions, number of rediscount sellout transactions, number of rediscount sellout transactions (outside the system), number of sell-back transactions, number of rediscount transactions, non-performing asset ratio of bills, number of non-performing assets, balance of non-performing assets, number of overdue assets, amount of overdue assets, discount interest income from the beginning of the year, discount spread income from the beginning of the year, rediscount interest income from the beginning of the year, rediscount spread income from the beginning of the year, buy-back interest income from the beginning of the year, discount yield from the beginning of the year, buy-back yield from the beginning of the year, amount of bill acceptance processed, amount issued (issued), amount received (issued), amount endorsed, amount endorsed, amount presented for payment, amount of bills in the pool for custody, amount of bills pledged in the pool, and balance of bills in the pool for custody.

[0067] It should be noted that the type and dimensions of the above wide table of bill information can be adjusted as needed, and the data dimensions in each wide table of bill information can also be adjusted as needed. The above content is only for illustrative purposes and does not constitute actual limitation.

[0068] As an optional implementation, when processing the second incremental data from each invoice information table into the invoice information wide table corresponding to the dimensions of the invoice information table within the private area of ​​the data lake, the following method can be used: For each invoice information wide table to be processed, determine the type of the invoice information wide table; when the invoice information wide table is a topic aggregation wide table, merge the second incremental data from the invoice information table corresponding to the dimensions of the invoice information wide table with the existing data in the invoice information wide table to obtain the full invoice information wide table; when the invoice information wide table is an indicator aggregation wide table, calculate the incremental indicator based on the second incremental data from the invoice information table corresponding to the dimensions of the invoice information wide table, and merge the incremental indicator with the existing indicators in the invoice information wide table to obtain the full invoice information wide table. During data fusion, for data with conflicting primary keys, the newly processed data shall prevail.

[0069] Understandably, when processing data in the private area of ​​the data lake, for any dimension of the wide table of invoice information, it is only necessary to process a full wide table at the initial moment. In subsequent processing, it is only necessary to merge the incremental data with the current stock data to obtain the full data. Compared with the method of processing the full data every night in related technologies, this method can greatly improve the data processing efficiency and obtain the full wide table of invoice information in near real-time.

[0070] Optionally, within the private area of ​​the data lake, the Spark computing engine can be used to process the second incremental data in each invoice information table into a wide invoice information table corresponding to the dimension of the invoice information table; wherein, when processing each wide invoice information table, the amount of system resources allocated to the thread processing the wide invoice information table is determined based on the total data capacity of the second incremental data in the invoice information table corresponding to the dimension of the wide invoice information table.

[0071] The system resources allocated above mainly include CPU and memory. Since all data processing in the Spark computing engine is performed in memory, there is no need to read intermediate output files, which can further improve data processing efficiency.

[0072] As an optional implementation, when processing the second incremental data from each bill information table into a bill information wide table corresponding to the dimension of the bill information table within the private area of ​​the data lake, the second incremental data can also be classified according to the institution to which it belongs, resulting in third incremental data belonging to the target institution and fourth incremental data not belonging to the target institution; the third incremental data and the fourth incremental data are then processed into the bill information wide table respectively, wherein the system resources allocated for processing the third incremental data are greater than the system resources allocated for processing the fourth incremental data.

[0073] Understandably, when processing wide table data, organizations often operate on an organizational basis, which inevitably leads to data skew and reduced processing efficiency. Therefore, it is advisable to process the data of one's own organization and other organizations separately before merging them to avoid problems such as memory overruns or excessive processing time caused by data skew.

[0074] After all the bill information wide tables are processed, they are stored to facilitate subsequent bill business queries and retrieval.

[0075] As an optional implementation, the invoice data platform can respond to the terminal's query operation, generate a target data report corresponding to the query operation based on multiple invoice information wide tables, and feed the target data report back to the terminal. Figure 4 This demonstrates an inventory analysis report generated based on a wide table of invoice inventory details.

[0076] Figure 5 This is a complete schematic diagram of an optional bill data processing process provided by an embodiment of this application. Relevant bill information is obtained from the bill management system, customer management system, and credit management system, respectively. Then, corresponding incremental bill information files are determined in the corresponding bill management oracle, customer management oracle, and credit management oracle. These incremental files are imported into various bill information tables in the public area of ​​the data lake. In the private area of ​​the data lake, the incremental data from each bill information table is processed into the corresponding wide bill information table. Finally, in response to a query operation from the terminal, a corresponding data report is generated and returned. The terminal can provide functions such as report analysis, chart display, and dashboard display.

[0077] The invoice data processing method provided in this application involves obtaining invoice data from an invoice management system and writing it into a first relational database. Then, it periodically determines the first incremental data in the first relational database and imports it into the public area of ​​a data lake. Within the public area of ​​the data lake, the first incremental data is loaded into a source data table, which includes multiple dimensions of invoice information tables. In the private area of ​​the data lake, the second incremental data from each invoice information table is processed into a wide invoice information table corresponding to the dimensions of the invoice information table. Finally, in response to a terminal query operation, a target data report corresponding to the query operation is generated based on multiple wide invoice information tables, and the target data report is fed back to the terminal. By periodically importing incremental invoice data from the invoice management system into the source data table in the public area of ​​the data lake and processing the data in the private area of ​​the data lake using a combination of existing and incremental data, the efficiency of invoice data processing can be greatly improved, thus solving the technical problem of low invoice data processing efficiency in related technologies, which makes it difficult to meet user needs.

[0078] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0079] This application also provides a bill data processing apparatus. It should be noted that the bill data processing apparatus of this application can be used to execute the bill data processing method provided in this application. The bill data processing apparatus provided in this application is described below.

[0080] Figure 6 This is a schematic diagram of an optional ticket data processing apparatus provided according to an embodiment of this application. Figure 6 As shown, the device includes: an acquisition module 61, a determination module 62, a loading module 63, a processing module 64, and a query module 65.

[0081] The acquisition module 61 is used to acquire invoice data from the invoice management system and write the invoice data into the first relational database;

[0082] Module 62 is used to periodically determine the first incremental data in the first relational database and import the first incremental data into the public area of ​​the data lake;

[0083] Loading module 63 is used to load the first incremental data into the source data table in the public area of ​​the data lake, wherein the source data table includes multiple dimensions of invoice information tables;

[0084] Processing module 64 is used to process the second incremental data of each bill information table into a bill information wide table corresponding to the dimension of the bill information table within the private area of ​​the data lake;

[0085] The query module 65 is used to respond to the query operation of the terminal, generate the target data report corresponding to the query operation based on multiple wide tables of invoice information, and feed the target data report back to the terminal.

[0086] As an optional implementation, when the determining module periodically determines the first incremental data in the first relational database and imports the first incremental data into the public area of ​​the data lake, it can do so in the following way: read the archived logs of the first relational database according to a preset time period; capture change data in the archived logs to obtain the first incremental data, wherein the first incremental data is the incremental data between the previous reading time and the current reading time, and the first incremental data includes incremental files of multiple dimensions; synchronize the first incremental data to the Kafka message queue, wherein each incremental file corresponds to a topic in the Kafka message queue; and import the first incremental data into the public area of ​​the data lake through the Kafka message queue.

[0087] In order to improve the real-time performance of the data, the preset time period is usually short, such as 10 minutes or 15 minutes. It can be set according to the needs, and no specific limitation is made here.

[0088] The aforementioned incremental files typically include, according to their dimensions, the following: incremental files for basic information on bills, incremental files for endorsement information on bills, incremental files for acceptance information on bills, incremental files for purchase information on bills, incremental files for sale information on bills, incremental files for payment presentation information on bills, and incremental files for corporate customer information, etc.

[0089] As an optional implementation, in order to improve the accuracy of the data entering the lake, after determining the first incremental data in the first relational database, the determination module can also generate a verification file corresponding to each incremental file in the first incremental data, wherein the verification file includes the file attribute information of the incremental file; and then send the verification file to the public area of ​​the data lake.

[0090] Specifically, for each incremental file, the determination module can generate a verification file with the same name ending in CHK based on information such as the file name, file size, and number of records of the incremental file, and then import the incremental file and the verification file into the public area of ​​the data lake.

[0091] Subsequently, when the loading module loads the first incremental data into the source data table within the data lake's public area, for each incremental file in the first incremental data, it can first verify the incremental file based on the corresponding verification file; only when the incremental file passes verification will it be added to the invoice information table corresponding to the dimension of the incremental file. The primary key can be used as a reference when adding data.

[0092] Optionally, the dimensions of the bill information table in the source data table typically include: basic bill information table, bill endorsement information table, bill acceptance information table, bill purchase information table, bill sale information table, bill payment reminder table, and corporate customer information table.

[0093] As an optional implementation, in order to improve the richness of the data entering the lake, after determining the first incremental data in the first relational database, the determining module can also determine the fifth incremental data in the second relational database associated with the first relational database, and import the fifth incremental data into the public area of ​​the data lake; then, the loading module loads the fifth incremental data into the source data table in the public area of ​​the data lake.

[0094] The second relational database can be an Oracle database for the customer management system, and its corresponding fifth incremental data includes: incremental files of legal person customer certificates, incremental files of legal person customer information, incremental files of customer system dictionary, etc.; the second relational database can also be an Oracle database for the credit management system, and its corresponding fifth incremental data includes: incremental files of teller management, incremental files of loan contract ledgers, incremental files of loan receipt ledgers, incremental files of statistical bill issuance, incremental files of online guarantee agreement ledgers, etc.

[0095] Optionally, the bill information wide tables in the private area of ​​the data lake can be divided into two types: thematic aggregation wide tables and indicator aggregation wide tables. The dimensions of thematic aggregation wide tables include: customer summary information wide table, institution summary information wide table, bill purchase transaction details wide table, bill sale transaction details wide table, bill inventory details wide table, bill pool business details wide table, bill endorsement transaction details wide table, bill circulation details wide table, bill acceptance details wide table, bill payment reminder wide table, bill pledge details wide table, bill recovery processing details wide table, and bill recovery processing details wide table. The dimensions of indicator aggregation wide tables include: bill customer indicator wide table and bill institution indicator wide table.

[0096] The types and dimensions of the above wide tables for bill information can be adjusted as needed, and the data dimensions in each wide table can also be adjusted as needed; no specific limitations are made here.

[0097] As an optional implementation, when the processing module processes the second incremental data from each invoice information table into the invoice information wide table corresponding to the dimension of the invoice information table within the private area of ​​the data lake, it can be done in the following way: For each invoice information wide table to be processed, determine the type of the invoice information wide table; when the invoice information wide table is a topic aggregation type wide table, merge the second incremental data from the invoice information table corresponding to the dimension of the invoice information wide table with the existing data in the invoice information wide table to obtain the full invoice information wide table; when the invoice information wide table is an indicator aggregation type wide table, calculate the incremental indicator based on the second incremental data from the invoice information table corresponding to the dimension of the invoice information wide table, and merge the incremental indicator with the existing indicators in the invoice information wide table to obtain the full invoice information wide table. During data fusion, for data with conflicting primary keys, the newly processed data shall prevail.

[0098] Understandably, when processing data in the private area of ​​the data lake, for any dimension of the wide table of invoice information, it is only necessary to process a full wide table at the initial moment. In subsequent processing, it is only necessary to merge the incremental data with the current stock data to obtain the full data. Compared with the method of processing the full data every night in related technologies, this method can greatly improve the data processing efficiency and obtain the full wide table of invoice information in near real-time.

[0099] Optionally, within the private area of ​​the data lake, the processing module can use the Spark computing engine to process the second incremental data from each invoice information table into an invoice information wide table corresponding to the dimension of the invoice information table; wherein, when processing each invoice information wide table, the amount of system resources allocated to the thread processing the invoice information wide table is determined based on the total data capacity of the second incremental data in the invoice information table corresponding to the dimension of the invoice information wide table.

[0100] The system resources allocated above mainly include CPU and memory. Since all data processing in the Spark computing engine is performed in memory, there is no need to read intermediate output files, which can further improve data processing efficiency.

[0101] As an optional implementation, when the processing module processes the second incremental data from each bill information table into a bill information wide table corresponding to the dimension of the bill information table within the private area of ​​the data lake, it can also classify the second incremental data according to the institution to which it belongs, to obtain the third incremental data belonging to the target institution and the fourth incremental data not belonging to the target institution; the third incremental data and the fourth incremental data are processed into the bill information wide table respectively, wherein the system resources allocated for processing the third incremental data are greater than the system resources allocated for processing the fourth incremental data.

[0102] Understandably, when processing wide table data, organizations often operate on an organizational basis, which inevitably leads to data skew and reduced processing efficiency. Therefore, it is advisable to process the data of one's own organization and other organizations separately before merging them to avoid problems such as memory overruns or excessive processing time caused by data skew.

[0103] After all the bill information wide tables have been processed, the processing module can store them for subsequent bill business queries and retrieval.

[0104] The query module can respond to the query operation of the terminal, generate the target data report corresponding to the query operation based on multiple wide tables of invoice information, and feed the target data report back to the terminal.

[0105] The invoice data processing apparatus provided in this application embodiment acquires invoice data from an invoice management system through an acquisition module and writes the invoice data into a first relational database. Then, a determination module periodically determines the first incremental data in the first relational database and imports the first incremental data into the public area of ​​a data lake. A loading module loads the first incremental data into a source data table within the public area of ​​the data lake, wherein the source data table includes multiple dimensions of invoice information tables. A processing module processes the second incremental data from each invoice information table into a wide invoice information table corresponding to the dimensions of the invoice information table within the private area of ​​the data lake. Finally, a query module can respond to the terminal's query operation, generate a target data report corresponding to the query operation based on multiple wide invoice information tables, and feed the target data report back to the terminal. By periodically importing incremental invoice data from the invoice management system into the source data table in the public area of ​​the data lake, and processing the data in the private area of ​​the data lake using a combination of existing and incremental data, the processing efficiency of invoice data can be greatly improved, thereby solving the technical problem of low invoice data processing efficiency in related technologies, which makes it difficult to meet user needs.

[0106] This application provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described invoice data processing method.

[0107] This application provides a processor for running a program, wherein the program executes the above-described invoice data processing method during runtime.

[0108] like Figure 7As shown, this application embodiment provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining invoice data from an invoice management system and writing the invoice data into a first relational database; periodically determining first incremental data in the first relational database and importing the first incremental data into the public area of ​​a data lake; loading the first incremental data into a source data table in the public area of ​​the data lake, wherein the source data table includes multiple dimensions of invoice information tables; processing the second incremental data from each invoice information table into a wide invoice information table corresponding to the dimensions of the invoice information table in the private area of ​​the data lake; responding to a query operation from a terminal, generating a target data report corresponding to the query operation based on multiple wide invoice information tables, and feeding the target data report back to the terminal.

[0109] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the aforementioned document data processing method can be implemented by adjusting kernel parameters.

[0110] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0111] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: obtaining invoice data from an invoice management system and writing the invoice data into a first relational database; periodically determining first incremental data in the first relational database and importing the first incremental data into a public area of ​​a data lake; loading the first incremental data into a source data table in the public area of ​​the data lake, wherein the source data table includes multiple dimensions of invoice information tables; processing second incremental data from each invoice information table into a wide invoice information table corresponding to the dimensions of the invoice information table in a private area of ​​the data lake; and, in response to a query operation from a terminal, generating a target data report corresponding to the query operation based on multiple wide invoice information tables and feeding the target data report back to the terminal.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0117] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0118] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0119] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0120] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of processing ticket data, characterized by, include: Obtain invoice data from the invoice management system and write the invoice data into the first relational database; The process of periodically determining the first incremental data in the first relational database and importing the first incremental data into the public area of ​​the data lake includes: reading the archived logs of the first relational database according to a preset time period; capturing change data in the archived logs to obtain the first incremental data, wherein the first incremental data is incremental data between the previous reading time and the current reading time, and the first incremental data includes incremental files of multiple dimensions; synchronizing the first incremental data to a Kafka message queue, wherein each incremental file corresponds to a topic in the Kafka message queue; and importing the first incremental data into the public area of ​​the data lake through the Kafka message queue. Within the public area of ​​the data lake, the first incremental data is loaded into the source data table, wherein the source data table includes multiple dimensions of invoice information tables; Within the private data lake area, the second incremental data from each of the bill information tables is processed into a bill information wide table corresponding to the dimension of the bill information table. This includes: classifying the second incremental data according to the institution to which it belongs, obtaining third incremental data belonging to the target institution and fourth incremental data not belonging to the target institution; and processing the third incremental data and the fourth incremental data into the bill information wide table, wherein the system resources allocated for processing the third incremental data are greater than the system resources allocated for processing the fourth incremental data. In response to a query operation from the terminal, a target data report corresponding to the query operation is generated based on multiple wide tables of bill information, and the target data report is fed back to the terminal.

2. The method of claim 1, wherein, After determining the first incremental data in the first relational database, the method further includes: For each incremental file in the first incremental data, a verification file corresponding to the incremental file is generated, wherein the verification file includes the file attribute information of the incremental file; The verification file is sent to the public area of ​​the data lake.

3. The method of claim 2, wherein, Loading the first incremental data into the source data table within the public area of ​​the data lake includes: For each incremental file in the first incremental data, the incremental file is verified according to the verification file corresponding to the incremental file; When the incremental file passes verification, the incremental file is added to the invoice information table corresponding to the dimension of the incremental file.

4. The method of claim 1, wherein, Within the private data lake region, the second incremental data from each of the aforementioned invoice information tables is processed into a wide invoice information table corresponding to the dimensions of the aforementioned invoice information tables, including: Within the private area of ​​the data lake, the Spark computing engine is used to process the second incremental data from each of the bill information tables into the bill information wide table corresponding to the dimension of the bill information table; Specifically, when processing each of the bill information wide tables, the amount of system resources allocated to the thread processing the bill information wide table is determined based on the total data capacity of the second incremental data in the bill information table corresponding to the dimension of the bill information wide table.

5. The method of claim 1, wherein, Within the private data lake region, the second incremental data from each of the aforementioned invoice information tables is processed into a wide invoice information table corresponding to the dimensions of the aforementioned invoice information tables, including: For each of the wide tables containing invoice information to be processed, determine the type of the wide table containing invoice information; When the bill information wide table is a topic aggregation wide table, the second incremental data in the bill information table corresponding to the dimension of the bill information wide table is merged with the existing data in the bill information wide table to obtain the full bill information wide table; When the bill information wide table is an indicator aggregation wide table, incremental indicators are calculated based on the second incremental data in the bill information table corresponding to the dimension of the bill information wide table. The incremental indicators are then merged with the existing indicators in the bill information wide table to obtain the full bill information wide table.

6. The method according to claim 1, characterized in that, After determining the first incremental data in the first relational database, the method further includes: Identify the fifth incremental data in the second relational database associated with the first relational database, and import the fifth incremental data into the public area of ​​the data lake; The fifth incremental data is loaded into the source data table within the public area of ​​the data lake.

7. The method according to any one of claims 1 to 6, characterized in that, The dimensions of the bill information table include: basic bill information table, bill endorsement information table, bill acceptance information table, bill purchase information table, bill sale information table, bill payment reminder table, and corporate customer information table; The types of the bill information wide tables include: thematic aggregation wide tables and indicator aggregation wide tables. The dimensions of the thematic aggregation wide tables include: customer summary information wide tables, institution summary information wide tables, bill purchase transaction details wide tables, bill sale transaction details wide tables, bill inventory details wide tables, bill pool business details wide tables, bill endorsement transaction details wide tables, bill circulation details wide tables, bill acceptance details wide tables, bill payment reminder wide tables, bill pledge details wide tables, bill recovery processing details wide tables, and bill recovery processing details wide tables. The dimensions of the indicator aggregation wide tables include: bill customer indicator wide tables and bill institution indicator wide tables.

8. A document data processing device, characterized in that, include: The acquisition module is used to acquire invoice data from the invoice management system and write the invoice data into the first relational database; A determination module is used to periodically determine the first incremental data in the first relational database and import the first incremental data into the public area of ​​the data lake. This includes: reading the archived logs of the first relational database according to a preset time period; capturing change data in the archived logs to obtain the first incremental data, wherein the first incremental data is incremental data between the previous read time and the current read time, and the first incremental data includes incremental files of multiple dimensions; synchronizing the first incremental data to a Kafka message queue, wherein each incremental file corresponds to a topic in the Kafka message queue; and importing the first incremental data into the public area of ​​the data lake through the Kafka message queue. A loading module is used to load the first incremental data into the source data table within the public area of ​​the data lake, wherein the source data table includes a multi-dimensional invoice information table; The processing module is used to process the second incremental data from each of the bill information tables into a bill information wide table corresponding to the dimension of the bill information table within the private area of ​​the data lake. This includes: classifying the second incremental data according to the institution to which it belongs, obtaining third incremental data belonging to the target institution and fourth incremental data not belonging to the target institution; and processing the third incremental data and the fourth incremental data into the bill information wide table, wherein the system resources allocated for processing the third incremental data are greater than the system resources allocated for processing the fourth incremental data. The query module is used to respond to the query operation of the terminal, generate a target data report corresponding to the query operation based on multiple wide tables of ticket information, and feed the target data report back to the terminal.

9. A ticket data processing system, characterized in that, include: Invoice management system, invoice data platform and terminal; The invoice data platform includes a first relational database and a distributed storage cluster, wherein the distributed storage cluster includes a public area of ​​the data lake and a private area of ​​the data lake; the invoice data platform is used to execute the invoice data processing method according to any one of claims 1 to 7.

10. A processor, characterized in that, The processor is used to run a program, wherein the program executes the invoice data processing method according to any one of claims 1 to 7.

11. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the invoice data processing method according to any one of claims 1 to 7.

Images