Batch data real-time processing method and device, computer device and storage medium

Abstract

The present application relates to the field of financial technology, and particularly relates to a batch data real-time processing method and device, computer equipment and a storage medium. The method comprises the following steps: analyzing user real-time requirements to obtain report input parameters corresponding to a user required report, the report input parameters comprising data assets; determining a data asset list related to the report input parameters according to a report configuration management table, the data asset list comprising data sources; judging whether the data sources are updated on the current day according to data update logic in the report configuration management table; if yes, obtaining new increment data, processing the new increment data according to the report configuration management table to obtain a processing result; storing the processing result and a last day end processing result of the data sources in an application layer as a user requirement report; and if no, taking the last day end processing result as the user requirement report. The present application overcomes the shortcomings in the prior art of data analysis technology, and supports real-time analysis and calculation requirements of a large amount of data with less impact on OLTP transaction processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of financial technology, and in particular to a batch processing data real-time processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the continuous acceleration of the opening-up pace of the capital market and the continuous improvement of the degree of innovation, active management transformation has gradually become the consensus of asset management institutions. The investment data and analysis needs of institutional investors have shown explosive growth. The existing method of relying on traditional relational databases to uniformly process online transaction processing (OLTP) data and online analytical processing (OLAP) data has appeared performance bottlenecks and cannot meet the real-time and analysis computing needs of the explosive growth of users.

[0003] One method in the prior art is to continue to rely on traditional relational databases, and to separately split OLAP and OLTP processing tasks to two independent databases for processing, and real-time synchronization is required between the two databases. In this way, an efficient data synchronization device needs to be built between the OLAP and OLTP databases to copy data from the OLTP database to the OLAP database, which will cause performance consumption and resource occupation of the OLTP database. When the business transaction volume continues to rise, the OLTP database will also appear performance bottlenecks.

[0004] Another method in the prior art is to batch synchronize the data of transactions that occurred in a recent period of time to a big data cluster during a low peak period of business transactions, and to initiate large-scale offline computing processing by the big data cluster. This method cannot utilize the big data cluster to process the latest transaction data that occurred after the last batch, which will result in that the data timeliness cannot meet the business requirements. For example, the business modifies the original data after the last batch, which will not be covered by the big data computing. SUMMARY

[0005] To solve the problems of large database synchronization resource occupation and poor offline computing timeliness in the prior art, the embodiments of the present application provide a batch processing data real-time processing method and device, computer equipment and a storage medium.

[0006] The embodiment of the present application provides a batch data real-time processing method, which comprises the following steps: analyzing user real-time requirements to obtain report input parameters corresponding to a report required by a user, wherein the report input parameters comprise data assets; determining a data asset list related to the report input parameters according to a report configuration management table, wherein the data asset list comprises data sources; judging whether the data sources are updated on the current day according to data update logic in the report configuration management table; if yes, obtaining new increment data, and processing the new increment data according to the report configuration management table to obtain a processing result; storing the processing result and a last-day-end processing result of the data sources in an application layer as a user requirement report; and if no, taking the last-day-end processing result as the user requirement report.

[0007] According to an aspect of the embodiment of the present application, determining a data asset list related to the report input parameters according to a report configuration management table comprises the following steps: determining a report number corresponding to a report type in the report input parameters; determining data asset numbers required for forming a report represented by the report number from the report configuration management table according to the report number; and determining dependent data asset numbers of the data asset numbers from the configuration management table according to the data asset numbers, wherein the data assets and the dependent data assets constitute the data asset list.

[0008] According to an aspect of the embodiment of the present application, the report configuration management table records data sources and asset types of various types of reports, wherein the data sources comprise at least one of a source layer, an aggregation layer and an extraction layer, and the dependent data assets comprise source layer data assets, aggregation layer data assets and extraction layer data assets.

[0009] According to an aspect of the embodiment of the present application, judging whether the data asset list is updated on the current day comprises the following steps:

[0010] According to the source layer data assets, determining data update checking logic corresponding to the source layer data assets from the report configuration management table; determining whether there is increment data in an upstream OLTP database by using the data update checking logic; and if yes, determining that the data sources are updated on the current day.

[0011] According to an aspect of the embodiment of the present application, before obtaining new increment data and processing the new increment data according to the report configuration management table to obtain a processing result, the method comprises the following steps: determining increment data obtaining logic corresponding to the source layer data assets from the report configuration management table; obtaining the increment data from the OLTP database by using the increment data obtaining logic; and replacing historical data assets in the source layer with the increment data to obtain the latest data of the source layer.

[0012] According to an aspect of the embodiments herein, the method further comprises determining data processing logic corresponding to the aggregated layer data asset and the extracted layer data asset from the report configuration management table, the data processing logic corresponding to each type of data source; and processing the data source distribution belonging to each type of asset according to the data processing logic and the latest data of the source layer to form a processing result.

[0013] According to an aspect of the embodiments herein, the method comprises if the data asset of the source layer is block data, performing block update on the source layer using the latest data of the source layer.

[0014] The embodiments herein provide a batch data real-time processing method and device, the device comprising: a user real-time requirement analysis unit configured to obtain report input parameters corresponding to a user required report, the report input parameters comprising data assets; a data asset list determination unit configured to determine a data asset list related to the report input parameters according to a report configuration management table, the data asset list comprising data sources, and the configuration management table comprising a report configuration table and a data asset configuration table; a determination unit configured to determine whether the data sources have been updated on the current day according to data update logic in the configuration management table; a processing result acquisition unit configured to, if yes, acquire current latest data of the data sources and process the current latest data of the data sources according to the configuration management table to obtain a processing result; and a user requirement report determination unit configured to store the processing result and a previous day end processing result of the data sources in an application layer as a user requirement report; and if no, store the previous day end processing result as the user requirement report.

[0015] The embodiments herein further provide a computer device, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the batch data real-time processing method when executing the computer program.

[0016] The embodiments herein further provide a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the batch data real-time processing method.

[0017] The present solution can overcome the deficiencies in the prior art data analysis techniques and support real-time analysis and calculation of a large amount of data with less impact on OLTP transaction processing. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0019] Figure 1 A flow chart of a batch data real-time processing method according to an embodiment of the present disclosure is shown;

[0020] Figure 2 A flow chart of a method for determining a data asset list related to a report parameter according to an embodiment of the present disclosure is shown;

[0021] Figure 3 A flow chart of a method for determining whether the data asset list is updated on the same day according to an embodiment of the present disclosure is shown;

[0022] Figure 4 A flow chart of another batch data real-time processing method according to an embodiment of the present disclosure is shown;

[0023] Figure 5 A flow chart of a method for processing a data asset to form a processing result according to an embodiment of the present disclosure is shown;

[0024] Figure 6 A structural schematic diagram of a batch data real-time processing device according to an embodiment of the present disclosure is shown;

[0025] Figure 7 A schematic diagram of a data flow according to an embodiment of the present disclosure is shown;

[0026] Figure 8 A schematic diagram of a real-time update of a data asset according to an embodiment of the present disclosure is shown;

[0027] Figure 9 A structural schematic diagram of a computer device according to an embodiment of the present disclosure is shown.

[0028] Explanation of drawing symbols:

[0029] 601, user demand analysis unit;

[0030] 602, data asset list determination unit;

[0031] 603, judgment unit;

[0032] 604, processing result acquisition unit;

[0033] 605, user demand report determination unit;

[0034] 902, computer device;

[0035] 904, Processor;

[0036] 906. Memory;

[0037] 908. Drive mechanism;

[0038] 910. Input / Output Module;

[0039] 912. Input devices;

[0040] 914. Output devices;

[0041] 916. Presentation equipment;

[0042] 918. Graphical User Interface;

[0043] 920. Network interface;

[0044] 922. Communication link;

[0045] 924. Communication bus. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments herein will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments herein, and not all of the embodiments. Based on the embodiments herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0047] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0048] The specification provides method operation steps as described in the embodiments or flowcharts, but can include more or fewer operation steps based on routine or non-creative labor. The order of steps listed in the embodiments is only one of the many step execution orders, and does not represent the only execution order. In actual system or device product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel.

[0049] It should be noted that the batch data real-time processing method herein can be used in the field of financial technology, and can also be used in the field of computers, and the application field of the batch data real-time processing method and device herein is not limited.

[0050] Figure 1 The flowchart of the batch data real-time processing method according to an embodiment of the present disclosure is shown, and specifically includes the following steps:

[0051] Step 101, parse user demand to obtain report input parameters corresponding to a report required by the user, the report input parameters including a target product, a time parameter, a report type, and data assets included in the report type. In some embodiments of the present specification, the user includes an individual, a fund manager, an institutional investor, etc. The user demand is information related to a certain product that the user wants to query, and a report of a specified requirement is generated. Specifically, the user demand is report data of a certain product at a certain time point or within a certain period of time. For example, the user requests to generate a net value size table of holding product A on December 31, 2022 at 8:30 on January 19, 2023. Therefore, by parsing the user demand in this step, the input parameters of the report required by the user can be obtained, including the target product A, the time parameter December 31, 2022, and the report type net value size table. Further, according to the report type and the preset mapping relationship between the report type and the data assets, the data assets included in the report type involved in the user demand are determined. If the report type of the user demand is the net value size table, the data assets of the net value size table are portfolio net value and portfolio name.

[0052] Step 102, determining a data asset list related to the report input parameters according to a report configuration management table, the data asset list including a data source.

[0053] In some embodiments of the present specification, the report configuration management module records all data assets contained in each report type in advance to form a data asset list. Specifically, the report configuration management module records the data source of each report and the flow direction of the data in each report. Among them, the data source includes at least one of the source layer, the aggregation layer, and the extraction layer. In this step, by querying the data asset list related to the report input parameters in the report configuration management table, the data source of the data asset list of the user demand can be determined.

[0054] As Figure 8 shown, Figure 8 The flow direction of different layer data is shown in the middle. Application layer data (ADS) uses a relational database for storage, supporting user association query data and updating; extraction layer data (DWS), aggregation layer data (DWD) and dimension data (DI) use columnar storage, supporting user real-time point query and updating in key-value pair mode; the source layer data uses file storage, only for offline batch computing module, so it only needs to support offline query. In some embodiments of the present specification, the extraction layer, the aggregation layer, and the dimension data can be shared by multiple scenes for common digital assets, and these data are easily used by users in real-time query, so these data need to support real-time query. The source layer data maps the original data of the upstream OLTP system, most of which has been transferred to the extraction layer, the aggregation layer and the dimension data, and is not frequently used in the user real-time query scenario, so it does not need to support real-time query, only offline query is needed.

[0055] In some other embodiments of the present specification, in addition to recording the data sources, the data asset list also records the data flow direction of different layer data, as shown in Figure 7 The data flow direction of the source layer ODS data, the aggregation layer DWD data set and the extraction layer DWS data is shown in the figure. A001 in the source layer is dependent data of the aggregation layer theme A101, and the data flow direction is A001 pointing to A101; A002 in the source layer is dependent data of the aggregation layer theme A102, and the aggregation layer theme A102 is dependent data of the extraction layer data A201, and the data flow direction is: the source layer data A002 points to the aggregation layer data A102, and the aggregation layer data A102 points to the extraction layer data A201.

[0056] Step 103, judging whether the data source occurs update today according to the data update logic in the report configuration management table. In this step, according to the data asset determined in the foregoing steps, the data update logic is obtained from the corresponding report configuration management table, and further according to the data update logic, it is judged whether the data source corresponding to the data asset occurs real-time update today.

[0057] Step 104, if yes, obtaining the latest incremental data, and processing the latest incremental data according to the report configuration management table to obtain the processing result. If it is determined that the data source occurs update today, the latest update data (i.e. the latest incremental data) is obtained. According to the incremental data obtaining logic of the data source, the report configuration management table processes the latest incremental data to obtain the processing result.

[0058] Step 105, store the processing result and the previous day-end processing result of the data source in the application layer as the user demand report. In this step, according to the time parameter in the user demand, the processing result corresponding to the newly added data is combined with the processing result of the previous day-end of the data source and stored in the application layer to form the user demand report and show the user.

[0059] Step 106, if not, use the previous day-end processing result as the user demand report. If it is determined in the foregoing step that no update occurs to the data source on the current day, the processing result of the previous day-end is stored in the application layer to form the user demand report. In some embodiments of the present specification, the BI visualization tool can be used to configure the report in the form of a combination of charts and tables to present the final analysis result in a picture-text combination manner, which is convenient for the user to read.

[0060] Figure 2 The method flow chart for determining the data asset list related to the report input parameter is shown, and specifically includes the following steps:

[0061] Step 201, determine the report number corresponding to the report type in the report input parameter. In some embodiments of the present specification, the report number corresponding to different report types can be preset. For example, the report type of the user demand is a combination size table, and according to the preset, it is determined that the report number corresponding to the combination size table is R10000001. In some other embodiments of the present specification, other types of reports have other corresponding report numbers. Different report types have their unique independent report numbers. In some embodiments of the present specification, the report numbers corresponding to different report types can be recorded and stored by the report management configuration management table.

[0062] Step 202, determine the data asset number required for forming the report represented by the report number from the report configuration management table. In some embodiments of the present specification, the report configuration management table can also record the relationship between the report number and the data asset number contained in the report. For example, according to the report number R10000001 corresponding to the combination size table, it can be determined that the data asset number contained in the combination size table is A101, A201. Among them, A101 represents the combination name, and A201 represents the combination net value.

[0063] Step 203: Based on the data asset number, determine the dependent data asset number from the configuration management table. The data asset and its dependent data assets constitute the data asset list. In some embodiments of this specification, dependent data assets represent the data source of data assets belonging to different layers in the database. For example, the dependent data asset of a data asset belonging to the aggregation layer is source layer data; the dependent data asset of a data asset belonging to the extraction layer is aggregation layer data; the source layer has no dependent data assets; and the dependent data asset of a data asset belonging to the application layer is extraction layer data. Therefore, based on the dependent data assets of the data asset (number) required for the report, the data flow and data source can be determined. Based on this, a data asset list can be formed according to the data assets and dependent data assets required for the report.

[0064] Figure 3 The diagram shows a flowchart of a method for determining whether a data asset list has been updated on the same day, according to an embodiment of this paper. The method includes the following steps:

[0065] Step 301: Based on the source layer data assets, determine the data update check logic corresponding to the source layer data assets from the report configuration management table. The report configuration management table allows tracing the data update check logic of the source data assets. Specifically, based on the source layer data assets in the dependent data assets, determine the data update check logic corresponding to the source layer data assets. The data update check logic is an SQL statement.

[0066] like Figure 7 As shown, the source layer contains two types of source layer data, A001 and A002. A001 represents source combination information, and A002 represents a combination valuation table. The data update check logic corresponding to the A001 source combination information and the A002 combination valuation table can be obtained from the report configuration management table. Through the aforementioned steps, based on the report configuration management table, the source layer data assets numbered A001 and A002 are ultimately traced back to, and two types of data update check statements for these two source layer data assets are further obtained.

[0067] For example, the data update check logic for the source combination information of A001 is: SELECT COUNT(1) FROM FUND_INFO_EDITLOG WHERE DATE='$SYSDATE' AND FUND_CODE='$FUND_CODE'; the data update check logic for the combination valuation table of A002 is: SELECT COUNT(1) FROM FUND_VALUE_SOURCE_EDITLOG WHERE DATE='$SYSDATE' AND FUND_CODE='$FUND_CODE'.

[0068] Step 302, using the data to update the checking logic, determine whether there is incremental data from the upstream OLTP database. According to the data update checking logic of the data asset of the source layer, determine whether there is incremental data from the OLTP database. Execute two data update checking statements SQL of A001 source combination information and A002 combination valuation table corresponding to the upstream OLTP system, and then determine whether there is incremental data. If the feedback is that the incremental data result is not 0, it means that there is incremental data.

[0069] Step 303, if yes, determine that the data source has updated on the day. In this step, if there is incremental data, it is determined that the data asset of the source layer has been updated, that is, the data source and the data asset list have been updated on the day.

[0070] Figure 4 The flowchart shown is another batch data real-time processing method of the embodiment of the present application, which specifically includes the following steps:

[0071] Step 401, determine the incremental data acquisition logic corresponding to the data asset of the source layer from the report configuration management table. Before acquiring the latest incremental data and processing the latest incremental data according to the report configuration management table to obtain the processing result, the incremental data acquisition logic corresponding to the data asset of the source layer needs to be acquired according to the report configuration management table.

[0072] Step 402, use the incremental data acquisition logic to acquire the incremental data from the OLTP database. In some embodiments of the present application, only the data of the source layer can have data update checking logic and incremental data acquisition logic in the data asset configuration table. The incremental data acquisition logic of A001 source combination information is: SELECT*FROM FUND_INFO WHERE DATE=‘$SYSDATE’AND FUND_CODE=‘$FUND_CODE’, and the incremental data acquisition logic of A002 combination valuation table is: SELECT*FROM FUND_VALE WHERE DATE=‘$SYSDATE’AND FUND_CODE=‘$FUND_CODE’.

[0073] Step 403, replace the historical data asset in the source layer with the incremental data to obtain the latest data of the source layer. After acquiring the incremental data, delete the offline batch result of the source layer, splice the incremental data, and obtain the latest data of the source layer to realize the data integration with the minimum performance consumption.

[0074] Figure 5 The flowchart shown is a method of processing data assets to form a processing result, which specifically includes the following steps:

[0075] Step 501, determining the data processing logic corresponding to the aggregated layer data asset and the extracted layer data asset from the report configuration management table, the data processing logic corresponding to each type of data source. After the data update and data integration of the source layer are completed, the data processing logic SQL statement of the aggregated layer data asset and the extracted layer data asset is obtained from the report configuration management table. The data assets of the upper aggregated layer and the extracted layer are integrated in turn.

[0076] Step 502, processing the data source distribution belonging to each type of asset according to the data processing logic and the latest data of the source layer, and forming a processing result.

[0077] For example, the A101 combination name belongs to the aggregated layer theme, and the processing logic is: SELECT FUND_CODE, FUNDNAME FROM FUND_INFO, wherein FUND_INFO represents the data asset relied on by A101, which is the data table of the source layer, that is, the physical table storing A001; for another example, the A201 combination net value belongs to the extracted layer index, and the processing logic is: SELECT FUND_CODE, DATE, VALUE FROM FUND_VALUE, wherein FUND_VALUE is the next layer aggregated table relied on by A201, that is, the physical table storing A102; the A102 combination layer combination valuation detail table belongs to the aggregated layer theme, and the processing logic is: SELECT FUND_CODE, DATE, VALUE, A1, A2, A3 FROM FUND_VALUE_SOURCE.

[0078] According to the data processing logic of each layer data, the assets of the upper aggregated layer and the extracted layer are integrated in turn to obtain the integrated processing result. The application makes full use of the result data of the source layer offline batch processing, improves the reuse degree of data, avoids repeated calculation of data that has not changed on the same day, and reduces the time occupied and the resources consumed by online calculation.

[0079] In the application, a unified standard SQL is used to define the processing logic between layers of data, which is convenient for developers to understand and reduces the research and development management cost.

[0080] As Figure 6 The structure of the batch processing data real-time processing device is shown in the figure, and the basic structure of the device is described, wherein the functional units and modules can be realized by software, general chips or special chips. The device specifically includes:

[0081] The user demand analysis unit 601 is configured to obtain report input parameters corresponding to a report required by a user, wherein the report input parameters include a target product, a time parameter, a report type, and data assets included in the report type.

[0082] The data asset list determination unit 602 is configured to determine a data asset list related to the report input parameters according to a report configuration management table, wherein the data asset list includes data sources and data flow directions, and the configuration management table includes a report configuration table and a data asset configuration table.

[0083] The judgment unit 603 is configured to determine whether the data sources are updated on the current day according to data update logic in the configuration management table.

[0084] The processing result acquisition unit 604 is configured to acquire current latest data of the data sources and process the current latest data of the data sources according to the configuration management table to obtain a processing result when the data sources are updated on the current day.

[0085] The user demand report determination unit 605 is configured to store the processing result and a previous day-end processing result of the data sources in an application layer as a user demand report when the data sources are not updated on the current day, or store the previous day-end processing result as the user demand report.

[0086] In the present application, the data asset list determination unit 602, the judgment unit 603, and the processing result acquisition unit 604 can be independently deployed and independent of an OLTP system in which upstream transactions occur, thereby reducing system coupling and reducing the performance impact of OLAP analysis and calculation tasks on OLTP transactions.

[0087] The present solution overcomes the shortcomings in the prior data analysis technology and can support real-time analysis and calculation of a large amount of data with less impact on OLTP transaction processing.

[0088] Figure 7 A schematic diagram of data flow directions in an embodiment of the present application is shown. The data flow directions of the source layer, the aggregation layer, and the extraction layer are shown in the diagram. The data form of the aggregation layer is a wide table formed by multiple tables of the source layer, which is a theme of the aggregation layer. For example, the aggregation layer includes data related to fund financing dimensions, and the transfer instruction status table and the transfer business detail table are integrated to obtain a new theme of the aggregation layer. The extraction layer can be classified by time, for example, data related to fund financing dimensions of each day in December 2022. In the diagram, source layer data A001 and A002 flow to the aggregation layer and are used as dependent data of aggregation layer data A101 and A102, respectively. Aggregation layer data A102 flows to the extraction layer and is used as dependent data of extraction layer data A201. Therefore, the data flow directions in the embodiments of the present application can be: source layer data flows to aggregation layer data; source layer data flows to aggregation layer data, and aggregation layer data flows to extraction layer data.

[0089] Figure 8 The diagram illustrates a real-time update of data assets according to an embodiment of this paper. When the user's requirement is to produce report A, and Table 1, used to generate report A, is confirmed to have been updated on the same day after the most recent end-of-day batch processing, the report configuration management table is used to determine the report input parameters corresponding to report A and the related data asset list. Furthermore, based on the report configuration management table, the data source and data flow of the data asset list are determined.

[0090] In this application, the remaining data assets (extraction layer data indicator 1, extraction layer data indicator 2, and aggregated data theme 1) involved in Table 1 (source layer data) will be updated. This will yield the updated Table 1, new theme 1, new indicator 1, and new indicator 2 for the current day, which will then generate new element 1 and new element 2 at the application layer. The final processed new element 1 and new element 2, along with other unupdated elements 3 to n, will be used as the report generation result.

[0091] In some embodiments of this specification, if the data assets in the source layer are block data, the latest data in the source layer is used to perform block updates on the source layer. Specifically, the update of block data will be based on the primary key of the table's block definition. For example, in a portfolio accounting table, there are approximately 2,000 accounting records under the same portfolio and date. After the business accounting is updated, the data under the same portfolio and date will be overwritten in blocks. Therefore, subsequent processing of data assets at each layer also needs to be updated according to the block definition.

[0092] In addition to supporting block data updates, this application also supports bar data updates, which will update the data according to the table's logical primary key.

[0093] like Figure 9As shown, a computer device 902 providing an embodiment herein can include one or more processors 904, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 902 can also include any memory 906 for storing any kind of information, such as code, settings, data, etc. Without limitation, for example, the memory 906 can include any one or combination of: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory can use any technology for storing information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 902. In one case, the computer device 902 can perform any operation of the associated instructions when executed by the processor 904, which are stored in any memory or combination of memories. The computer device 902 also includes one or more drive mechanisms 908, such as a hard disk drive mechanism, an optical disk drive mechanism, etc., for interacting with any memory.

[0094] The computer device 902 can also include an input / output module 910 (I / O) for receiving various inputs (via input devices 912) and for providing various outputs (via output devices 914). One particular output mechanism can include a presentation device 916 and an associated graphical user interface (GUI) 918. In other embodiments, the input / output module 910 (I / O), the input devices 912, and the output devices 914 can also not be included, just as a computer device in a network. The computer device 902 can also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the above-described components together.

[0095] The communication links 922 can be implemented in any manner, such as through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication links 922 can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols.

[0096] Corresponding to the method in Figures 1 to 5 The embodiments herein also provide a computer readable storage medium, having stored thereon a computer program, which when executed by a processor, performs the steps of the above method.

[0097] The embodiments herein also provide a computer readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the steps of the above method. Figures 1 to 5The method shown.

[0098] It should be understood that the size of the sequence number of each process described above in various embodiments herein does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments herein.

[0099] It should also be understood that in the embodiments herein, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships. For example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0100] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this paper.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0102] In several embodiments provided herein, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displays or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0103] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiments herein.

[0104] In addition, each functional unit in each embodiment herein can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0105] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions herein, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment herein. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.

[0106] The principles and implementation manners of the present application are described herein by using specific embodiments, and the above description of the embodiments is only used to help understand the method and its core idea; meanwhile, for those skilled in the art, according to the idea herein, the specific implementation manners and application ranges can be changed, and the above description of the specification should not be understood as a limitation of the present application.

Claims

1. A batch data real-time processing method, characterized in that, The method comprises: resolving user real-time requirements to obtain report input parameters corresponding to user required reports, the report input parameters comprising data assets; determining a data asset list related to the report input parameters according to a report configuration management table, the data asset list comprising data sources; judging whether the data sources have been updated on the current day according to data update logic in the report configuration management table; if yes, obtaining new incremental data and processing the new incremental data according to the report configuration management table to obtain a processing result; storing the processing result and a previous day-end processing result of the data sources in an application layer as a user demand report; if no, using the previous day-end processing result as the user demand report; wherein the determining of the data asset list related to the report input parameters according to the report configuration management table comprises: determining a report number corresponding to the report type according to the report type in the report input parameters; determining data asset numbers required for forming a report represented by the report number from the report configuration management table according to the report number; determining dependent data asset numbers of the data asset numbers from the report configuration management table according to the data asset numbers, wherein the data assets and the dependent data assets constitute the data asset list; the report configuration management table records data sources and asset types of each type of report, the data sources comprising at least one of a source layer, an aggregation layer and an extraction layer, and the dependent data assets comprising source layer data assets, aggregation layer data assets and extraction layer data assets; the judging of whether the data sources have been updated on the current day comprises: determining data update checking logic corresponding to the source layer data assets from the report configuration management table according to the source layer data assets; determining whether there is incremental data in an upstream online transaction processing (OLTP) database using the data update checking logic; if yes, determining that the data sources have been updated on the current day.

2. The batch data real-time processing method according to claim 1, wherein, Before the obtaining of the new incremental data and the processing of the new incremental data according to the report configuration management table to obtain the processing result, the method comprises: determining incremental data acquisition logic corresponding to the source layer data assets from the report configuration management table; acquiring the incremental data from the OLTP database using the incremental data acquisition logic; replacing historical data assets in the source layer with the incremental data to obtain the latest data of the source layer.

3. The batch data real-time processing method according to claim 2, wherein, The method further comprises: determining data processing logic corresponding to the aggregation layer data assets and the extraction layer data assets from the report configuration management table, wherein the data processing logic corresponds to each type of data source; processing data source distributions belonging to each type of asset according to the data processing logic and the latest data of the source layer to form a processing result.

4. The batch data real-time processing method according to claim 2, wherein, The method comprises: if the data assets of the source layer are block data, performing block updating on the source layer using the latest data of the source layer.

5. A batch data real-time processing apparatus characterized by comprising: The device comprises: a user real-time requirement analysis unit configured to analyze user real-time requirements to obtain report input parameters corresponding to user required reports, the report input parameters comprising target products, time parameters, report types and data assets included in the report types. The data asset list determination unit is configured to determine a data asset list related to the report input parameter according to a report configuration management table, the data asset list including data sources and data flow directions, and the report configuration management table including a report configuration table and a data asset configuration table. The judgment unit is configured to judge whether the data sources are updated on the current day according to data update logic in the report configuration management table. The processing result acquisition unit is configured to acquire current latest data of the data sources and process the current latest data of the data sources according to the report configuration management table to obtain a processing result if the data sources are updated on the current day. The user demand report determination unit is configured to store the processing result and a previous day-end processing result of the data sources in an application layer as a user demand report, or store the previous day-end processing result as the user demand report if the data sources are not updated on the current day. The data asset list determination unit is specifically configured to determine a report number corresponding to a report type in the report input parameter, determine data asset numbers required for forming a report represented by the report number from the report configuration management table according to the report number, and determine dependent data asset numbers of the data asset numbers from the report configuration management table according to the data asset numbers, the data asset and the dependent data asset constituting the data asset list. The judgment unit is specifically configured to determine data update check logic corresponding to the source layer data asset from the report configuration management table according to the source layer data asset, determine whether there is incremental data in an upstream online transaction processing (OLTP) database using the data update check logic, and determine that the data sources are updated on the current day if there is the incremental data.

6. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores the computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 4.

Images