Batch processing method, device and equipment for business data, medium and program product

Abstract

The invention provides a business data batch processing method and device, equipment, a medium and a program product. Relates to the technical field of data processing. The method comprises the following steps: acquiring daily cross-section data of a multi-source system and loading the daily cross-section data to a data lake pasting source layer; extracting section data of a preset date regularly every day, and processing to obtain preliminary integrated data; acquiring a data missing detection threshold value, judging whether the data is missing or not according to the data missing detection threshold value, and if yes, circularly complementing missing date data to obtain target integrated data and loading the target integrated data to an integration layer; the integration layer stores the target integration data according to daily increment; and loading the daily target integration data to a summary layer, and storing the monthly target integration data by the summary layer according to monthly increment. According to the method, automation and standardization of business data batch processing are achieved, missing data can be automatically recognized and complemented, timeliness and integrity of data processing are considered under limited computing resources, the storage and computing resource utilization rate is optimized, and reliable support is provided for downstream business data application with different granularities.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, equipment, medium, and program product for batch processing of business data. Background Technology

[0002] Industries such as finance, e-commerce, and freight, which have high requirements for the timeliness and accuracy of data analysis, typically need to process massive amounts of data. Core processes in these industries, such as business decision-making, operational monitoring, and strategy optimization, all require the support of massive amounts of data. Multi-source, heterogeneous cross-sectional data covers core information across the entire business chain. To achieve efficient management and value extraction of this data, enterprises generally adopt data lakes or massively parallel processing systems to build centralized data storage architectures. Through data extraction, transformation, and loading operations, they clean, integrate, and process raw data, ultimately forming directly applicable operational indicator data to provide data support for business scenarios.

[0003] The typical data recovery method in the existing technology is based on offline batch processing. First, the real-time data of the source system is imported into the offline library to form a data profile. Then, data processing operations are performed in the offline library to integrate multi-source data. Subsequently, missing or abnormal data is extracted to form recovery data. Finally, the recovery data is written into the source library and the target data is generated.

[0004] Existing technologies struggle to dynamically adjust processing strategies based on upstream data latency fluctuations. Under the constraint of limited computing resources, they cannot effectively solve the problem of automating data recovery, making it difficult to balance the timeliness and accuracy of data integration. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for batch processing of business data, which can supplement missing business data and ensure the integrity and timeliness of integrated and summarized data.

[0006] In a first aspect, embodiments of this application provide a method for batch processing of business data, including:

[0007] Obtain daily cross-sectional data from multiple source systems and load it into the source-attached layer of the data lake;

[0008] Every day, all cross-sectional data for a preset date are extracted from the data lake at a set time, and the cross-sectional data for the preset date are processed to obtain preliminary integrated data;

[0009] Obtain the data missing detection threshold;

[0010] The preliminary integrated data is judged for missing data according to the data missing detection threshold to obtain the missing judgment result. When the missing judgment result is detected as missing, a data loop processing operation is performed to complete all cross-sectional data of the source system corresponding to the missing data date to obtain the target integrated data of the preset date, and the target integrated data of the preset date is loaded into the integration layer.

[0011] The target integrated data is stored daily in an incremental manner through the integration layer;

[0012] The target integration data corresponding to each date is loaded into the summary layer, and the monthly target integration data corresponding to each date is stored in the summary layer in a monthly incremental manner.

[0013] In one possible implementation, the data missing detection threshold includes a time threshold and a rerun number threshold;

[0014] The process involves determining data missing from the initially integrated data based on the data missing detection threshold, obtaining a missing data determination result, and then, when a missing data determination result is detected, performing a data loop processing operation to complete all cross-sectional data of the source system corresponding to the missing data date, thereby obtaining the target integrated data for the preset date, including:

[0015] Based on the time threshold, detect whether the preliminary integrated data is missing, and determine the missing date;

[0016] Based on the missing dates, perform daily batch data processing to obtain integrated data for the missing dates, and load the integrated data for the missing dates into the integration layer until all the integrated data for the missing dates is completed or the rerun threshold is reached.

[0017] All the integrated data with missing dates are used as filler data and inserted into the initial integrated data according to the date to obtain the target integrated data for the preset date.

[0018] In one possible implementation, storing the daily target integrated data via the integration layer in a daily incremental manner further includes:

[0019] When manually entered data is detected, the manually entered data is copied to the integration layer through the data visualization platform, and the integration layer stores the manually entered data in daily increments.

[0020] In one possible implementation, after determining the missing data in the initially integrated data based on the data missing detection threshold and obtaining the missing data determination result, the method further includes:

[0021] When the missing data determination result is not missing, the preliminary integrated data is directly processed by daily batch data processing to obtain the daily target integrated data.

[0022] In one possible implementation, after storing the target integrated data in daily increments through the integration layer, the process further includes:

[0023] The daily target integration data of the integration layer is copied to the downstream system for daily display and processing of the target integration data.

[0024] In one possible implementation, after storing the monthly target consolidation data corresponding to each date in a monthly incremental manner through the aggregation layer, the method further includes:

[0025] The monthly target data from the aggregation layer is copied to the downstream system for display processing.

[0026] Secondly, embodiments of this application provide a business data batch processing apparatus, comprising:

[0027] The data acquisition module is used to acquire daily cross-sectional data from multiple source systems and load it into the source-attached layer of the data lake;

[0028] The initial processing module is used to extract all cross-sectional data for a preset date from the data lake at a set time every day, and to process all cross-sectional data for the preset date to obtain preliminary integrated data;

[0029] The acquisition module is used to acquire the data missing detection threshold;

[0030] The data completion module is used to perform data missing judgment on the preliminary integrated data according to the data missing detection threshold, and obtain the missing judgment result. When the missing judgment result is detected as missing, a data loop processing operation is performed to complete all cross-sectional data of the source system corresponding to the missing data date, obtain the target integrated data of the preset date, and load the target integrated data of the preset date into the integration layer.

[0031] The data replenishment module is also used to store the daily target integrated data through the integration layer in a daily incremental manner;

[0032] The data replenishment module is also used to load the target integrated data corresponding to each date into the summary layer, and to store the monthly target integrated data corresponding to each date in a monthly incremental manner through the summary layer.

[0033] In one possible implementation, the data recovery module is specifically used for:

[0034] Based on the time threshold, detect whether the preliminary integrated data is missing, and determine the missing date;

[0035] Based on the missing dates, perform daily batch data processing to obtain integrated data for the missing dates, and load the integrated data for the missing dates into the integration layer until all the integrated data for the missing dates is completed or the rerun threshold is reached.

[0036] All the integrated data with missing dates are used as filler data and inserted into the initial integrated data according to the date to obtain the target integrated data for the preset date.

[0037] In one possible implementation, the data recovery module is specifically used for:

[0038] When manually entered data is detected, the manually entered data is copied to the integration layer through the data visualization platform, and the integration layer stores the manually entered data in daily increments.

[0039] In one possible implementation, the data recovery module is further configured to:

[0040] When the missing data determination result is not missing, the preliminary integrated data is directly processed by daily batch data processing to obtain the daily target integrated data.

[0041] In one possible implementation, the data recovery module is further configured to:

[0042] The daily target integration data of the integration layer is copied to the downstream system for daily display and processing of the target integration data.

[0043] In one possible implementation, the data recovery module is further configured to:

[0044] The monthly target data from the aggregation layer is copied to the downstream system for display processing.

[0045] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0046] The memory stores computer-executed instructions;

[0047] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0048] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0049] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0050] The business data batch processing method, apparatus, device, medium, and program products provided in this application collect daily snapshot data from multiple source systems and store it in the raw storage layer of a data lake. Each day, all data for a specified date is extracted from the data lake for preliminary processing and integration. A set data missing detection standard is used to determine if any data in the preliminary integration is missing. When missing data is detected, the system automatically processes and fills in the missing dates, forming complete target integrated data, which is then stored as newly added data each day. Finally, the daily data is loaded into the summary layer and stored as newly added data each month. By automatically identifying and actively filling in missing data, manual operation is eliminated, reducing human error and waiting time, and ensuring the integrity of the daily integrated data. Simultaneously, the daily and monthly incremental storage mode avoids resource waste caused by repeated data processing and ensures timely data updates, allowing subsequent use of this data to obtain timely and complete information. Ultimately, this improves the automation level of business data batch processing and ensures the integrity and timeliness of integrated and summarized data. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1 A flowchart illustrating the batch processing method for business data provided in this application;

[0053] Figure 2 A schematic diagram of the business data batch processing device provided in this application;

[0054] Figure 3 A schematic diagram of the structure of the electronic device provided in this application.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0057] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0058] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0059] First, let me explain the terms used in this application:

[0060] Data lake: refers to a distributed storage system used for centralized storage of multi-source heterogeneous raw data (including structured and unstructured data), providing a unified data storage foundation for the integration and processing of business data in this application;

[0061] Source layer: refers to the basic data storage layer of the data lake, which is used to store the raw data of the source system without processing, maintaining the original structure and content of the data, so as to facilitate subsequent traceability and reuse;

[0062] Cross-sectional data refers to a snapshot of business data collected at a specific point in time. In this application, it specifically refers to the business data generated daily by each source system.

[0063] Integration layer: This refers to the intermediate data layer used to store the daily target data after processing, integration, and completion. It uses a daily incremental method to store data, ensuring data integrity and timeliness.

[0064] The aggregation layer is used to store data that has been aggregated and processed monthly based on the daily data from the integration layer. It stores data in monthly increments to meet the analysis needs of monthly indicators.

[0065] Data missing detection threshold: refers to the quantitative standard used to determine whether the initially integrated data is missing, including time threshold and rerun number threshold, which are used to locate the missing date and constrain the number of loops for data completion, respectively;

[0066] Daily incremental storage: This refers to a storage method that only writes newly added target data to the integration layer each day, avoiding duplicate storage of historical data and optimizing storage resource usage.

[0067] Monthly incremental storage: This refers to a storage method that only writes newly added monthly summary data to the summary layer, thereby improving the efficiency of monthly data storage and updates.

[0068] In existing technologies, data replenishment through offline batch processing relies on manual triggering or fixed threshold processing strategies. This results in the inability to dynamically adapt to data latency fluctuations and the difficulty in automating data replenishment while ensuring data integrity and timeliness under limited computing resources.

[0069] The business data batch processing method provided in this application solves the technical problems in the prior art that it is impossible to dynamically adapt to data latency fluctuations and is difficult to automate data replenishment under limited computing resources while taking into account data integrity and timeliness. It collects daily cross-sectional data from multiple sources to the data lake source layer, extracts data for preset dates for preliminary integration on a daily schedule, automatically identifies missing data based on a data missing detection threshold and cyclically completes the target integrated data, and then stores it to the integration layer on a daily incremental basis and to the summary layer on a monthly incremental basis.

[0070] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0071] Figure 1 A flowchart illustrating the batch processing method for business data provided in this application is shown below. Figure 1 As shown in the figure, this embodiment provides a method for batch processing of business data, which includes the following steps:

[0072] Step S101: Obtain daily cross-sectional data from multiple source systems and load them into the source layer of the data lake.

[0073] Specifically, based on the raw data storage characteristics of the data lake's source layer, multiple source systems are connected via data acquisition tools or interfaces to capture cross-sectional data according to the daily time dimension. The unprocessed raw data is then fully loaded into the source layer, maintaining its original structure and content. This achieves centralized aggregation of multi-source heterogeneous cross-sectional data, breaking down data silos between different source systems. It provides a unified and traceable raw data source for subsequent data processing, ensuring the integrity and traceability of the raw data, laying the data foundation for the entire batch data processing workflow, and avoiding deviations in subsequent processed data caused by dispersed data sources.

[0074] Step S102: Extract all cross-sectional data for a preset date from the data lake at a set time each day, and process all cross-sectional data for the preset date to obtain preliminary integrated data.

[0075] Specifically, by triggering data processing tasks at set intervals, full cross-sectional data for a preset date is extracted from the data lake's source layer. Through standardized processing operations such as data cleaning, field mapping, and multi-source association, the initial integration and transformation of the raw data is completed. This transforms the scattered raw data into structured, pre-integrated data, achieving format unification and logical association among multi-source data. This provides standardized processing objects for subsequent data missing detection. Preliminary integrated data is output at fixed time intervals, ensuring the regularity and standardization of data processing, reducing the complexity of subsequent data processing, and providing a fundamental guarantee for data timeliness.

[0076] Step S103: Obtain the data missing detection threshold.

[0077] Specifically, based on the business's need for timely data, the historical patterns of data latency in upstream systems, and the computing resource capacity of the MPP (Massively Parallel Processing) cluster, a quantitative standard for data missing detection is pre-defined. This standard includes core parameters such as time thresholds and rerun thresholds. It provides an objective and actionable quantitative basis for data missing judgment, avoiding detection biases caused by subjective judgment. Simultaneously, it sets boundary conditions for subsequent automatic completion operations, establishing a standardized missing detection benchmark to ensure the consistency and rationality of data missing judgments. This provides precise support for triggering the automatic completion process, balancing data completion effectiveness with resource consumption.

[0078] Step S104: Based on the data missing detection threshold, perform a data missing judgment on the preliminary integrated data to obtain the missing judgment result. When the missing judgment result is detected as missing, perform a data loop processing operation to complete all cross-sectional data of the source system corresponding to the missing data date, obtain the target integrated data of the preset date, and load the target integrated data of the preset date into the integration layer.

[0079] Specifically, the system performs integrity checks on the initially integrated data using preset detection thresholds, identifying the date ranges corresponding to missing data. When missing data is found, a loop re-run mechanism is triggered, invoking daily batch processing logic to re-extract the original data for the missing dates and integrate it until the data is complete or the re-run threshold is reached. Finally, the complete target integrated data is loaded into the integration layer. This achieves automatic identification and accurate completion of missing data, replacing the traditional manual triggering method, solving the problem of incomplete integrated data caused by upstream data delays, and avoiding excessive consumption of computing resources through re-run threshold constraints. It significantly improves the automation level of data completion, ensures the integrity of the target data in the integration layer, balances data completion efficiency and resource utilization under limited computing resources, and reduces the lag and errors caused by manual intervention.

[0080] Step S105: Store the daily target integration data in an incremental manner through the integration layer.

[0081] Specifically, incremental storage technology, based on timestamps or data version identifiers, only newly added target integrated data each day is written to the integration layer, avoiding duplicate writing and overwriting of historically stored data. This optimizes the storage resource usage of the integration layer, reduces input / output overhead during data writing, ensures the continuity and traceability of daily data, facilitates subsequent incremental data queries and updates, improves data storage and update efficiency, reduces waste of computing and storage resources, and ensures that the integrated layer data can reflect the latest business situation in a timely manner, providing efficient support for downstream daily data applications.

[0082] Step S106: Load the target integration data corresponding to each date into the summary layer, and store the monthly target integration data corresponding to each date in a monthly incremental manner through the summary layer.

[0083] Specifically, based on a monthly time period, the daily target integration data for the corresponding month in the integration layer is aggregated and calculated to generate monthly summary data. An incremental storage mechanism is used to write only the newly added summary results for the current month to the summary layer. This achieves a hierarchical transformation from daily detailed data to monthly summary data, forming a two-level daily and monthly data storage structure. This meets the downstream needs for operational indicator data of different granularities, ensures the accuracy and completeness of monthly summary data, optimizes the generation and updating efficiency of monthly data, and provides high-quality data support for downstream monthly summary analysis, business decision-making, and other scenarios.

[0084] The business data batch processing method provided in this invention achieves automation and standardization of business data batch processing through a full-process design including multi-source data collection and lake entry, timed standardized preliminary integration, threshold-based missing detection and automatic cyclic completion, and daily and monthly incremental storage. It effectively solves the data missing problem caused by upstream data delays, balances the timeliness and completeness of data processing under limited computing resources, and optimizes the utilization rate of storage and computing resources, providing reliable support for downstream business data applications of different granularities.

[0085] This embodiment details the process described in the above embodiment of judging data missing from the initially integrated data based on a data missing detection threshold, obtaining a missing data judgment result, and then performing a data loop processing operation when a missing data judgment result is detected. This operation completes all cross-sectional data of the source system corresponding to the missing data date to obtain the target integrated data for the preset date. The data missing detection threshold includes a time threshold and a rerun number threshold. The specific implementation of this process includes the following steps:

[0086] Step a1: Detect whether the initially integrated data is missing based on the time threshold, and determine the missing dates.

[0087] Specifically, based on preset time thresholds, data integrity verification rules are constructed to check the continuity of the time dimension of the initially integrated data. By comparing the date data that should exist within the preset time range with the date data in the actual initially integrated data, dates that do not match are located and identified as missing data dates. By accurately identifying the specific time nodes missing in the initially integrated data, clear target objects are provided for subsequent data completion operations, avoiding the blindness of the completion process, achieving precise location of missing data, reducing invalid completion operations, laying the foundation for efficient subsequent data completion work, and improving the pertinence of the overall data processing workflow.

[0088] Step a2: Perform daily batch data processing based on the missing dates to obtain the integrated data for the missing dates, and load the integrated data for the missing dates into the integration layer until all the integrated data for the missing dates is completed or the rerun threshold is reached.

[0089] Specifically, for a known missing date, the data processing logic is consistent with the daily batch processing. The full cross-sectional data of the corresponding missing date is extracted from the data lake source layer again, and a standardized integration processing operation is performed to generate integrated data of the missing date and load it into the integration layer. At the same time, a rerun threshold is introduced as a safety constraint. When the data is completed or the threshold is reached, the loop processing is terminated to avoid unlimited occupation of computing resources.

[0090] By performing specific missing data completion operations, the existing batch processing logic is reused to ensure the consistency of the completed data. By balancing the data completion demand and computing resource capacity through the rerun threshold, the inefficiency of the traditional completion process that relies on manual triggering is solved, and the automatic completion of missing data is achieved. This ensures the integrity of the data in the integration layer, while avoiding excessive consumption of computing resources, taking into account both data completion efficiency and resource utilization, and reducing processing delays caused by manual intervention.

[0091] Step a3: Insert the integrated data with all missing dates as the completion data into the initial integrated data according to the date to obtain the target integrated data for the preset date.

[0092] Specifically, based on the date field as the core association identifier, data insertion is used to integrate the missing date data obtained from the completion process into the initial integrated dataset. The integrated data is then structured and organized according to chronological order to form a complete dataset covering the preset dates. This process merges the completed data with the original initial integrated data, eliminating data gaps and forming logically coherent and chronologically complete target integrated data. This provides standardized data for subsequent incremental storage and aggregation processing, ensuring the integrity and temporal continuity of the target integrated data within the preset dates, improving data usability, and providing high-quality foundational data for downstream business scenarios such as daily display and monthly aggregation.

[0093] This invention achieves precise and automated data missing completion by using a time threshold to accurately locate the date of missing data, repeatedly calling batch processing logic to complete the data and controlling resource consumption with a rerun threshold, and merging the completed data by date with the initial integrated data. This ensures the integrity of the target integrated data, optimizes the utilization of computing resources, and effectively solves the problem of data incompleteness caused by upstream data delays.

[0094] In some optional implementations, when storing daily target integration data in a daily incremental manner through the integration layer, the method further includes:

[0095] When manually entered data is detected, the data visualization platform copies the manually entered data to the integration layer, and the integration layer stores the manually entered data in daily increments.

[0096] Specifically, based on the data operation monitoring mechanism, the manually entered data generated by the data visualization platform is perceived in real time. The entered data is transmitted to the integration layer through a standardized data synchronization link. Following the technical specifications of daily incremental storage, the business date corresponding to the entered data is used as the core identifier to perform incremental write operations. At the same time, data consistency verification logic is embedded to ensure that the entered data is structurally compatible and logically consistent with the existing target integration data in the integration layer.

[0097] By supplementing data sources beyond those automatically collected by the system, the coverage of the integration layer data is improved, addressing the need for supplementing data that the upstream system cannot push or has missed. Through the collaborative design of consistency verification and incremental storage, the seamless integration of manually entered data with the original target integration data is achieved, avoiding data redundancy or logical breaks and ensuring the integrity of the integration layer data.

[0098] It effectively improves the integrity and comprehensiveness of the integrated layer data, enabling the integrated data to cover both automatic collection and manual data entry scenarios, eliminating data gaps; at the same time, it maintains the efficiency and standardization of integrated layer data storage, providing more comprehensive and reliable basic data support for downstream business applications such as daily data display and monthly summary processing.

[0099] The embodiments of the present invention improve the data supplementation mechanism by monitoring manually entered data, standardizing and synchronizing it to the integration layer, and performing incremental storage and consistency verification. This enhances the integrity and compatibility of the data in the integration layer and effectively adapts to the scenario of manually entered supplementary data sources.

[0100] In some optional implementations, after determining the missing data in the initially integrated data based on a missing data detection threshold and obtaining the missing data determination result, the method further includes:

[0101] When the missing data detection result is not missing, the preliminary integrated data is directly processed by daily batch data processing to obtain the daily target integrated data.

[0102] Specifically, based on the data missing detection result judgment logic, when the verification confirms that the initially integrated data is not missing, the standardized data processing logic of daily batch processing can be directly reused without triggering an additional data completion loop. The initially integrated data is processed through unified field cleaning, format conversion, and multi-source association rules to ensure the consistency of data processing logic. At the same time, it avoids the repeated development of independent processing modules and solves the problems of redundant processing logic and disconnection from daily processing flow in the original process.

[0103] By clearly defining the processing path in scenarios with no missing data, a complete closed-loop process of detection, judgment, and processing is formed, avoiding processing delays caused by unclear process branches. By reusing existing batch processing logic, the system development and maintenance costs are reduced, while ensuring that the processing standards for data without missing data are consistent with the processing standards for data after completion, eliminating data differences that may be caused by different processing paths.

[0104] It effectively simplifies the data processing workflow, shortens the data processing cycle in scenarios without missing data, and improves the timeliness of data output; relying on a unified processing logic, it ensures the structured and standardized nature of daily target data integration, providing high-quality and consistent data input for subsequent daily incremental storage and monthly summary processing, and ensuring the reliability of downstream data applications.

[0105] This invention, through its design of directly reusing the daily batch processing logic when there is no missing data, clarifies the process branches and eliminates logical redundancy, thereby improving data processing efficiency and ensuring data processing consistency, making the entire batch processing process more complete and practical.

[0106] In some optional implementations, after storing the target integrated data in daily increments through the integration layer, the method further includes:

[0107] The daily target integration data from the integration layer is copied to the downstream system for daily target integration data display and processing.

[0108] Specifically, based on the structured characteristics of daily incremental storage in the integration layer, a standardized data synchronization protocol is adopted to transmit the daily target integration data, which has been completed and verified, to the downstream system. During the synchronization process, data format adaptation and integrity verification logic is embedded to ensure that the data conforms to the data reception specifications of the downstream system. This solves the logical problems of incompatible data synchronization formats and lack of transmission integrity in the original process. At the same time, the incremental synchronization principle is followed, and only newly added daily target integration data is transmitted to avoid redundant data transmission and resource consumption.

[0109] By accurately pushing standardized and complete daily target data to downstream display scenarios, the data flow path from processing to display is clearly defined, meeting downstream needs for detailed daily business data display. Simultaneously, a synchronous verification mechanism avoids information loss or deviation during data flow, ensuring consistency between displayed data and integrated layer data. This enables downstream systems to obtain complete and standardized daily target data in a timely manner, ensuring the timeliness and reliability of daily data display. It provides direct and high-quality data support for operational monitoring and decision analysis at the business end, avoiding business judgment biases caused by data transmission delays or incompleteness, and improving the efficiency and practicality of downstream data applications.

[0110] This invention, through a collaborative design that integrates incremental data synchronization, format adaptation, and integrity verification at the integration layer, connects the entire data processing and display chain. This ensures the timeliness and accuracy of daily data display downstream while optimizing data transmission resource usage, effectively meeting the core business requirements for daily detailed data display.

[0111] In some optional implementations, after storing the monthly target consolidation data for each date in a monthly incremental manner through the aggregation layer, the method further includes:

[0112] The monthly target data from the summary layer is copied to the downstream system for display and processing.

[0113] Specifically, based on the structured and complete characteristics of the monthly incremental storage of the aggregation layer, a standardized data transmission protocol is adopted to push the aggregated and verified monthly target integrated data to the downstream system. During the synchronization process, a data format adaptation and consistency verification mechanism is incorporated to ensure that the data conforms to the display specifications of the downstream system. This solves the problems of disconnection between the monthly and daily data synchronization logic, format incompatibility, and lack of guarantee of transmission integrity in the original process. At the same time, the incremental synchronization principle is followed, and only the newly added aggregated data of the current month is transmitted to reduce resource redundancy consumption.

[0114] By constructing a closed-loop monthly data flow from the aggregation layer to downstream systems, the complete path of monthly aggregated data from processing to display is clearly defined. This meets the downstream's needs for displaying monthly aggregated operational metrics. Through a verification mechanism during the synchronization process, the consistency between the displayed data and the original data at the aggregation layer is ensured, avoiding information deviations in the data flow process. Simultaneously, it complements the daily data display process, improving the downstream multi-granularity data display system. This enables downstream systems to obtain standardized and comprehensive monthly target integration data in a timely manner, ensuring the accuracy and timeliness of monthly data display. It provides reliable data support for monthly review and strategy optimization scenarios on the business side, avoiding inefficient business decisions caused by delayed or inaccurate monthly data transmission, and further enhancing the comprehensiveness and practicality of overall data application.

[0115] This invention, through a collaborative design involving targeted synchronization of monthly incremental data at the aggregation layer, format adaptation, and consistency verification, streamlines the process from monthly data processing to display. This ensures the accuracy and timeliness of downstream monthly data display, optimizes data transmission efficiency, and improves the multi-granularity data display system, effectively meeting the core business needs for displaying monthly aggregated indicators.

[0116] Figure 2 A schematic diagram of the business data batch processing device provided in this application. Figure 2 As shown, the business data batch processing device 20 includes:

[0117] The data acquisition module 201 is used to acquire daily cross-sectional data from multiple source systems and load it into the source-attached layer of the data lake;

[0118] The initial processing module 202 is used to extract all cross-sectional data of a preset date from the data lake at a fixed time every day, and to process all cross-sectional data of the preset date to obtain preliminary integrated data;

[0119] Module 203 is used to acquire the data missing detection threshold;

[0120] The data replenishment module 204 is used to make a data missing judgment on the preliminary integrated data according to the data missing detection threshold, and obtain the missing judgment result. When the missing judgment result is detected as missing, a data loop processing operation is performed to complete all cross-sectional data of the source system corresponding to the missing data date, obtain the target integrated data of the preset date, and load the target integrated data of the preset date into the integration layer.

[0121] The data replenishment module 204 is also used to store the daily target integrated data in an incremental manner through the integration layer;

[0122] The data replenishment module 204 is also used to load the target integration data corresponding to each date into the summary layer, and store the monthly target integration data corresponding to each date in a monthly incremental manner through the summary layer.

[0123] In one possible implementation, the data recovery module 204 is specifically used for:

[0124] Based on time threshold detection, preliminary integrated data is checked for missing data, and the missing dates are determined.

[0125] Based on the missing dates, perform daily batch data processing to obtain integrated data for the missing dates, and load the integrated data for the missing dates into the integration layer until all the integrated data for the missing dates is completed or the rerun threshold is reached.

[0126] All missing dates in the integrated data are used as filler data and inserted into the initial integrated data according to the date to obtain the target integrated data for the preset date.

[0127] In one possible implementation, the data recovery module 204 is specifically used for:

[0128] When manually entered data is detected, the data visualization platform copies the manually entered data to the integration layer, and the integration layer stores the manually entered data in daily increments.

[0129] In one possible implementation, the data recovery module 204 is further configured to:

[0130] When the missing data detection result is not missing, the preliminary integrated data is directly processed by daily batch data processing to obtain the daily target integrated data.

[0131] In one possible implementation, the data recovery module 204 is further configured to:

[0132] The daily target integration data from the integration layer is copied to the downstream system for daily target integration data display and processing.

[0133] In one possible implementation, the data recovery module 204 is further configured to:

[0134] The monthly target data from the summary layer is copied to the downstream system for display and processing.

[0135] The business data batch processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0136] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. Figure 3 As shown, the electronic device 30 provided in this embodiment includes at least one processor 301 and a memory 302. Optionally, the electronic device 30 further includes a communication component 303. The processor 301, memory 302, and communication component 303 are connected via a bus.

[0137] In a specific implementation, at least one processor 301 executes computer execution instructions stored in memory 302, causing at least one processor 301 to perform the above-described method.

[0138] The specific implementation process of processor 301 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0139] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0140] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0141] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0142] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0143] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0144] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0145] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0146] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0149] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0151] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for batch processing of business data, characterized in that, include: Obtain daily cross-sectional data from multiple source systems and load it into the source-attached layer of the data lake; Every day, all cross-sectional data for a preset date are extracted from the data lake at a set time, and the cross-sectional data for the preset date are processed to obtain preliminary integrated data; Obtain the data missing detection threshold; The preliminary integrated data is judged for missing data according to the data missing detection threshold to obtain the missing judgment result. When the missing judgment result is detected as missing, a data loop processing operation is performed to complete all cross-sectional data of the source system corresponding to the missing data date to obtain the target integrated data of the preset date, and the target integrated data of the preset date is loaded into the integration layer. The target integrated data is stored daily in an incremental manner through the integration layer; The target integration data corresponding to each date is loaded into the summary layer, and the monthly target integration data corresponding to each date is stored in the summary layer in a monthly incremental manner.

2. The method according to claim 1, characterized in that, The data missing detection threshold includes a time threshold and a rerun number threshold; The process involves determining data missing from the initially integrated data based on the data missing detection threshold, obtaining a missing data determination result, and then, when a missing data determination result is detected, performing a data loop processing operation to complete all cross-sectional data of the source system corresponding to the missing data date, thereby obtaining the target integrated data for the preset date, including: Based on the time threshold, detect whether the preliminary integrated data is missing, and determine the missing date; Based on the missing dates, perform daily batch data processing to obtain integrated data for the missing dates, and load the integrated data for the missing dates into the integration layer until all the integrated data for the missing dates is completed or the rerun threshold is reached. All the integrated data with missing dates are used as filler data and inserted into the initial integrated data according to the date to obtain the target integrated data for the preset date.

3. The method according to claim 1, characterized in that, When storing the daily target integrated data in a daily incremental manner through the integration layer, it further includes: When manually entered data is detected, the manually entered data is copied to the integration layer through the data visualization platform, and the integration layer stores the manually entered data in daily increments.

4. The method according to claim 1, characterized in that, After determining the missing data in the initially integrated data based on the data missing detection threshold and obtaining the missing data determination result, the method further includes: When the missing data determination result is not missing, the preliminary integrated data is directly processed by daily batch data processing to obtain the daily target integrated data.

5. The method according to claim 1, characterized in that, After storing the target integrated data in daily increments through the integration layer, the process further includes: The daily target integration data of the integration layer is copied to the downstream system for daily display and processing of the target integration data.

6. The method according to any one of claims 1 to 5, characterized in that, After storing the monthly target integrated data corresponding to each date in a monthly incremental manner through the aggregation layer, the process also includes: The monthly target data from the aggregation layer is copied to the downstream system for display processing.

7. A business data batch processing device, characterized in that, include: The data acquisition module is used to acquire daily cross-sectional data from multiple source systems and load it into the source-attached layer of the data lake; The initial processing module is used to extract all cross-sectional data for a preset date from the data lake at a set time every day, and to process all cross-sectional data for the preset date to obtain preliminary integrated data; The acquisition module is used to acquire the data missing detection threshold; The data completion module is used to perform data missing judgment on the preliminary integrated data according to the data missing detection threshold, and obtain the missing judgment result. When the missing judgment result is detected as missing, a data loop processing operation is performed to complete all cross-sectional data of the source system corresponding to the missing data date, obtain the target integrated data of the preset date, and load the target integrated data of the preset date into the integration layer. The data replenishment module is also used to store the daily target integrated data through the integration layer in a daily incremental manner; The data replenishment module is also used to load the target integrated data corresponding to each date into the summary layer, and to store the monthly target integrated data corresponding to each date in a monthly incremental manner through the summary layer.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images