Batch data cross-day zero-latency handover method and apparatus

Abstract

The embodiment of the application provides a batch data cross-sky zero-delay switching method and device, and relates to the technical field of data processing. The method comprises the following steps: obtaining a real-time query task; in the case that there is an unfinished batch running task at present, the real-time query task is decomposed into a plurality of offline batch tasks and a plurality of real-time streaming tasks, each task is used to obtain a corresponding aggregation result; obtaining a real-time query result according to the obtained aggregation result, the real-time query result is used as a response to the real-time query task. The embodiment provided by the application not only solves the problem that the batch running data is inaccurate during the zero-point switching, but also avoids the resource problem caused by the traditional use of real-time calculation to solve the problem, so that the batch data calculation can play a role in a wider range of scenarios.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a batch data cross-day zero-delay switching method, a batch data cross-day zero-delay switching device, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In the field of big data processing, daily batch data queries are a common and crucial business operation. In traditional daily batch data query processes, the batch task is scheduled to start at midnight. If a user initiates a query request before the batch task is completed, hoping to retrieve data up to midnight, the correct query results will not be obtained. This problem is particularly pronounced in real-time query scenarios. The inability to obtain accurate data in a timely manner directly leads to discrepancies between the queried data and the actual situation, severely impacting the accuracy and effectiveness of business decisions.

[0003] To address the aforementioned issues, the industry commonly employs real-time streaming computing solutions to replace traditional batch processing. Real-time streaming computing can process data streams in real time, meeting the needs of real-time queries to a certain extent and avoiding inaccurate queries caused by incomplete batch processing tasks. However, this solution also has significant drawbacks in practical applications. When dealing with aggregation calculations over large time spans, real-time streaming computing needs to continuously maintain a large amount of state information, placing extremely high demands on computing resources. Especially for aggregation calculations without time constraints, as the amount of data accumulates and processing time increases, the consumption of computing resources increases dramatically, severely impacting system performance and even leading to situations where normal processing fails. This significant challenge to computing resources greatly limits the application of batch data in business scenarios with high data accuracy requirements, causing numerous difficulties in data processing and querying for related businesses. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for zero-latency switching of batch data across days, and to propose a batch + aggregation fusion calculation method to at least solve some of the problems in the background art.

[0005] To achieve the above objectives, this application provides a method for zero-latency switching of batch data across days. The method includes: obtaining a real-time query task; if there are currently unfinished batch tasks, decomposing the real-time query task into several offline batch tasks and several real-time streaming tasks, each task being used to obtain a corresponding aggregation result; obtaining a real-time query result based on the obtained aggregation result, the real-time query result serving as a response to the real-time query task.

[0006] Optionally, the real-time query task is decomposed into several offline batch tasks and several real-time streaming tasks, including: decomposing the real-time query task into a first offline batch task, a second offline batch task, a first real-time streaming task, and a second real-time streaming task; the first offline batch task is used to obtain historical detailed data up to 0:00 on T-2 day and calculate the aggregation result as the first aggregation result; the second offline batch task is used to obtain historical detailed data up to 0:00 on T-1 day and calculate the aggregation result as the second aggregation result; the first real-time streaming task is used to obtain detailed data from 0:00 on T-2 day to the current time and the first aggregation result, and after performing aggregation operation according to the current time, obtain the third aggregation result; the second real-time streaming task is used to obtain detailed data from 0:00 on T-1 day to the current time and the first aggregation result, and after performing aggregation operation according to 0:00 on T+1 day, obtain the fourth aggregation result.

[0007] Optionally, obtaining real-time query results based on the obtained aggregation results includes: returning the latest data with the latest data update time from the second aggregation result, the third aggregation result, and the fourth aggregation result; and using the returned data as the real-time query result.

[0008] Optionally, the historical detailed data calculation and aggregation results up to 0:00 on T-2 day and the historical detailed data calculation and aggregation results up to 0:00 on T-1 day are obtained by executing a scheduled task daily and written into the corresponding result table, which is distinguished by the execution date.

[0009] Optionally, the method further includes: copying the result table to a cache or main memory before the batch task starts; and removing the result table from the cache or main memory after the batch task ends.

[0010] Optionally, the method further includes: displaying the real-time query results to the user through a microservice, wherein the configuration parameters of the microservice are determined based on the data template or user profile corresponding to the initiator of the real-time query task.

[0011] Optionally, the method further includes: obtaining the execution process log of the real-time query task, verifying the execution results of the decomposed first offline batch task, second offline batch task, first real-time streaming task and second real-time streaming task based on the execution process log, and setting the corresponding confidence level for the real-time query result according to the verification result.

[0012] This application also provides a batch data cross-day zero-latency switching device, the device comprising: a task acquisition module for acquiring real-time query tasks; a task decomposition module for decomposing the real-time query tasks into several offline batch tasks and several real-time streaming tasks when there are currently unfinished batch tasks, each task being used to obtain a corresponding aggregation result; and a result return module for obtaining real-time query results based on the obtained aggregation results, the real-time query results serving as a response to the real-time query tasks.

[0013] This application also provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the aforementioned batch data cross-day zero-latency switching method by executing the instructions stored in the memory.

[0014] This application also provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned batch data cross-day zero-latency switching method.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned batch data cross-day zero-latency switching method.

[0016] The above technical solution has the following beneficial effects: By integrating real-time and offline computing, this innovation avoids the challenges to computing resources posed by the need to maintain a large state for aggregate calculations with long time spans during real-time data updates. This innovation not only solves the problem of data inaccuracy during zero-point switching in batch processing, but also avoids the resource problems associated with traditional real-time computing for addressing this issue, enabling batch data computing to be effective in a wider range of scenarios.

[0017] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 This illustration schematically shows the steps of a batch data cross-day zero-delay handover method according to an embodiment of this application; Figure 2 The schematic diagram illustrates the principle of the batch data cross-day zero-delay switching method according to the embodiments of this application; Figure 3 This illustration schematically shows a structural diagram of a batch data cross-day zero-delay switching device according to an embodiment of this application; Figure 4 The diagram schematically illustrates the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the embodiments of this application.

[0020] Figure 1 The illustration schematically depicts the steps of a batch data cross-day zero-delay handover method according to an embodiment of this application. For example... Figure 1 As shown, a method for zero-latency batch data switching across days includes: S01, Obtain real-time query tasks; S02. If there are currently unfinished batch processing tasks, the real-time query task is decomposed into several offline batch tasks and several real-time streaming tasks, with each task used to obtain the corresponding aggregation result. S03. Obtain real-time query results based on the obtained aggregation results, and use the real-time query results as a response to the real-time query task.

[0021] Through the above implementation methods, by decomposing tasks and integrating real-time and offline computing, the challenges to computing resources posed by the need to maintain a large state for aggregate computing with a long time span in real-time data updates are avoided. This innovation not only solves the problem of data inaccuracy during zero-point switching of batch data, but also avoids the resource problems caused by traditional real-time computing to solve this problem, enabling batch data computing to be effective in a wider range of scenarios.

[0022] In some embodiments of this application, the real-time query task is decomposed into several offline batch tasks and several real-time streaming tasks, including: decomposing the real-time query task into a first offline batch task, a second offline batch task, a first real-time streaming task, and a second real-time streaming task; the first offline batch task is used to obtain historical detailed data up to 0:00 on T-2 day and calculate the aggregation result as the first aggregation result; the second offline batch task is used to obtain historical detailed data up to 0:00 on T-1 day and calculate the aggregation result as the second aggregation result; the first real-time streaming task is used to obtain detailed data from 0:00 on T-2 day to the current time and the first aggregation result, and after performing aggregation operation according to the current time, obtain a third aggregation result; the second real-time streaming task is used to obtain detailed data from 0:00 on T-1 day to the current time and the first aggregation result, and after performing aggregation operation according to 0:00 on T+1 day, obtain a fourth aggregation result. This embodiment describes the specific details of task decomposition, and the final real-time query result is generated based on the aforementioned multiple aggregation results.

[0023] In some embodiments of this application, obtaining real-time query results based on the obtained aggregation results includes: returning the latest data with the latest update time from the second, third, and fourth aggregation results; and using the returned data as the real-time query result. During data querying, the latest data with the latest update time is retrieved from the real-time data result table for day T, the real-time result table for day T+1, and the offline data result table for day T+1. Through the above steps, while the data updated on the current day is calculated and updated in real-time by real-time streaming task 1, the data to be updated on day T+1 is calculated by real-time streaming task 2 and written into the data result table for day T+1. Since the aggregation results to be updated on day T+1 are data that has changed on day T, while the data that has not changed on day T has been pre-calculated by batch task 2, the correct and complete aggregation results can always be found when switching at midnight on day T+1.

[0024] In some embodiments of this application, the historical detailed data aggregation results up to 00:00 on T-2 day and the historical detailed data aggregation results up to 00:00 on T-1 day are obtained daily by a scheduled task and written into the corresponding result table, which is distinguished by the execution date. In this embodiment, the data processing flow is automatically executed daily by a scheduled task, specifically including: obtaining historical detailed data up to 00:00 on T-2 day and calculating its corresponding first aggregation result; simultaneously, obtaining historical detailed data up to 00:00 on T-1 day and calculating its corresponding second aggregation result. The scheduled task is usually triggered during the low-load period in the early morning of each day to ensure efficient utilization of computing resources and system stability. After the calculation is completed, the first aggregation result and the second aggregation result are written into their respective result tables. To facilitate data management and historical traceability, the result table adopts a design strategy that distinguishes by execution date, for example, by including a date suffix in the table name or using a date field as the partition key. This scheme effectively ensures the timeliness, consistency, and maintainability of the aggregated data, providing stable and reliable data support for subsequent data analysis, report generation, or business decisions.

[0025] In some embodiments of this application, the method further includes: copying the result table to a cache or main memory before the batch processing task begins; and removing the result table from the cache or main memory after the batch processing task ends. The invention also includes a step of dynamically managing the result table's cache to optimize the execution performance of the batch processing task. Specifically, before the batch processing task starts, the system automatically copies the result table completely from a persistent storage database (such as MySQL or HDFS) to a cache or server main memory. This cache is preferably a distributed memory storage system (such as Redis or Ignite) that supports low-latency, high-concurrency data access; alternatively, the result table can be directly loaded into the local main memory of the task execution node, forming a temporary memory table. This copying process is implemented through a dedicated data synchronization tool or cache loading component, ensuring that the cached data and the source result table are consistent in structure and content. By pre-positioning the result table on a high-speed storage medium, the batch processing task can directly read the cached data when executing computational logic, avoiding network transmission overhead and disk I / O bottlenecks caused by frequent access to remote databases, thereby significantly shortening the overall task execution time.

[0026] During the batch task execution phase, the cached result table provides a fast data source for operations such as aggregation calculations, data transformation, or correlation analysis. For example, when generating daily statistical reports or performing data quality checks, the task workflow can query the cached table in real time to obtain historical aggregation results, significantly improving processing efficiency. Simultaneously, the system monitors the cache status to ensure data availability and consistency throughout task execution.

[0027] After the batch processing task is completed, the system automatically removes the result table from the cache or main memory according to a preset cleanup strategy. The removal operation can be achieved by calling the cache eviction interface or releasing memory objects, thereby promptly reclaiming storage resources and preventing memory leaks or cache bloat. Optionally, the system can also record cache usage logs for subsequent performance analysis and optimization.

[0028] This implementation effectively balances data processing speed and system resource consumption by introducing a cache lifecycle management mechanism, making it particularly suitable for large-scale, high-frequency batch processing scenarios and enhancing the practicality and scalability of the method.

[0029] In some embodiments of this application, the method further includes: displaying the real-time query results to the user through a microservice, wherein the configuration parameters of the microservice are determined based on the data template or user profile corresponding to the initiator of the real-time query task. This embodiment, after obtaining the real-time query results, also includes a key step: displaying the real-time query results to the user through an independent microservice. The specific display logic and configuration parameters of this microservice are not fixed but dynamically determined based on the specific characteristics of the initiator of the real-time query task. Specifically, the determination of the configuration parameters is based on two aspects: first, the specific data template used by the initiator when initiating this query, which is usually associated with specific business scenarios and data structure requirements; second, the user profile of the initiator, which includes personalized information such as the user's historical behavioral preferences, role permissions, and display terminal type. The system integrates information from these two sources to deduce and match the most suitable microservice configuration parameters for this query task and the current user in real time, such as the type of chart to be displayed, data granularity, refresh frequency, interaction method, and visual theme.

[0030] In some embodiments of this application, the method further includes: acquiring the execution process log of the real-time query task; verifying the execution results of the decomposed first offline batch task, second offline batch task, first real-time streaming task, and second real-time streaming task based on the execution process log; and setting a corresponding confidence level for the real-time query result based on the verification result. In this embodiment, firstly, the system acquires the complete execution process log of the real-time query task in the hybrid execution engine through a background log acquisition module. This log records in detail the execution status, key indicators (such as data throughput, processing latency, and error codes) of the first offline batch task, second offline batch task, first real-time streaming task, and second real-time streaming task generated after task decomposition, as well as a summary of the final output result. Then, the system calls a dedicated verification and analysis module to cross-verify the execution results of the above four sub-tasks based on the acquired execution process log. For example, the verification module can compare the historical statistical baseline calculated by the offline batch task with the real-time data indicators generated by the real-time streaming task to check their logical consistency and numerical rationality. At the same time, it also analyzes whether the execution of each sub-task is successful, whether it is completed within the expected time window, and whether its data processing logic conforms to the preset business rules. Finally, based on the verification results output by the verification and analysis module, the system assigns a quantified confidence level to the final real-time query results. This confidence level is a comprehensive evaluation value, such as a percentage score or a "high / medium / low" rating. The more ideal the verification results (e.g., all subtasks execute successfully and data consistency is high), the higher the assigned confidence level; conversely, if verification reveals data anomalies, task failures, or logical conflicts, the confidence level decreases accordingly. This confidence level, along with the real-time query results, will be displayed to the user through the aforementioned microservices, providing crucial evidence for the user to judge the reliability of the results, thereby supporting more reliable decision-making.

[0031] Figure 2 The illustration schematically shows a principle diagram of a batch data cross-day zero-delay handover method according to an embodiment of this application. For example... Figure 2As shown in the diagram, the cylinders represent data tables, and the boxes represent calculation tasks. Table T2_T refers to the aggregated result data with data up to 0:00 on day T-2 and calculation time at 0:00 on day T. This data is written by the batch task T2_TOMORROW(T-1). Table T2_T+1 refers to the aggregated result data relative to day T+1, with data up to 0:00 on day T-1 and calculation time at day T+1. This data is written by the batch task T2_TOMORROW(T). Table RT_T refers to the aggregated result data calculated in real-time from day T-2 to the present, with calculation time at the current time. This data is written by the real-time calculation task AGG, which reads the detailed data from 0:00 on day T-2 to the present and merges it with the aggregated result data of T2_T. Table RT_T_1 refers to the aggregated result data calculated in real-time from day T-2 to the present, with calculation time at 0:00 on day T+1. This data is written by the real-time calculation task AGG, which reads the detailed data from 0:00 on day T-1 to the present and merges it with the aggregated result data of T2_T. When querying data, retrieve the latest data from the real-time data result table for day T, the real-time data result table for day T+1, and the offline data result table for day T+1, and return the data with the latest data update time.

[0032] Based on the same inventive concept, this application also provides a batch data cross-day zero-delay switching device. Figure 3 A schematic diagram illustrating the structure of a batch data cross-day zero-delay switching device according to an embodiment of this application is shown. Figure 3 As shown, the device includes: a task acquisition module for acquiring real-time query tasks; a task decomposition module for decomposing the real-time query task into several offline batch tasks and several real-time streaming tasks when there are currently unfinished batch tasks, each task being used to obtain a corresponding aggregation result; and a result return module for obtaining real-time query results based on the obtained aggregation results, the real-time query results serving as a response to the real-time query task.

[0033] In some optional implementations, the real-time query task is decomposed into several offline batch tasks and several real-time streaming tasks, including: decomposing the real-time query task into a first offline batch task, a second offline batch task, a first real-time streaming task, and a second real-time streaming task; the first offline batch task is used to obtain historical detailed data up to 0:00 on T-2 day and calculate the aggregation result as the first aggregation result; the second offline batch task is used to obtain historical detailed data up to 0:00 on T-1 day and calculate the aggregation result as the second aggregation result; the first real-time streaming task is used to obtain detailed data from 0:00 on T-2 day to the current time and the first aggregation result, and after performing aggregation operation according to the current time, obtain a third aggregation result; the second real-time streaming task is used to obtain detailed data from 0:00 on T-1 day to the current time and the first aggregation result, and after performing aggregation operation according to 0:00 on T+1 day, obtain a fourth aggregation result.

[0034] In some optional implementations, obtaining real-time query results based on the obtained aggregation results includes: returning the latest data with the latest data update time from the second aggregation result, the third aggregation result, and the fourth aggregation result; and using the returned data as the real-time query result.

[0035] In some optional implementations, the historical detailed data calculation and aggregation results up to 0:00 on T-2 day and the historical detailed data calculation and aggregation results up to 0:00 on T-1 day are obtained by executing a scheduled task daily and written into the corresponding result table, which is distinguished by the execution date.

[0036] In some alternative implementations, the apparatus further includes: copying the result table to a cache or main memory before the batch task begins; and removing the result table from the cache or main memory after the batch task ends.

[0037] In some optional embodiments, the apparatus further includes: displaying the real-time query results to the user through a microservice, wherein the configuration parameters of the microservice are determined based on the data template or user profile corresponding to the initiator of the real-time query task.

[0038] In some optional embodiments, the apparatus further includes: acquiring the execution process log of the real-time query task, verifying the execution results of the decomposed first offline batch task, second offline batch task, first real-time streaming task and second real-time streaming task based on the execution process log, and setting a corresponding confidence level for the real-time query result according to the verification result.

[0039] The specific limitations of each functional module in the aforementioned batch data cross-day zero-delay handover device can be found in the limitations of the batch data cross-day zero-delay handover method described above, and will not be repeated here. Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module. This also solves the problem of data inaccuracy during the zero-point handover of batch data, and avoids the high resource consumption problem caused by traditional real-time computing to solve this problem.

[0040] In some embodiments of this application, an electronic device is also provided, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which executes the aforementioned batch data cross-day zero-latency switching method. Its internal structure diagram can be shown as follows. Figure 4 As shown. Figure 4 This schematic diagram illustrates the internal structure of an electronic device according to an embodiment of this application. The electronic device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a batch data cross-day zero-latency switching method.

[0041] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0042] In one embodiment provided in this application, a machine-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, cause the processor to be configured to perform the aforementioned batch data cross-day zero-latency switching method.

[0043] In one embodiment provided in this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned batch data cross-day zero-latency switching method.

[0044] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0045] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0047] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0049] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0050] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0051] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0052] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for zero-latency switching of batch data across days, characterized in that, The method includes: Get real-time query tasks; If there are currently unfinished batch processing tasks, the real-time query task is decomposed into several offline batch tasks and several real-time streaming tasks, with each task used to obtain the corresponding aggregation result. The real-time query results are obtained based on the aggregated results, and these real-time query results serve as a response to the real-time query task.

2. The method according to claim 1, characterized in that, The real-time query task is decomposed into several offline batch tasks and several real-time streaming tasks, including: The real-time query task is decomposed into a first offline batch task, a second offline batch task, a first real-time streaming task, and a second real-time streaming task. The first offline batch task is used to obtain historical detailed data up to 0:00 on day T-2 and calculate the aggregation result as the first aggregation result; The second offline batch task is used to obtain historical detailed data up to 0:00 on day T-1 and calculate the aggregation result as the second aggregation result; The first real-time streaming task is used to obtain detailed data from 0:00 on day T-2 to the current time and the first aggregation result. After performing aggregation operation according to the current time, the third aggregation result is obtained. The second real-time streaming task is used to obtain detailed data from 0:00 on day T-1 to the current time and the first aggregation result. After performing aggregation calculations according to 0:00 on day T+1, the fourth aggregation result is obtained.

3. The method according to claim 2, characterized in that, The real-time query results are obtained based on the aggregated results, including: Return the latest data from the second, third, and fourth aggregation results; The returned data is used as the real-time query result.

4. The method according to claim 2, characterized in that, The historical detailed data calculation and aggregation results up to 0:00 on T-2 day and the historical detailed data calculation and aggregation results up to 0:00 on T-1 day are obtained by executing a scheduled task daily and written into the corresponding result table, which is distinguished by the execution date.

5. The method according to claim 4, characterized in that, The method further includes: copying the result table to a cache or main memory before the batch processing task begins; and removing the result table from the cache or main memory after the batch processing task ends.

6. The method according to claim 1, characterized in that, The method further includes: displaying the real-time query results to the user through microservices, wherein the configuration parameters of the microservices are determined based on the data template or user profile corresponding to the initiator of the real-time query task.

7. The method according to claim 2, characterized in that, The method further includes: obtaining the execution process log of the real-time query task, verifying the execution results of the decomposed first offline batch task, second offline batch task, first real-time streaming task and second real-time streaming task based on the execution process log, and setting the corresponding confidence level for the real-time query result according to the verification result.

8. A batch data cross-day zero-delay switching device, characterized in that, The device includes: The task acquisition module is used to acquire real-time query tasks; The task decomposition module is used to decompose the real-time query task into several offline batch tasks and several real-time streaming tasks when there are currently unfinished batch tasks. Each task is used to obtain the corresponding aggregation result. The result return module is used to obtain real-time query results based on the obtained aggregation results, and the real-time query results serve as a response to the real-time query task.

9. An electronic device, characterized in that, include: At least one processor; A memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the batch data cross-day zero-latency switching method according to any one of claims 1 to 7 by executing the instructions stored in the memory.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the batch data cross-day zero-delay switching method as described in any one of claims 1 to 7.

Images