Real-time report generation method and system based on flink

Abstract

The application belongs to the technical field of big data real-time computing, and relates to a real-time report generation method based on Flink, which comprises the following steps: when a real-time data stream generated by a Flink job is generated, the real-time data stream is transmitted to a Kafka business system; the real-time data stream received by the Kafka business system is written into a disk structure of the Kafka business system; real-time data stream in the disk structure is extracted to a temporary intermediate layer for conversion processing through ETL to generate real-time list data; the real-time list data is sent to a Druid database; when a data query operation is detected, the type of the data query operation is identified, corresponding operation instructions are generated according to the determined type of the data query operation, and real-time reports are generated by extracting a Druid engine in the Druid database according to the operation instructions; and the real-time reports are pushed to a front-end platform. The application also provides a real-time report generation system based on Flink, a computer device and a storage medium thereof.

Description

Technical Field

[0001] This application relates to the field of real-time big data computing technology, and in particular to a real-time report generation method, system, computer device and storage medium based on Flink. Background Technology

[0002] Traditional reporting projects combine reporting tools, data warehouses, and ETL (data warehouse technology). However, this approach suffers from long data generation times. Reading data directly from the production system puts immense pressure on the production database, creating performance bottlenecks and directly impacting business operations. As customers increasingly prioritize the real-time nature of source data, the timeliness of real-time reports is becoming ever more crucial. Summary of the Invention

[0003] The purpose of this application is to propose a real-time report generation method, system, computer device and storage medium based on Flink, so as to solve the technical problem of long generation time for real-time reports.

[0004] To address the aforementioned technical problems, this application provides a real-time report generation method based on Flink, employing the following technical solution: The method includes the following steps:

[0005] When a Flink job generates a real-time data stream, the real-time data stream is transmitted to the Kafka business system;

[0006] Write the real-time data stream received by the Kafka business system into the disk structure of the Kafka business system;

[0007] The real-time data stream from the disk structure is extracted by ETL and transformed in a temporary intermediate layer to generate real-time inventory data.

[0008] Send real-time inventory data to the Druid database;

[0009] When a data query operation is detected, the type of the data query operation is identified, and a corresponding operation instruction is generated based on the determined type of the data query operation. Based on the operation instruction, the Druid engine in the Druid database is extracted to generate a real-time report.

[0010] Push real-time reports to the front-end platform.

[0011] Furthermore, after the step of extracting the real-time data stream from the disk structure via ETL to a temporary intermediate layer for transformation processing and generating real-time inventory data, the method further includes:

[0012] Real-time inventory data is transmitted to a Hive data warehouse for storage and backup via Flume data pipeline;

[0013] When a Flink job fails, the Hive tool is used to query the real-time inventory data backed up in the Hive data warehouse and find that it differs from the real-time inventory data generated by the Kafka business system, and then the target inventory data is obtained.

[0014] Replenish the target list data to the Druid database.

[0015] Furthermore, the step of transmitting the real-time inventory data to the Flume system for backup includes:

[0016] Receive real-time inventory data and transmit the received real-time inventory data to one or more paving guides using Flume;

[0017] Store the real-time inventory data values ​​transmitted by Flume in the Hive data warehouse.

[0018] Furthermore, the step of extracting real-time data streams from the disk structure via ETL, converting them in a temporary intermediate layer, and generating real-time inventory data includes:

[0019] The real-time data stream received by the Kafka business system is extracted to a temporary intermediate layer through ETL. After cleaning, transforming and integrating the real-time data in the temporary intermediate layer, real-time manifest data in a standard format is generated.

[0020] Transmit real-time inventory data to the Kafka business system.

[0021] Furthermore, the step of pushing real-time reports to the front-end platform includes:

[0022] Configure the indicator columns, time columns, and dimension columns for real-time reports;

[0023] Real-time reports are pushed to the front-end platform as a logical set of indicator columns, time columns, or dimension columns.

[0024] Furthermore, after the step of pushing real-time reports to the front-end platform, the method further includes:

[0025] When the preset identifier of the front-end platform generates dynamic actions such as dragging, pulling, or dropping, the indicator column, time column, or dimension column of the real-time report is obtained.

[0026] Furthermore, after the step of pushing real-time reports to the front-end platform, the method further includes:

[0027] When the real-time content generated by the front-end platform based on the real-time report does not match the target content corresponding to the data query operation, the pre-stored list data stored in the Hive data warehouse is obtained through the Hive tool. The pre-stored list data is compared with the real-time list data obtained from the Druid database to locate the target fault location in the Druid database.

[0028] Extract the target fault data at the target fault location in the Druid database, and the target list data at the corresponding location in the Hive data warehouse;

[0029] Replace the target list data with the target fault data and populate it back into the target fault location in the Druid database;

[0030] When the front-end platform generates real-time content based on real-time reports that matches the target content of the data query operation, it updates the pre-stored list data in the Hive data warehouse.

[0031] To address the aforementioned technical problems, this application also provides a real-time report generation system based on Flink, the system comprising:

[0032] The transmission module is used to transmit real-time data streams generated by Flink jobs to the Kafka business system.

[0033] The processing module is used to write the real-time data stream received by the Kafka business system into the disk structure of the Kafka business system; extract the real-time data stream from the disk structure through ETL to a temporary intermediate layer for transformation and processing, and generate real-time manifest data.

[0034] The sending module is used to send real-time inventory data to the Druid database;

[0035] The extraction module is used to identify the type of data query operation when a data query operation is detected, generate corresponding operation instructions based on the identified data query operation type, and extract the Druid engine from the Druid database to generate real-time reports based on the operation instructions.

[0036] The push module is used to push real-time reports to the front-end platform.

[0037] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution: it includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the Flink-based real-time report generation method described above.

[0038] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the following technical solution: the computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the Flink-based real-time report generation method as described above.

[0039] Compared with existing technologies, the embodiments of this application have the following main advantages: When a Flink job generates a real-time data stream, it transmits the real-time data stream to the Kafka business system; the real-time data stream received by the Kafka business system is written into the disk structure of the Kafka business system; the real-time data stream in the disk structure is extracted through ETL and processed in a temporary intermediate layer to generate real-time manifest data; the real-time manifest data is sent to the Druid database; when a data query operation is detected, the type of the data query operation is identified, and a corresponding operation instruction is generated based on the determined type of data query operation; the Druid engine in the Druid database is extracted based on the operation instruction to generate a real-time report; and the real-time report is pushed to the front-end platform. This achieves end-to-end real-time report generation, improves data fault tolerance, accelerates the generation speed of real-time reports from a data-wide perspective, improves the timeliness of real-time reports, and increases data accuracy. Attached Figure Description

[0040] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0042] Figure 2 This is a flowchart of an embodiment of a real-time report generation method based on Flink;

[0043] Figure 3 This is a schematic diagram of the structure of an embodiment of a real-time report generation system based on Flink;

[0044] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0046] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0048] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0049] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0050] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0051] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0052] It should be noted that the real-time report generation method based on Flink provided in this application embodiment is generally executed by the server, and correspondingly, the real-time report generation system based on Flink is generally set up in the server.

[0053] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0054] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the Flink-based real-time report generation method according to this application. The Flink-based real-time report generation method includes the following steps:

[0055] Step S201: When the Flink job generates a real-time data stream, transmit the real-time data stream to the Kafka business system.

[0056] It's important to note that the core of Flink (an open-source stream processing framework) is a distributed streaming data engine written primarily in Java and Scala. Flink executes arbitrary streaming data programs using distributed data parallelism and pipelined methods. In this streaming data processing system, both data and computation are stored locally, such as in memory or on local disk.

[0057] In this embodiment, Flink's state is based on locality, which is mainly reflected in the fact that each operator subtask maintains its own state storage. The states of operator subtasks cannot be accessed by each other. For example, when the number of parallel instances of an operator or the number of subtasks of an operator changes, the application needs to shut down or start a new target operator subtask and update the state data on the original operator subtask to the target operator subtask.

[0058] In Flink, state can be understood as a variable on the current instance of an operator's subtasks. This variable records the results generated by the historical data flowing through the current operator. Specifically, when new data records flow in, we need to combine the current result (i.e., the Flink state) for computation. In fact, Flink's state is created and managed by operator subtasks. An operator subtask receives the input stream, obtains the corresponding state, and updates the state based on the new computation result. For example, when summing an integer field flowing in within a time window, when the operator subtask receives a new element, it obtains the value already stored in the state (i.e., the summation result of the historical data), adds the current value as input to the Flink state, and updates the Flink state data.

[0059] In this embodiment, Flink's processing of real-time data streams is divided into receiving data, processing data, and outputting processing results. Receiving data specifically involves receiving one or more data sources, such as HDFS and Kafka. Processing data specifically involves executing several transformation operators required by the user. Outputting processing results specifically involves outputting the transformed results to the Kafka business system.

[0060] Step S202: Write the real-time data stream received by the Kafka business system into the disk structure of the Kafka business system;

[0061] Kafka (a high-throughput distributed publish-subscribe messaging system) is a storage system that primarily separates the delivery of published messages to message queues and consumption, storing messages to be published first. Real-time data streams written to Kafka are written to disk and replicated to Druid (a data query system) to ensure fault tolerance and allow waiting for message acknowledgments until the message is completely written. In this embodiment, the disk structure of Kafka is mainly controlled by Druid (the data query system) to determine where to read the real-time data stream, unlike existing technologies where the client controls the reading location. Kafka is a special-purpose distributed file system for commit log storage, replication, and propagation. Kafka's stream processing continuously acquires real-time data streams from Flink, processes them through data ETL using business logic, and then writes them to the output Druid (the data query system).

[0062] Step S203: Extract the real-time data stream from the disk structure through ETL to a temporary intermediate layer for transformation and processing, and generate real-time inventory data;

[0063] This embodiment uses ETL (Data Warehouse Technology) for data processing, building complex applications that aggregate computations or connect streams together. This helps handle unordered data, reprocess code changes, and perform Flink state computations. Business logic is described by a combination of modular components and basic call processing modules, outlining the process for each business characteristic.

[0064] It should be noted that ETL extracts the real-time data stream to a temporary intermediate layer for cleaning, transformation, and integration, before finally loading it into the Kafka business system to complete online analytical processing and data mining. Data ETL processing transforms the real-time data stream into a standard format.

[0065] The system writes the real-time data stream received by the Kafka business system into the disk structure of the Kafka business system; after extracting the real-time data stream from the disk structure through ETL to a temporary intermediate layer for transformation and processing, and generating real-time manifest data, the real-time manifest data is also transmitted to the Hive (data warehouse tool) data warehouse for storage and backup through the Flume (log collection system) data pipeline.

[0066] Within Flume, one or more agents are configured. Each agent has an independent daemon process that retrieves data from the receiving end or other agents and transmits it to the gateway node or agent of the next node. The received data is passed to one or more bootstrap channels in Flume's event format. Flume provides various data reception methods, including data serialization and remote procedure call frameworks. In this embodiment, Flume transmits the data to the next node, the Hive data warehouse.

[0067] In this embodiment, the Hive data warehouse is used to store, query, and analyze large-scale data stored in Hadoop (a distributed system infrastructure). The Hive data warehouse tool maps structured data files to a database table and provides SQL (database language) query functionality, transforming SQL statements into MapReduce (programming model) tasks for execution. The SQL query function allows for querying and analyzing the required content.

[0068] Specifically, in this embodiment, when a Flink job fails, the Hive tool queries the real-time manifest data backed up in the Hive data warehouse to find differences between the real-time manifest data and the data generated by the Kafka business system. The different real-time manifest data is extracted and set as the target manifest data, which is then fed back to the Druid database. The Hive data warehouse tool maps structured data files to a database table and provides complete SQL query functionality. Therefore, when a failure occurs, the Hive data warehouse tool quickly queries the required content from the Hive data warehouse and promptly feeds the necessary content data back to the Druid database; that is, the required content is the data at the point of failure.

[0069] Step S204: Send real-time inventory data to the Druid database;

[0070] The Druid database is primarily used for aggregating and querying large amounts of time-series data. When real-time inventory data is fed into the Druid database, it becomes immediately searchable without changing its original state, ensuring the integrity of the real-time inventory data. Typically, the Druid database is structured based on time-series inventory data; once inventory data is generated and entered into Druid, external systems can then query that data.

[0071] Step S205: When a data query operation is detected, identify the type of the data query operation, generate the corresponding operation instruction according to the determined type of the data query operation, and extract the Druid engine from the Druid database to generate a real-time report according to the operation instruction.

[0072] Extracting the Druid engine from the Druid database offers excellent timeliness, delivering user results within seconds. Processed real-time inventory data is then integrated into the Druid engine for downstream users to query reports, lowering the barrier to entry for users and increasing report query efficiency. In this embodiment, data query operations include dynamic actions performed by the user on the display screen, such as dragging, pulling, or dropping.

[0073] Step S206: Push the real-time report to the front-end platform.

[0074] Specifically, the Druid database serves not only as a distributed data analysis platform but also as a time-series database. Its data structure consists of time columns, dimension columns, and metric columns. The time column identifies the time value of each row of data; the dimension columns identify the various categories of data rows; and the metric columns are used for aggregation and calculation. When the Druid engine generates real-time reports, the metric, time, and dimension columns of the real-time report are set, and the report is pushed to the front-end platform as a logical set of either metric, time, or dimension columns.

[0075] When the preset identifiers on the front-end platform undergo dynamic actions such as dragging, pulling, or dropping, the indicator, time, or dimension columns change. This altered and recombined report becomes a real-time report, and the indicator, time, or dimension columns of the real-time report can then be retrieved. While the real-time report can be obtained simply by the preset identifiers on the front-end platform undergoing such dynamic actions, it can also be obtained by customizing the indicator, time, or dimension columns.

[0076] When the real-time content generated by the front-end platform based on real-time reports does not match the target content corresponding to the data query operation, the pre-stored inventory data stored in the Hive data warehouse is retrieved using Hive tools. This pre-stored inventory data is compared with the real-time inventory data retrieved from the Druid database to locate the target fault location in the Druid database. The target fault data at the target fault location in the Druid database is extracted, along with the target inventory data at the corresponding location in the Hive data warehouse. The target inventory data is then replaced with the target fault data and updated back to the target fault location in the Druid database. When the real-time content generated by the front-end platform based on real-time reports matches the target content corresponding to the data query operation, the pre-stored inventory data stored in the Hive data warehouse is updated.

[0077] This embodiment transmits real-time data streams generated by Flink jobs to the Kafka business system; writes the received real-time data streams into the Kafka business system's disk structure; extracts the real-time data streams from the disk structure using ETL, processes them in a temporary intermediate layer, and generates real-time manifest data; sends the real-time manifest data to the Druid database; when a data query operation is detected, it identifies the type of the data query operation, generates corresponding operation instructions based on the identified type, extracts the Druid engine from the Druid database based on the operation instructions to generate real-time reports, and pushes the real-time reports to the front-end platform. This achieves end-to-end real-time report generation, improves data fault tolerance, accelerates real-time report generation from a holistic data perspective, enhances the timeliness of real-time reports, and increases data accuracy.

[0078] It should be emphasized that, to further ensure the privacy and security of the aforementioned real-time reports, these reports can also be stored in a blockchain node.

[0079] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0080] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0081] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0083] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0084] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a real-time report generation system based on Flink. This system embodiment is similar to... Figure 2 The method embodiments shown correspond to those described.

[0085] like Figure 3 As shown, the real-time report generation system 300 based on Flink described in this embodiment includes: a transmission module 301, a processing module 302, a sending module 303, an extraction module 304, and a push module 305. Wherein:

[0086] Transmission module 301 is used to transmit real-time data streams generated by Flink jobs to the Kafka business system.

[0087] The processing module 302 is used to write the real-time data stream received by the Kafka business system into the disk structure of the Kafka business system; extract the real-time data stream from the disk structure through ETL to a temporary intermediate layer for transformation and processing, and generate real-time inventory data.

[0088] Sending module 303 is used to send real-time inventory data to the Druid database;

[0089] Extraction module 304 is used to identify the type of data query operation when a data query operation is detected, generate corresponding operation instructions based on the determined type of data query operation, and extract the Druid engine from the Druid database to generate real-time reports based on the operation instructions.

[0090] The push module 305 is used to push real-time reports to the front-end platform.

[0091] This embodiment transmits real-time data streams generated by Flink jobs to the Kafka business system; writes the received real-time data streams into the Kafka business system's disk structure; extracts the real-time data streams from the disk structure using ETL, processes them in a temporary intermediate layer, and generates real-time manifest data; sends the real-time manifest data to the Druid database; when a data query operation is detected, it identifies the type of the data query operation, generates corresponding operation instructions based on the identified type, extracts the Druid engine from the Druid database based on the operation instructions to generate real-time reports, and pushes the real-time reports to the front-end platform. This achieves end-to-end real-time report generation, improves data fault tolerance, accelerates real-time report generation from a holistic data perspective, enhances the timeliness of real-time reports, and increases data accuracy.

[0092] In some optional implementations of this embodiment, the system 300 further includes:

[0093] The backup module is used to transfer real-time inventory data to the Hive data warehouse for storage and backup via Flume data pipeline;

[0094] The query module is used to query the real-time inventory data backed up in the Hive data warehouse and the real-time inventory data generated by the Kafka business system when a Flink job fails, and to obtain the target inventory data.

[0095] The replenishment module is used to replenish the target list data into the Druid database.

[0096] In some optional implementations of this embodiment, the backup module includes:

[0097] The receiving unit is used to receive real-time inventory data and transmit the received real-time inventory data to one or more paving guides via Flume.

[0098] Storage unit, used to store real-time manifest data values ​​after Flume transmission to the Hive data warehouse.

[0099] In some optional implementations of this embodiment, the processing module 302 includes:

[0100] The processing unit is used to extract the real-time data stream received by the Kafka business system to a temporary intermediate layer through ETL, and to clean, transform and integrate the real-time data in the temporary intermediate layer to generate real-time manifest data in a standard format.

[0101] The transmission unit is used to transmit real-time inventory data to the Kafka business system.

[0102] In some optional implementations of this embodiment, the push module 305 includes:

[0103] The settings unit is used to configure the indicator columns, time columns, and dimension columns of real-time reports;

[0104] The push unit is used to push real-time reports to the front-end platform as a logical set of indicator columns, time columns, or dimension columns.

[0105] In some optional implementations of this embodiment, the system 300 further includes:

[0106] The operation module is used to obtain the indicator column, time column, or dimension column of the real-time report when the preset identifier of the current platform generates dynamic actions such as dragging, pulling, or dragging.

[0107] In some optional implementations of this embodiment, the system 300 further includes:

[0108] The fault location module is used to locate the target fault location in the Druid database when the real-time content generated by the front-end platform based on the real-time report does not match the target content corresponding to the data query operation. It obtains the pre-stored list data stored in the Hive data warehouse through the Hive tool, compares the pre-stored list data with the real-time list data obtained from the Druid database, and locates the target fault location in the Druid database.

[0109] The data overlay module is used to extract the target fault data at the target fault location in the Druid database and the target list data at the corresponding location in the Hive data warehouse, replace the target list data with the target fault data, and then backfill it to the target fault location in the Druid database.

[0110] The update module is used to update the pre-stored list data in the Hive data warehouse when the real-time content generated by the front-end platform matches the target content corresponding to the data query operation.

[0111] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed] for details. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0112] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that... Figure 4 Only a computer device 4 with components 41-43 is shown in the illustration. However, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0113] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0114] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for a real-time report generation method based on Flink. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0115] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the Flink-based real-time report generation method.

[0116] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0117] In this embodiment, when a Flink job generates a real-time data stream, it is transmitted to the Kafka business system. The real-time data stream received by the Kafka business system is written into the disk structure of the Kafka business system. The real-time data stream in the disk structure is extracted through ETL and processed in a temporary intermediate layer to generate real-time manifest data. The real-time manifest data is sent to the Druid database. When a data query operation is detected, the type of data query operation is identified, and a corresponding operation instruction is generated based on the determined data query operation type. Based on the operation instruction, the Druid engine in the Druid database is extracted to generate a real-time report. The real-time report is then pushed to the front-end platform. This achieves end-to-end real-time report generation, improves data fault tolerance, accelerates the generation speed of real-time reports from a data-wide perspective, improves the timeliness of real-time reports, and increases data accuracy.

[0118] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the Flink-based real-time report generation method described above.

[0119] In this embodiment, when a Flink job generates a real-time data stream, it transmits the real-time data stream to the Kafka business system; the real-time data stream received by the Kafka business system is written into the disk structure of the Kafka business system; the real-time data stream in the disk structure is extracted through ETL and processed in a temporary intermediate layer to generate real-time manifest data; the real-time manifest data is sent to the Druid database; when a data query operation is detected, the type of data query operation is identified, and a corresponding operation instruction is generated based on the determined data query operation type. Based on the operation instruction, the Druid engine in the Druid database is extracted to generate a real-time report; the real-time report is pushed to the front-end platform. This achieves end-to-end real-time report generation, improves data fault tolerance, accelerates the generation speed of real-time reports from a data-wide perspective, improves the timeliness of real-time reports, and increases data accuracy.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0121] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A real-time report generation method based on Flink, characterized in that, The method includes the following steps: When a Flink job generates a real-time data stream, the real-time data stream is transmitted to the Kafka business system; Write the real-time data stream received by the Kafka business system into the disk structure of the Kafka business system; The real-time data stream from the disk structure is extracted by ETL and transformed in a temporary intermediate layer to generate real-time inventory data. Send real-time inventory data to the Druid database; When a data query operation is detected, the type of the data query operation is identified, and a corresponding operation instruction is generated based on the determined type of the data query operation. Based on the operation instruction, the Druid engine in the Druid database is extracted to generate a real-time report. Push real-time reports to the front-end platform; After the step of extracting real-time data streams from the disk structure via ETL, converting them to a temporary intermediate layer, and generating real-time inventory data, the method further includes: Real-time inventory data is transmitted to a Hive data warehouse for storage and backup via Flume data pipelines. When a Flink job fails, the Hive tool is used to query the real-time inventory data backed up in the Hive data warehouse and find that it differs from the real-time inventory data generated by the Kafka business system, and then the target inventory data is obtained. Replenish the target list data to the Druid database; The steps of transferring real-time inventory data to a Hive data warehouse for storage and backup via Flume data pipeline include: Receive real-time inventory data and transmit the received real-time inventory data to one or more paving guides using Flume; Store the real-time inventory data values ​​transmitted by Flume in the Hive data warehouse.

2. The real-time report generation method based on Flink according to claim 1, characterized in that, The steps of extracting real-time data streams from the disk structure via ETL, transforming them in a temporary intermediate layer, and generating real-time inventory data include: The real-time data stream received by the Kafka business system is extracted to a temporary intermediate layer through ETL. After cleaning, transforming and integrating the real-time data in the temporary intermediate layer, real-time manifest data in a standard format is generated. Transmit real-time inventory data to the Kafka business system.

3. The real-time report generation method based on Flink according to claim 1, characterized in that, The steps for pushing real-time reports to the front-end platform include: Configure the indicator columns, time columns, and dimension columns for real-time reports; Real-time reports are pushed to the front-end platform as a logical set of indicator columns, time columns, or dimension columns.

4. The real-time report generation method based on Flink according to claim 3, characterized in that, After the step of pushing real-time reports to the front-end platform, the method further includes: When the preset identifier of the front-end platform generates dynamic actions such as dragging, pulling, or dropping, the indicator column, time column, or dimension column of the real-time report is obtained.

5. The real-time report generation method based on Flink according to claim 1, characterized in that, After the step of pushing real-time reports to the front-end platform, the method further includes: When the real-time content generated by the front-end platform based on the real-time report does not match the target content corresponding to the data query operation, the pre-stored list data stored in the Hive data warehouse is obtained through the Hive tool. The pre-stored list data is compared with the real-time list data obtained from the Druid database to locate the target fault location in the Druid database. Extract the target fault data at the target fault location in the Druid database, and the target list data at the corresponding location in the Hive data warehouse; Replace the target list data with the target fault data and populate it back into the target fault location in the Druid database; When the front-end platform generates real-time content based on real-time reports that matches the target content of the data query operation, it updates the pre-stored list data in the Hive data warehouse.

6. A real-time report generation system based on Flink, the system being used to implement the real-time report generation method based on Flink as described in any one of claims 1 to 5, characterized in that, The system includes: The transmission module is used to transmit real-time data streams generated by Flink jobs to the Kafka business system. The processing module is used to write the real-time data stream received by the Kafka business system into the disk structure of the Kafka business system, extract the real-time data stream from the disk structure through ETL to a temporary intermediate layer for transformation and processing, and generate real-time manifest data. The sending module is used to send real-time inventory data to the Druid database; The extraction module is used to identify the type of data query operation when a data query operation is detected, generate corresponding operation instructions based on the identified data query operation type, and extract the Druid engine from the Druid database to generate real-time reports based on the operation instructions. The push module is used to push real-time reports to the front-end platform.

7. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the Flink-based real-time report generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the Flink-based real-time report generation method as described in any one of claims 1 to 5.

Images