A real-time data flow statistics method and device

Abstract

The present invention discloses a method and device for real-time data stream statistics, and relates to the field of computer technology. A specific implementation of the method includes: registering a virtual table of the data stream according to the read data stream; loading and running a preset SQL statement file on the virtual table of the data stream; and storing the real-time statistical results obtained after the running. This implementation overcomes the technical problem of complex processing when the data stream computing framework processes multiple data streams based on various API operators, thereby achieving the technical effect of replacing complex conversion calculation functions by writing SQL statements to achieve statistics of real-time data streams in business scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a statistical method and device for real-time data stream. Background Art

[0002] At present, the real-time processing technology of big data is developing rapidly, and it is becoming more and more common to apply real-time processing engines to production environments with rich user data. This not only adapts to the trend of refined business operations, but also helps businesses improve the accuracy of decision-making. In business processing, it is often necessary to connect multiple data streams to the stream data computing framework, filter the data streams, convert fields, aggregate statistics of different dimensions, and output statistical results according to different business logic scenarios. In order to ensure that data is not lost or duplicated during the real-time statistics process, the checkpoint mechanism of the stream data computing framework itself can be restored from the last checkpoint after a failure and restart, ensuring that data is not lost.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] 1. In actual business, it is necessary to process various business logics on multiple data streams simultaneously. However, the current stream data computing framework provides step-by-step processing based on various API operators, and the processing process is relatively complicated.

[0005] 2. The checkpoint mechanism ensures that data is not lost when a failure occurs, but it cannot guarantee that data is not duplicated when it is accessed. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a real-time data stream statistics method and device, which can solve the problem of complex processing when the current stream data computing framework processes business logic of multiple data streams based on various API operators.

[0007] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a real-time data stream statistics method is provided, comprising: registering a virtual table of the data stream according to the read data stream; loading and running a preset SQL statement file on the virtual table of the data stream; and storing the real-time statistical results obtained after the running.

[0008] Optionally, before registering the virtual table of the data stream according to the read data stream, the method also includes: taking the checkpoint file storage path with the latest generation time and carrying the set identifier as the first path, and taking the result data file storage path with the latest generation time and carrying the set identifier as the second path; determining the latest checkpoint file according to the generation time of the first path and the generation time of the second path; and reading the data stream from the latest checkpoint file.

[0009] Optionally, the latest checkpoint file is determined according to the generation time of the first path and the generation time of the second path, including: if the generation time of the second path is later than the generation time of the first path, deleting the result data file storage path whose generation time is later than the generation time of the first path, and using the checkpoint file of the first path as the latest checkpoint file; if the generation time of the second path is earlier than the generation time of the first path, deleting the checkpoint file storage path whose generation time is later than the generation time of the second path, and using the checkpoint file corresponding to the remaining checkpoint file storage path with the latest generation time as the latest checkpoint file; if the generation time of the second path is equal to the generation time of the first path, using the checkpoint file of the first path as the latest checkpoint file.

[0010] Optionally, registering a virtual table of the data stream according to the read data stream includes: presetting an SQL statement file in a configuration file according to business logic; determining the number and name of the virtual tables of the data stream according to the category and number of the read data stream, and registering the virtual tables of the data stream.

[0011] Optionally, the real-time statistical results obtained after the operation are stored, including: the real-time statistical results obtained after loading and running a preset SQL statement file include result data and offset data; storing the result data and generating a result data file, judging whether the storage is successful, if successful, setting a setting identifier in the storage path of the result data file, and if unsuccessful, the process ends abnormally; storing the offset data and generating a checkpoint file, judging whether the storage is successful, if successful, setting the setting identifier in the storage path of the checkpoint file, and if unsuccessful, the process ends abnormally.

[0012] Optionally, after storing the real-time statistical results obtained after the operation, the method also includes: modifying the table name and / or date conversion function in the preset SQL statement file according to the offline environment data; using the modified SQL statement file to generate offline indicator results based on the offline environment data; and performing data comparison and verification on the real-time statistical results and the offline indicator results to determine the accuracy of the real-time statistical results.

[0013] According to another aspect of an embodiment of the present invention, a real-time data stream statistics device is provided, including: a virtual table registration module, used to: register a virtual table of the data stream according to the read data stream; an execution statement module, used to: load and run a preset SQL statement file on the virtual table of the data stream; and a data storage module, used to: store the real-time statistical results obtained after the execution.

[0014] Optionally, the device also includes a reading module, which is used to: use the storage path of the checkpoint file with the latest generation time and carrying a set identifier as the first path, and use the storage path of the result data file with the latest generation time and carrying the set identifier as the second path; determine the latest checkpoint file based on the generation time of the first path and the generation time of the second path; and read the data stream from the latest checkpoint file.

[0015] Optionally, the reading module is also used to: if the generation time of the second path is later than the generation time of the first path, delete the result data file storage path whose generation time is later than the generation time of the first path, and use the checkpoint file of the first path as the latest checkpoint file; if the generation time of the second path is earlier than the generation time of the first path, delete the checkpoint file storage path whose generation time is later than the generation time of the second path, and use the checkpoint file corresponding to the remaining checkpoint file storage path with the latest generation time as the latest checkpoint file; if the generation time of the second path is equal to the generation time of the first path, use the checkpoint file of the first path as the latest checkpoint file.

[0016] Optionally, the virtual table registration module is further used to: pre-set an SQL statement file in the configuration file according to the business logic; determine the number and name of the virtual tables of the data stream according to the category and number of the read data stream, and register the virtual tables of the data stream.

[0017] Optionally, the data storage module is also used to: load and run the preset SQL statement file to obtain real-time statistical results including result data and offset data; store the result data and generate a result data file, determine whether the storage is successful, if successful, set a setting identifier in the storage path of the result data file, and if unsuccessful, the process ends abnormally; store the offset data and generate a checkpoint file, determine whether the storage is successful, if successful, set the setting identifier in the storage path of the checkpoint file, and if unsuccessful, the process ends abnormally.

[0018] Optionally, the device also includes a data verification module, which is used to: modify the table name and / or date conversion function in the preset SQL statement file according to the offline environment data; use the modified SQL statement file to generate offline indicator results based on the offline environment data; and compare and verify the real-time statistical results with the offline indicator results to determine the accuracy of the real-time statistical results.

[0019] According to another aspect of the embodiments of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the statistical method of real-time data stream proposed in the aforementioned embodiments.

[0020] According to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the statistical method of real-time data stream proposed in the above embodiments is implemented.

[0021] An embodiment of the above invention has the following advantages or beneficial effects: because a technical means is adopted to directly load and run an SQL statement file that meets business requirements on a virtual table of a registered data stream, the technical problem of complex processing when a data stream computing framework processes multiple data streams based on various API operators is overcome, thereby achieving the technical effect of replacing complex conversion calculation functions by writing SQL statements to realize statistics of real-time data streams in business scenarios.

[0022] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0024] Figure 1 is a schematic diagram of a basic process of a real-time data stream statistics method according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of a SQL statement execution process according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of a preferred process of a real-time data stream statistics method according to an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of basic modules of a real-time data stream statistics device according to an embodiment of the present invention;

[0028] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0029] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] Spark Streaming is a streaming data computing framework. It is widely used in the field of real-time processing in combination with the open source stream processing platform Kafka. In actual scenarios, it is often necessary to connect two or three Kafka data streams to the Spark Streaming framework, filter the data streams, convert fields, aggregate statistics of different dimensions, and output statistical results according to different business scenarios. Real-time statistics based on Spark streaming can ensure that data will not be lost through the checkpoint mechanism provided by itself, and can be restored from the last checkpoint after failure and restart. However, this mechanism can only ensure that data will not be lost, but cannot ensure that data will not be processed repeatedly, which is inconsistent with the requirement of exactly once processing when expecting real-time data processing. The existing technical solutions use various operators of the Spark Streaming API (such as foreachRDD, transform, reduceByKey, etc.) to perform data conversion and other processing operations on the connected data streams to implement relevant business logic.

[0032] In order to solve the problem that the checkpoint mechanism cannot guarantee that data will not be duplicated when accessed, the prior art uses the repartition operator (repartition) of the open source spark API to process the output action operator (action), so that multiple partitioned data can be turned into one partitioned data (partition), and then the transaction function of the database can be used to operate on this partitioned data to ensure that data duplication will not occur during the data processing process. Alternatively, the output result data contains offset data (also called checkpoint offset data), so that submitting the result and submitting the offset are completed in one operation, and there will be no data loss or repeated processing. The offset data in the last submitted result can be used during fault recovery.

[0033] However, the code framework of the existing solution is not very versatile. Different business processing logic-related spark operator codes need to be written for different business processing needs, and the logic implementation is relatively complex; it is not easy to troubleshoot problems when verifying DIFF (data diff) between real-time data and offline data. By setting the repartitioning parameters so that the action has only one partition, the concurrency of program processing is affected, and the processing delay within the data batch time is increased, especially when the data volume is large, which will affect the program performance; writing the checkpoint offset data and the result data together will cause the result data and the checkpoint offset data to be tightly coupled, which is inconvenient to use the result data separately later.

[0034] Figure 1 FIG. 1 is a schematic diagram of the basic process of the real-time data stream statistics method according to an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a real-time data flow statistics method, including:

[0035] Step S101. Register a virtual table of the data stream according to the read data stream;

[0036] Step S102: Load and run a preset SQL (Structured Query Language) statement file on the virtual table of the data stream;

[0037] Step S103: Store the real-time statistical results obtained after the operation.

[0038] Specifically, in the external configuration file of the Spark streaming processing program, SQL statement files are written according to the business logic. After Spark streaming reads multiple data streams, virtual tables of the data streams are registered respectively. The written SQL statement files are directly loaded and run on the virtual tables, and the calculation results are generated and stored. When there is a new business logic requirement, only a new SQL statement file needs to be written. Whether other Spark operator codes are modified does not need to be concerned, and the statistical function of the corresponding business logic can be quickly implemented. For example, in the logistics online advertising statistics business, it is necessary to count indicators such as the exposure of logistics online advertising and the number of clicks of users. By accessing the Kafka topic (Kafka data stream) of the exposure and click of logistics advertising, the exposure virtual table (impress_table) and the click virtual table (click_table) of the advertising are registered, and the business SQL statement is written: select exposure, click from exposure table UNION click table where (filter illegal clicks and exposures), execute the SQL statement, process the exposure and click data in real time, and generate the report data of the user's real-time exposure and click.

[0039] The embodiment of the present invention adopts the technical means of directly loading and running SQL statement files that meet business requirements on the virtual table of the registered data flow, thereby overcoming the technical problem of complex processing when the data flow computing framework processes multiple data flows based on various API (Application Programming Interface) operators, thereby achieving the technical effect of replacing complex conversion calculation functions by writing SQL statements to realize statistics of real-time data flows in business scenarios.

[0040] In an embodiment of the present invention, step S101 registers a virtual table of the data stream according to the read data stream, including: pre-setting an SQL statement file in a configuration file according to business logic; determining the number and name of the virtual tables of the data stream according to the category and number of the read data stream, and registering the virtual tables of the data stream.

[0041] The embodiment of the present invention further includes, before registering the virtual table of the data stream according to the read data stream in step S101: using the checkpoint file storage path with the latest generation time and carrying a set identifier (the set identifier may include but is not limited to a SUCCESS identifier) ​​as the first path, and using the result data file storage path with the latest generation time and carrying the set identifier as the second path; determining the latest checkpoint file according to the generation time of the first path and the generation time of the second path; and reading the data stream from the latest checkpoint file.

[0042] The embodiment of the present invention adopts a technical means of determining the latest checkpoint file according to the generation time of the first path and the generation time of the second path, and reading the data stream from the latest checkpoint file, thereby achieving exactly-once processing of data, and the data will not be lost or duplicated when a failure occurs and the system is restarted.

[0043] Based on the above embodiments, in an embodiment of the present invention, determining the latest checkpoint file according to the generation time of the first path and the generation time of the second path may include: if the generation time of the second path is later than the generation time of the first path, deleting the result data file storage path whose generation time is later than the generation time of the first path, and using the checkpoint file of the first path as the latest checkpoint file; if the generation time of the second path is earlier than the generation time of the first path, deleting the checkpoint file storage path whose generation time is later than the generation time of the second path, and using the checkpoint file corresponding to the remaining checkpoint file storage path with the latest generation time as the latest checkpoint file; if the generation time of the second path is equal to the generation time of the first path, using the checkpoint file of the first path as the latest checkpoint file.

[0044] The embodiment of the present invention determines the latest checkpoint file by comparing the generation time of the first path with the generation time of the second path, thereby realizing just-once processing of data in subsequent methods. When a failure occurs and restarts, data is read and processed from the latest checkpoint file, so that data will not be lost or repeated, thereby realizing real-time processing of data streams.

[0045] Step S103 in the embodiment of the present invention stores the real-time statistical results obtained after the operation, including: the real-time statistical results obtained after loading and running the preset SQL statement file include result data and offset data; storing the result data and generating a result data file, judging whether the storage is successful, if successful, setting a setting mark in the storage path of the result data file, and if unsuccessful, the process ends abnormally; storing the offset data and generating a checkpoint file, judging whether the storage is successful, if successful, setting the setting mark in the storage path of the checkpoint file, and if unsuccessful, the process ends abnormally.

[0046] The embodiment of the present invention stores result data and offset data separately, and stores the generated offset data separately after each SQL statement is executed. The next operation directly restores and reads the Kafka topic data from the last saved result data and offset data, which can ensure that the data is not repeated, solve the problem of data duplication or loss after restart that may be caused by various abnormalities, and ensure the accuracy of the data. Specifically, the checkpoint data and the result data can be stored separately in HDFS (Hadoop Distributed File SystemHadoop, distributed file system), which not only ensures the stability of the data, but also allows the result data to be used independently.

[0047] After storing the real-time statistical results obtained after the operation in step S103, the embodiment of the present invention also includes: modifying the table name and / or date conversion function in the preset SQL statement file according to the offline environment data; using the modified SQL statement file to generate offline indicator results based on the offline environment data; and performing data comparison and verification on the real-time statistical results and the offline indicator results to determine the accuracy of the real-time statistical results.

[0048] The accuracy verification and comparison of data indicators produced by real-time data processing is usually verified by comparing offline data indicator results to verify whether the real-time results are correct, and the SQL statements used in the above embodiments can be used for verification. Because the business logic of real-time and offline is consistent, the SQL statements generated by the above real-time virtual table can be directly run in the offline environment by simply modifying the table name and individual functions to generate offline indicator result data, which is very convenient for comparing real-time and offline result data, and is also simpler for verifying DIFF with offline data.

[0049] Figure 2 Schematic diagram of the SQL statement execution process according to an embodiment of the present invention. Figure 2 As shown in the figure, after Spark streaming reads multiple data streams from the latest checkpoint, it registers virtual tables of the data streams respectively, directly loads and runs the written SQL statement files on the virtual tables, and stores the generated incremental data (i.e., result data) and checkpoint data in HDFS respectively. The problem of multi-data source fusion and step-by-step processing of business logic can be solved by registering multiple data sources as multiple virtual tables and converting business logic into various filter conditions of SQL statements. Executing SQL statements that meet business requirements on the registered virtual tables solves the cumbersome operation problems of calling multiple data sources and executing multiple filter conditions in steps. Users can write SQL statements that meet business requirements in advance through configuration files, and determine the number and name of registered virtual tables according to the type and number of input Kafka data streams. For new businesses and new requirements, only corresponding SQL statements need to be written to implement them, and there is no need to modify the specific code in the framework.

[0050] Figure 3 FIG. 1 is a schematic diagram of a preferred process of a real-time data stream statistics method according to an embodiment of the present invention. Figure 3 As shown, after the program starts, find the path P1 with the closest time to the SUCCESS mark in the HDFS checkpoint file. Find the path P2 with the closest time to the SUCCESS mark in the HDFS data file. Compare whether the time of data path P2 is greater than the time of checkpoint path P1. If it is greater, it means that the program is abnormal after the latest batch of result data files are written, resulting in the failure to write the checkpoint file successfully. The program needs to delete all data paths whose data path time is greater than the checkpoint path time of process 1 to ensure that there will be no data duplication after the program is started from the last checkpoint. If the time of data path P2 is equal to the time of checkpoint path P1, it means that the last batch of data paths and checkpoint paths are successfully written, and no special processing is required. If the time of data path P2 is less than the time of checkpoint path P1, it means that the data of the last batch of data paths has been deleted abnormally. The program needs to delete the checkpoint data files whose checkpoint path time is greater than the data path time to ensure that a copy of the abnormally lost data files will be pulled again when the program starts. After processing the above situations, read data from the latest checkpoint file for parsing and business processing. After the business is processed, result data will be generated and stored in HDFS. If the data is written successfully, the SUCCESS mark will be dropped. If it is not written successfully, the program will exit abnormally. After the result data is written successfully, the current checkpoint data will be written to HDFS. If the data is written successfully, the SUCCESS mark will be dropped. If it is not written successfully, the program will exit abnormally.

[0051] Figure 4 Schematic diagram of basic modules of a real-time data stream statistics device according to an embodiment of the present invention. Figure 4 As shown, the embodiment of the present invention provides a real-time data stream statistics device 400, including:

[0052] The virtual table registration module 401 is used to: register the virtual table of the data stream according to the read data stream;

[0053] The statement execution module 402 is used to: load and execute a preset SQL statement file on the virtual table of the data flow;

[0054] The data storage module 403 is used to store the real-time statistical results obtained after the operation.

[0055] The embodiment of the present invention adopts the technical means of directly loading and running SQL statement files that meet business requirements on the virtual table of the registered data flow, thereby overcoming the technical problem of complex processing when the data flow computing framework processes multiple data flows based on various API operators, thereby achieving the technical effect of replacing complex conversion calculation functions by writing SQL statements to realize statistics of real-time data flows in business scenarios.

[0056] In an embodiment of the present invention, the device also includes a reading module, which is used to: use the storage path of the checkpoint file with the latest generation time and carrying the set identifier as the first path, and use the storage path of the result data file with the latest generation time and carrying the set identifier as the second path; determine the latest checkpoint file based on the generation time of the first path and the generation time of the second path; and read the data stream from the latest checkpoint file.

[0057] The embodiment of the present invention adopts a technical means of determining the latest checkpoint file according to the generation time of the first path and the generation time of the second path, and reading the data stream from the latest checkpoint file, thereby achieving exactly-once processing of data, and the data will not be lost or duplicated when a failure occurs and the system is restarted.

[0058] In an embodiment of the present invention, the reading module is also used for: if the generation time of the second path is later than the generation time of the first path, deleting the result data file storage path whose generation time is later than the generation time of the first path, and using the checkpoint file of the first path as the latest checkpoint file; if the generation time of the second path is earlier than the generation time of the first path, deleting the checkpoint file storage path whose generation time is later than the generation time of the second path, and using the checkpoint file corresponding to the remaining checkpoint file storage path with the latest generation time as the latest checkpoint file; if the generation time of the second path is equal to the generation time of the first path, using the checkpoint file of the first path as the latest checkpoint file.

[0059] The embodiment of the present invention determines the latest checkpoint file by comparing the generation time of the first path with the generation time of the second path, thereby realizing just-once processing of data in subsequent methods. When a failure occurs and restarts, data is read and processed from the latest checkpoint file, so that data will not be lost or repeated, thereby realizing real-time processing of data streams.

[0060] In the embodiment of the present invention, the virtual table registration module 401 is further used to: pre-set the SQL statement file in the configuration file according to the business logic; determine the number and name of the virtual tables of the data flow according to the category and number of the read data flow, and register the virtual tables of the data flow.

[0061] The embodiment of the present invention can replace complex conversion calculation functions by writing SQL statements to process statistical problems of real-time data streams in business scenarios, thereby greatly improving processing efficiency.

[0062] In an embodiment of the present invention, the data storage module 403 is also used to: load and run the preset SQL statement file to obtain real-time statistical results including result data and offset data; store the result data and generate a result data file, determine whether the storage is successful, if successful, set a setting mark in the storage path of the result data file, if not successful, the process ends abnormally; store the offset data and generate a checkpoint file, determine whether the storage is successful, if successful, set the setting mark in the storage path of the checkpoint file, if not successful, the process ends abnormally.

[0063] After each SQL statement is executed, the generated offset data is stored separately. The next time the SQL statement is run, the topic data of Kafka is directly restored from the result data and offset data saved last time. This ensures that the data is not repeated, solves the problem of data duplication or loss after restart caused by various exceptions, and ensures the accuracy of the data. Specifically, the checkpoint data and result data can be stored separately in HDFS (Hadoop Distributed File SystemHadoop, distributed file system), which not only ensures the stability of the data, but also allows the result data to be used independently.

[0064] In an embodiment of the present invention, the device also includes a data verification module, which is used to: modify the table name and / or date conversion function in the preset SQL statement file according to the offline environment data; use the modified SQL statement file to generate offline indicator results based on the offline environment data; and perform data comparison and verification on the real-time statistical results and the offline indicator results to determine the accuracy of the real-time statistical results.

[0065] The accuracy verification and comparison of data indicators produced by real-time data processing is usually verified by comparing offline data indicator results to verify whether the real-time results are correct, and the SQL statements used in the above embodiments can be used for verification. Because the business logic of real-time and offline is consistent, the SQL statements generated by the above real-time virtual table can be directly run in the offline environment by simply modifying the table name and individual functions to generate offline indicator result data, which is very convenient for comparing real-time and offline result data, and is also simpler for verifying DIFF with offline data.

[0066] Figure 5 An exemplary system architecture 500 is shown to which a real-time data stream statistics method or a real-time data stream statistics apparatus according to an embodiment of the present invention may be applied.

[0067] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0068] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0069] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0070] Server 505 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 501, 502, and 503. The backend management server may analyze and process received data such as product information query requests, and feed back the processing results to the terminal device.

[0071] It should be noted that the real-time data stream statistics method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the real-time data stream statistics device is generally set in the server 505 .

[0072] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0073] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0074] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the real-time data stream statistical method proposed in the above-mentioned embodiment.

[0075] The computer-readable medium of the present invention stores a computer program, which, when executed by a processor, implements the real-time data stream statistics method proposed in the above-mentioned embodiment.

[0076] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0077] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0078] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0079] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0080] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0081] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, may be described as: a processor including: a virtual table registration module, an execution statement module, and a data storage module. The names of these modules do not constitute a limitation on the modules themselves in certain cases, for example, the virtual table registration module may also be described as "a module for registering a virtual table of the data stream according to the read data stream".

[0083] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: registering a virtual table of the data stream according to the read data stream; loading and running a preset SQL statement file on the virtual table of the data stream; and storing the real-time statistical results obtained after the running.

[0084] It can be seen from the statistical method of real-time data streams according to the embodiment of the present invention that, because a technical means is adopted to directly load and run an SQL statement file that meets business requirements on a virtual table of a registered data stream, the technical problem of complex processing when a data stream computing framework processes multiple data streams based on various API operators is overcome, thereby achieving the technical effect of replacing complex conversion calculation functions by writing SQL statements to realize statistics of real-time data streams in business scenarios.

[0085] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time data flow statistics method, characterized in that: include: According to the read data stream, register a virtual table of the data stream; Loading and running a preset SQL statement file on the virtual table of the data stream; Store the real-time statistical results obtained after running; Before registering the virtual table of the data stream according to the read data stream, the method further includes: The storage path of the checkpoint file with the latest generation time and carrying the setting identifier is used as the first path, and the storage path of the result data file with the latest generation time and carrying the setting identifier is used as the second path; Determining a latest checkpoint file according to the generation time of the first path and the generation time of the second path; Reading a data stream from the latest checkpoint file; Determining the latest checkpoint file according to the generation time of the first path and the generation time of the second path includes: If the generation time of the second path is later than the generation time of the first path, then deleting the result data file storage path whose generation time is later than the generation time of the first path, and taking the checkpoint file of the first path as the latest checkpoint file; If the generation time of the second path is earlier than the generation time of the first path, the checkpoint file storage path whose generation time is later than the generation time of the second path is deleted, and the checkpoint file corresponding to the remaining checkpoint file storage path with the latest generation time is used as the latest checkpoint file; If the generation time of the second path is equal to the generation time of the first path, the checkpoint file of the first path is used as the latest checkpoint file.

2. The method according to claim 1, characterized in that Registering a virtual table of the data stream according to the read data stream includes: Pre-set SQL statement files in the configuration file according to business logic; According to the type and quantity of the read data stream, the quantity and name of the virtual table of the data stream are determined, and the virtual table of the data stream is registered.

3. The method according to claim 1, characterized in that The real-time statistical results obtained after running are stored, including: The real-time statistical results obtained after loading and running the preset SQL statement file include result data and offset data; The result data is stored and a result data file is generated, and it is determined whether the storage is successful. If successful, a setting mark is set in the storage path of the result data file; if unsuccessful, the process ends abnormally; The offset data is stored and a checkpoint file is generated to determine whether the storage is successful. If successful, the setting identifier is set in the storage path of the checkpoint file. If unsuccessful, the process ends abnormally.

4. The method according to claim 1, characterized in that: After storing the real-time statistical results obtained after the operation, the method further includes: Modify the table name and / or date conversion function in the preset SQL statement file according to the offline environment data; Use the modified SQL statement file to generate offline indicator results based on offline environment data; The real-time statistical results are compared and verified with the offline indicator results to determine the accuracy of the real-time statistical results.

5. A real-time data stream statistics device, characterized in that: include: A virtual table registration module is used to: register a virtual table of the data stream according to the read data stream; An execution statement module, used to: load and run a preset SQL statement file on the virtual table of the data stream; The data storage module is used to store the real-time statistical results obtained after the operation; The device also includes a reading module, which is used to: The storage path of the checkpoint file with the latest generation time and carrying the setting identifier is used as the first path, and the storage path of the result data file with the latest generation time and carrying the setting identifier is used as the second path; Determining a latest checkpoint file according to the generation time of the first path and the generation time of the second path; Reading a data stream from the latest checkpoint file; The read module is also used to: If the generation time of the second path is later than the generation time of the first path, then deleting the result data file storage path whose generation time is later than the generation time of the first path, and taking the checkpoint file of the first path as the latest checkpoint file; If the generation time of the second path is earlier than the generation time of the first path, the checkpoint file storage path whose generation time is later than the generation time of the second path is deleted, and the checkpoint file corresponding to the remaining checkpoint file storage path with the latest generation time is used as the latest checkpoint file; If the generation time of the second path is equal to the generation time of the first path, the checkpoint file of the first path is used as the latest checkpoint file.

6. The device according to claim 5, characterized in that The registered virtual table module is also used to: Pre-set SQL statement files in the configuration file according to business logic; According to the type and quantity of the read data stream, the quantity and name of the virtual table of the data stream are determined, and the virtual table of the data stream is registered.

7. The device according to claim 5, characterized in that The data storage module is also used to: The real-time statistical results obtained after loading and running the preset SQL statement file include result data and offset data; The result data is stored and a result data file is generated, and it is determined whether the storage is successful. If successful, a setting mark is set in the storage path of the result data file; if unsuccessful, the process ends abnormally; The offset data is stored and a checkpoint file is generated to determine whether the storage is successful. If successful, the setting identifier is set in the storage path of the checkpoint file. If unsuccessful, the process ends abnormally.

8. The device according to claim 6, characterized in that The device also includes a data verification module, which is used to: Modify the table name and / or date conversion function in the preset SQL statement file according to the offline environment data; Use the modified SQL statement file to generate offline indicator results based on offline environment data; The real-time statistical results are compared and verified with the offline indicator results to determine the accuracy of the real-time statistical results.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images