Data playback display method and device and stream processing system

Abstract

The invention discloses a data playback display method and device and a stream processing system. The method comprises the steps that request information is received, the request information comprises at least one target moment, and the request information is used for requesting operation executed by a playback stream processing system at the target moment; index data corresponding to the target moment is read from a storage system, the index data comprises performance index data of all operators in the stream processing system, and each operator is a function unit in the stream processing system; a playback view is generated according to the index data, and static attribute information of each operator at the target moment and dynamic interaction information among the multiple operators are recorded in the playback view. The technical problems that operator-level monitoring cannot be performed on the stream processing system and playback operation for the stream processing system cannot be realized due to the fact that logs of operators of the stream processing system cannot be recorded in related technologies are solved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method and device for displaying data playback and a stream processing system. Background Art

[0002] With the rapid development of the Internet of Things (IoT), artificial intelligence (AI), and 5G communication technologies, especially in real-time business scenarios that require quick responses. The open-source stream processing system (Flink) has become the mainstream choice for real-time big data processing due to its high performance, high throughput, and low latency characteristics. The powerful stream processing capabilities provided by the stream processing system can process unbounded data streams and ensure the consistency and accuracy of data processing. Although the stream processing system itself has certain monitoring and status management functions, there are still a large number of limitations in terms of depth and flexibility. For example, the monitoring and status management functions inherent in the stream processing system can only provide historical information at the job level, lacking records of the internal execution details of the job; and it can only view the final data of completed jobs and cannot view the data throughout the entire life cycle, that is, the playback function; there is a lack of a function for logging operator logs and it is impossible to monitor at the operator level.

[0003] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of this application provide a method and device for displaying data playback and a stream processing system to at least solve the technical problems that the operator logs of the stream processing system cannot be recorded in the related art, resulting in the inability to monitor the operator level of the stream processing system and the inability to implement the playback operation for the stream processing system.

[0005] According to one aspect of the embodiments of this application, a method for displaying data playback is provided, including: receiving request information, where the request information includes at least one target time, and the request information is used to request to playback the operations performed by the stream processing system at the target time; reading the metric data corresponding to the target time from the storage system, where the metric data includes: the performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; generating a playback view according to the metric data, where the playback view records the static attribute information of each operator at the target time and the dynamic interaction information between multiple operators.

[0006] Optionally, read the metric data corresponding to the target time from the storage system. The storage system stores data in the following manner: Receive the real-time data generated by the stream processing system when performing operations at the current time, and store the real-time data as the first type of file. The real-time data includes: real-time metric data, the first timestamp corresponding to the real-time metric data, and the metric name of the real-time metric data. When receiving the second type of file composed of compressed data, determine the target first type of file corresponding to the second type of file among multiple first type of files, and delete the target first type of file. The compressed data is generated by compressing the historical data generated by the stream processing system when performing operations at multiple historical times. The historical data includes: historical metric data, the second timestamp corresponding to the historical metric data, and the metric name of the historical metric data.

[0007] Optionally, determining the target first type of file corresponding to the second type of file includes: determining the second timestamp included in the second type of file, where the second type of file includes multiple second timestamps; determining the target first timestamp indicating the same time as the second timestamp among multiple first timestamps; and determining the first type of file including the target first timestamp as the target first type of file corresponding to the second type of file.

[0008] Optionally, the compressed data is generated by the following method: Receive the real-time data generated by the stream processing system when performing operations at the current time, and store the real-time data in a circular buffer. The circular buffer is an array with a fixed memory size. After each storage of real-time data into the circular buffer, determine the memory margin of the circular buffer, where the memory margin is used to indicate the proportion of free memory in the circular buffer. When the memory margin is less than or equal to the preset memory margin, perform compression processing on the data stored in the circular buffer to obtain the compressed data.

[0009] Optionally, storing the real-time data in the circular buffer includes: determining the storage order according to the first timestamp included in the real-time data; storing the real-time data in the storage order. When storing, store the first timestamp and the metric name in the real-time data as keys, and store the real-time metric data as values.

[0010] Optionally, read the metric data corresponding to the target time from the storage system. The metric data corresponding to the target time is determined by the following method: Determine the target file including the target timestamp among multiple files stored in the storage system. The target timestamp is the timestamp indicating the same time as the target time. Each file consists of a file header, an index block, a file tail, and a data block. The index block records the time range corresponding to the file, and the data block is used to store metric data. Query the target metric data having a key-value relationship with the target timestamp in the target file; and determine the target metric data as the metric data corresponding to the target time.

[0011] Optionally, a playback view is generated according to the metric data, including: generating a list for describing the performance of an operator according to the static attribute information; and generating a ray for indicating the data stream between multiple lists according to the dynamic interaction information, where the data stream is used to indicate the flow direction and transformation process of data between multiple operators.

[0012] Optionally, the target moment is determined by the following method: detecting the operation performed by the target object on the interaction interface; and when it is detected that the operation is dragging the progress bar and the duration for which the target object stops operating is greater than the preset duration, determining the moment selected by the target object in the progress bar as the target moment.

[0013] According to another aspect of the embodiments of the present application, a stream processing system is further provided, including: a work processing node TaskMnager, a log collector and a log aggregator embedded in the TaskMnager, where the TaskMnager is used to provide resources for multiple operators, and each operator is a functional unit in the stream processing system; the log collector is used to collect the performance metric data of each operator in real time, and write the performance metric data into the log aggregator and the storage system connected to the stream processing system simultaneously; the log aggregator is used to receive the performance metric data, store the performance metric data in a circular buffer, and perform compression processing on the data stored in the circular buffer when the memory margin of the circular buffer is less than or equal to the preset memory margin, where the circular buffer is an array with a fixed memory, and the memory margin is used to indicate the proportion of the free memory in the circular buffer.

[0014] According to another aspect of the embodiments of the present application, a display device for data playback is further provided, including: a receiving module, configured to receive request information, where the request information includes at least one target moment, and the request information is used to request to playback the operations performed by the stream processing system at the target moment; a reading module, configured to read the metric data corresponding to the target moment from the storage system, where the metric data includes: the performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; a generating module, configured to generate a playback view according to the metric data, where the static attribute information of each operator at the target moment and the dynamic interaction information between multiple operators are recorded in the playback view.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, in which a computer program is stored, and when the device where the non-volatile storage medium is located runs the computer program, the above-mentioned method for displaying data playback is executed.

[0016] According to another aspect of the embodiments of the present application, an electronic device is further provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute the above-described method for presenting data playback through the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is further provided, including computer instructions, which implement the steps of the above-described method for presenting data playback when executed by a processor.

[0018] In the embodiments of the present application, request information is received, where the request information includes at least one target time, and the request information is used to request an operation to be performed by the playback stream processing system at the target time; metric data corresponding to the target time is read from the storage system, where the metric data includes: performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; a playback view is generated according to the metric data, where the playback view records the static attribute information of each operator at the target time and the dynamic interaction information between multiple operators. By using a log collector and a log aggregator embedded in the working processing nodes of the stream processing system to collect the logs of each functional unit during the process of the stream processing system handling tasks, the logs record the data throughout the entire life cycle of the stream processing system performing tasks, achieving the purpose of monitoring the stream processing system at the operator level. Generating a playback view that shows the process of the stream processing system performing tasks based on the logs of each functional unit thus realizes the technical effect of playing back the processing process of the stream processing system, and further solves the technical problems that the related art cannot record the logs of the operators of the stream processing system, resulting in the inability to monitor the stream processing system at the operator level and the inability to implement the playback operation for the stream processing system. Description of the Drawings

[0019] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0020] Figure 1 is a hardware structure block diagram of a computer terminal for implementing the method for presenting data playback according to the embodiments of the present application;

[0021] Figure 2 is a step flowchart of the method for presenting data playback according to the embodiments of the present application;

[0022] Figure 3 is a schematic diagram of a storage system storing data using a dual-write mechanism according to the embodiments of the present application;

[0023] Figure 4It is a schematic diagram of the structure of a file according to an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of a playback view according to an embodiment of the present application;

[0025] Figure 6 It is an architecture diagram of a stream processing system according to an embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of a display device for data playback according to an embodiment of the present application. Detailed implementation manners

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0029] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0030] Circular buffer: Also known as a circular queue or a cyclic buffer, it is a data structure that uses a buffer with a fixed size to implement the function of a queue.

[0031] In the related art, the monitoring function of the stream processing system itself can only provide historical information at the job level, lacking the operator-level log record that is accurate to each functional unit of the stream processing system, and it is impossible to view the data of the entire life cycle of the tasks executed by the stream processing system. Therefore, the playback function for the stream processing system cannot be realized. To solve this problem, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0032] According to an embodiment of the present application, a method embodiment of a method for data playback display is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0033] The method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a method for data playback display is shown. As Figure 1 shown, the computer terminal 10 may include one or more (shown as 102a, 102b,..., 102n in the figure) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0034] It should be noted that the above one or more processors 102 and / or other data processing circuits are generally referred to as "data processing circuits" in this article. The data processing circuit can be embodied as software, hardware, firmware, or any combination thereof in whole or in part. In addition, the data processing circuit can be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10. As involved in the embodiment of the present application, the data processing circuit is a processor control (such as the selection of a variable resistance terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data playback display method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned data playback display method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] The display can be, for example, a touch-screen liquid crystal display (LCD), and the liquid crystal display enables a user to interact with the user interface of the computer terminal 10.

[0038] The embodiments of the present application provide a data playback display method that can run in the above operating environment. Figure 2 It is a step flowchart of the data playback display method provided by the embodiments of the present application, as Figure 2 shown, and the method includes the following steps:

[0039] Step S202, receive request information, where the request information includes at least one target moment, and the request information is used to request the playback stream processing system to perform an operation at the target moment.

[0040] The present application provides a playback function for a stream processing system. By collecting the logs generated by the functional processing units in the stream processing system when processing tasks, a playback view of the stream processing system at a specific moment, or a playback view of the entire task execution cycle of the stream processing system, is generated based on the data recorded in the logs, achieving the technical effect of playing back the process of the stream processing system executing tasks. In the embodiments of the present application, specifically, which moment / which period of time of the stream processing system the playback view is generated for is specified by the user. When the user desires to play back the process of the stream processing system executing tasks, a request message will be sent through an interaction interface associated with the stream processing system (such as the Flink Web UI) or other terminal devices communicating with the stream processing system. The request message contains the playback moment specified by the user (i.e., the target moment). In step S202, after receiving the request message indicating to play back the operations performed during the task execution process of the stream processing system, the playback moment (i.e., the target moment) can be determined. If the user only specifies to play back the operations performed by the stream processing system when processing tasks at a certain time point, the playback moment (i.e., the target moment) contained in the request message is one; if the user only specifies to play back the operations performed by the stream processing system when processing tasks at multiple different time points, or indicates to play back the operations performed by the stream processing system when processing tasks within a certain period of time, the playback moments (i.e., the target moments) contained in the request message are multiple. On the other hand, the target moment is also the moment when the data requested for playback is located and the moment when the operation requested for playback is located. For example, a request to play back the operations performed by the stream processing system at 13:00 on year A, month B, and day C.

[0041] According to some optional embodiments of the present application, the target moment is determined by the following method: detecting the operations performed by the target object on the interaction interface; in the case where it is detected that the operation is dragging the progress bar and the duration when the target object stops operating is greater than the preset duration, determining the moment selected by the target object in the progress bar as the target moment.

[0042] The method provided by the embodiment of the present application allows a user (i.e., the target object) to drag a progress bar on an interaction interface to view the status of a stream processing task at different time nodes, including logs, metrics, and various status information, and can reproduce the history of task execution; and, a playback moment (i.e., the target moment) is selected by dragging the progress bar. When the user selects a playback moment by dragging the progress bar on the interaction interface, the playback moment can be determined by the method provided in this embodiment. By detecting the operation behavior of the user on the interaction interface (such as the Flink Web UI), it is determined whether the operation behavior performed by the user is dragging the progress bar, and by detecting the duration of the user's stop operation behavior, it is determined whether the user has ended the operation. When it is detected that the operation behavior performed by the user is dragging the progress bar and the user stops dragging the progress bar operation for more than a preset duration (such as 1 second), the time point selected by the user on the progress bar is determined as the playback moment. The time point selected by the user on the progress bar can be a certain moment on the progress bar or a period of time on the progress bar.

[0043] Step S204, read the metric data corresponding to the target moment from the storage system, where the metric data includes: the performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system.

[0044] Since the log sampler (LogSampler) and log collector (LogCollector) embedded in the stream processing system in the embodiments of the present application record the logs generated when each functional unit in the stream processing system executes tasks during the task execution process of the stream processing system, and store these logs in the storage system, and these logs contain the metric data of each functional unit at each moment of task execution. Therefore, in step S204, the metric data corresponding to the above playback moment (i.e., the target moment) can be read from the storage system; these metric data are used to indicate the performance metrics of each operator (i.e., functional unit) in the stream processing system (Flink). For example, processing speed, input data volume, output data volume, latency, etc. Among them, the processing speed (Processing Speed) is used to indicate the amount of data processed by the operator per second; the input data volume includes: the number of written data items (Input Data Count), that is, the number of data items received by the operator from the upstream operator per second; the size of the written data volume, that is, the amount of data received by the operator from the upstream operator per second; the output data volume includes: the number of written data items (OutputData Count), that is, the number of data items sent by the operator to the downstream operator per second, and the size of the written data volume (Output DataSize), that is, the amount of data sent by the operator to the downstream operator per second; and, the metric data can also include the idle time (Idle Time): used to indicate the time when the operator is in an idle state during the gap of processing data; the busy time (BusyTime): the time when the operator is in a busy state and actually processes data.

[0045] According to some optional embodiments of the present application, the metric data corresponding to the target moment is read from the storage system, where the storage system stores data in the following manner: receiving the real-time data generated by the stream processing system during the current moment operation, and storing the real-time data as the first type of file, where the real-time data includes: real-time metric data, the first timestamp corresponding to the real-time metric data, and the metric name of the real-time metric data; when receiving the second type of file composed of compressed data, determining the target first type of file corresponding to the second type of file among multiple first type of files, and deleting the target first type of file, where the compressed data is generated by compressing the historical data generated by the stream processing system during multiple historical moments, and the historical data includes: historical metric data, the second timestamp corresponding to the historical metric data, and the metric name of the historical metric data.

[0046] Figure 3It is a schematic diagram of a storage system using a dual-write mechanism for data storage. In this embodiment, the storage system for storing the logs of each operator of the stream processing system specifically uses the dual-write mechanism to store the data generated by the operations executed by the stream processing system. That is, when real-time metric data and other data related to the metric data are collected, they are simultaneously stored in the log collector (LogCollector) and the storage system. At this time, both the storage system and the log collector (LogCollector) have the real-time data generated by the operations executed by the stream processing system. During the process of the stream processing system executing tasks, the log collector (LogCollector) continuously receives data. When the memory occupied by the data stored in the log collector (LogCollector) reaches a preset condition, the log collector (LogCollector) will compress the previously received data, store the compressed data obtained after the compression process as a file (i.e., the second type of file), and send the compressed data file (i.e., the second type of file) to the storage system. The storage system will use the compressed data file to replace some pass-through files in the storage system. Specifically, it replaces the target pass-through file that records the same data as the compressed data file. Specifically, as Figure 3As shown, after the storage system receives the data (i.e., real-time data) collected in real time by the LogSampler, it will first store it as a pass-through file (i.e., the first type of file). Each time the storage system receives real-time data, it immediately stores it as a (first type) file. The LogCollector synchronously receives the data (i.e., real-time data) collected in real time by the LogSampler with the storage system. When the LogCollector determines that the compression condition is met according to the data volume, it compresses the stored data to obtain compressed data. These compressed data are stored as files (i.e., the second type of file) and then sent to the storage system. After the storage system receives the file composed of compressed data (i.e., the second type of file), it finds the target pass-through file (i.e., the target first type of file) containing the same metric data as the compressed data file (i.e., the second type of file) in the pass-through files (i.e., the first type of file) it stores, and deletes it, ensuring that when the same metric data is stored in both the pass-through file (i.e., the first type of file) and the compressed file (i.e., the second type of file) at the same time, only the compressed file (i.e., the second type of file) storing this type of metric data is retained in the storage system, saving storage space. That is to say, both compressed files (i.e., the second type of file) and pass-through files (i.e., the first type of file) can be stored in the storage system at the same time. However, when these two types of files appear in the storage system at the same time, the metric data contained in these two types of files must be different; among them, each pass-through file stores the real-time data generated by the stream processing system at a certain moment, including real-time metric data, the (first) timestamp corresponding to the real-time metric data (used to indicate the generation time of the real-time metric data), and the name of the real-time metric data, while the compressed file contains the data generated by the stream processing system at multiple previous moments. Therefore, the data contained in the compressed file is called historical data, including historical metric data, the (second) timestamp corresponding to the historical metric data (used to indicate the generation time of the historical metric data), and the name of the historical metric data.

[0047] Optionally, determining the target first type of file corresponding to the second type of file includes: determining the second timestamp included in the second type of file, where the second type of file includes multiple second timestamps; determining the target first timestamp indicating the same time as the second timestamp among multiple first timestamps; and determining the first type of file containing the target first timestamp as the target first type of file corresponding to the second type of file.

[0048] As mentioned in the previous embodiment, in order to save storage space, after receiving a file composed of compressed data (i.e., the second type of file), the storage system will use the compressed data file (i.e., the second type of file) to replace the file (i.e., the target first type of file) in the storage system that stores the same data as it; in this embodiment, since both the compressed data file and the pass-through file store the metric data of the stream processing system, the timestamp corresponding to the metric data, and the metric name of the metric data, therefore, in this embodiment, the target pass-through file (i.e., the target first type of file) replaced by the compressed data file can be determined in the pass-through file according to the timestamps stored in both the pass-through file and the compressed data file. When determining the target pass-through file replaced by the compressed data according to the timestamp, the specific steps are as follows: First, determine the (second) timestamp included in the compressed data file (i.e., the second type of file) and the (first) timestamp included in the pass-through file; further, determine multiple target first timestamps that indicate the same time as the multiple second timestamps included in a compressed data file, and determine the multiple pass-through files containing the multiple target first timestamps as the target pass-through files to be replaced by this compressed data file; in fact, since the compressed data file is generated by compressing the real-time data collected multiple times in real time, the compressed data file must contain multiple (second) timestamps; while the pass-through file is generated each time real-time data is received, and each pass-through file only contains one (first) timestamp; therefore, a compressed data file can replace multiple pass-through files, and the target pass-through files to be replaced will be deleted during the replacement process, thereby releasing storage space.

[0049] According to some other optional embodiments of the present application, the compressed data is generated by the following method: receiving the real-time data generated by the stream processing system performing operations at the current moment, and storing the real-time data in a circular buffer, where the circular buffer is an array with a fixed memory; after each time real-time data is stored in the circular buffer, determining the memory remaining amount in the circular buffer, where the memory remaining amount is used to indicate the proportion of the free memory in the circular buffer; in the case where the memory remaining amount is less than or equal to the preset memory remaining amount, performing compression processing on the data stored in the circular buffer to obtain compressed data.

[0050] As mentioned in the above embodiments, after the LogCollector determines that the compression condition is met based on the data volume, it will compress the stored data. Specifically, the method for determining whether the compression condition is met is as follows: the LogCollector will store the real-time data sent by the LogSampler in the circular buffer, and detect the remaining memory of the circular buffer after each storage. If the result detected after a certain data storage is that the remaining memory of the circular buffer is less than or equal to the preset remaining memory, it is determined that the compression condition has been met, and the data stored in the circular buffer will be compressed to generate compressed data. The above remaining memory refers to the proportion of the free memory in the circular buffer, that is, the percentage of the free memory in the circular buffer accounting for the total memory of the circular buffer; the preset remaining memory is the preset proportion of the free memory. For example, if the fixed size of the circular buffer is 10 megabytes (MB) and the preset proportion of the free memory is 20%, then when the data volume written into the circular buffer reaches 8 MB each time, the compression operation is triggered. When the compression operation is triggered, the data writing (i.e., storage) is not paused. At this time, the compression rate of the index file is generally as high as 1:25, and the index file only occupies about 0.32 megabits (mb), achieving the technical effect of releasing storage space. In the above solution, storing the real-time data sent by the LogSampler in the circular buffer means storing the data in a fixed-memory array, and this data structure of the array is the circular buffer mentioned in this embodiment.

[0051] Optionally, storing the real-time data in the circular buffer includes: determining the storage order according to the first timestamp included in the real-time data; storing the real-time data in the storage order. When storing, the first timestamp and the metric name in the real-time data are stored as keys, and the real-time metric data is stored as values.

[0052] In this embodiment, the LogCollector stores the real-time data sent by the LogSampler in a circular buffer according to the storage order. The above storage order is determined according to the timestamps included in the real-time data. For example, the timestamps are sorted according to the time indicated by the timestamps from early to late, and the sorting result indicates the generation time of the metric data. Therefore, the sorting result can be used as the storage order. In this embodiment, when writing real-time data into the circular buffer, the key-value relationship is followed. Specifically, the real-time metric data itself (or the value of the metric data) in the real-time data is written as the value on one side of the circular buffer, and the (first) timestamp + metric name in the real-time data is written as the key on the other side of the circular buffer. The above storage in the form of key-value pairs can ensure fast access and efficient storage of data. In addition, when reading data from the circular buffer, the keys can also be sorted and then the keys and values are written out together; for example, sorted according to the timestamps included in the keys, and then the keys and values are written out together. Highly similar key-values (when the similarity of two key-value pairs is greater than the preset similarity, these two key-value pairs are highly similar key-values) are columnar compressed after sorting.

[0053] According to some optional embodiments of the present application, the metric data corresponding to the target time is read from the storage system, where the metric data corresponding to the target time is determined by the following method: a target file containing the target timestamp is determined among multiple files stored in the storage system, where the target timestamp is the timestamp indicating the same time as the target time. Each file consists of a file header, an index block, a file tail, and a data block. The index block records the time range corresponding to the file, and the data block is used to store the metric data; the target metric data having a key-value relationship with the target timestamp is queried in the target file; and the target metric data is determined as the metric data corresponding to the target time.

[0054] When step S202 reads the metric data corresponding to the target time from the storage system, the metric data corresponding to the target time can be determined first according to the method provided in the embodiments of the present application. As mentioned in the above embodiments, whether the storage system stores compressed data files (i.e., the second type of files) or pass-through files (i.e., the first type of files), the stored files all contain the information of timestamps, and each timestamp corresponds to metric data (the metric data generated at the time indicated by the timestamp). Therefore, in this embodiment, the timestamp indicating the same time as the target time (i.e., the target timestamp) can be found among the multiple timestamps recorded in the files stored in the storage system first, and then the file to which the above target timestamp belongs (i.e., the target file) is further determined, and the metric data stored in these target files is determined as the metric data corresponding to the target time. In this embodiment, Figure 4It is a schematic diagram of the structure of a file. Whether it is a compressed data file or a pass-through file, it follows Figure 4 the structure shown, as Figure 4 shown, each file is composed of a file header (Header), a root index (RootIndex), a file footer (Foot), and a data block (DataStage); among them, the file header contains file version information, creation time, and file size information; the root index records the time range corresponding to the file to which it belongs (determined according to the timestamps contained in the file). The root index can be indexed by a key and contains a time index and an index of a certain key. Through the root index of each file, it can be quickly understood whether the file to which it belongs contains the time range or index key to be queried. As Figure 4 shown, each data block stores multiple stage identifiers (StageIndex) and stored data (Data). StageIndex represents the fast index of the data in the current data block (indexed according to the key-value relationship) and the position of the next StageIndex; in the last StageIndex of the data block, the position index of the next block is written first; Data is the specific metric data stored, that is, the metric data that the file to which the data block belongs should store (real-time metric data or historical metric data). If the file to which the data block belongs is a compressed data file (i.e., the second type of file), then Data is also compressed and needs to be decompressed to view. The file footer is used to store the check information of the file and the end identifier of the file to indicate the integrity of the entire file. In this embodiment, whether the file stored in the storage system is a compressed data file (i.e., the second type of file) or a pass-through file (i.e., the first type of file) can be determined according to the suffix in the file name. For example, the suffix of the file name of the compressed data file is ".metrics", while the suffix of the file name of the pass-through file is ".tmp_metrics".

[0055] Step S206, generating a playback view according to the metric data, where the playback view records the static attribute information of each operator at the target moment and the dynamic interaction information between multiple operators.

[0056] In step S206, a playback view is generated based on the metric data read from the storage system in step S204. The playback view details the static attribute information and dynamic interaction information of each operator (i.e., the functional unit of the stream processing system). The above-mentioned static attribute information refers to static data used to describe the type, identifier (such as number), configuration information, etc. of the operator. The dynamic interaction information refers to data indicating the state changes of the operator during task execution and dynamic data indicating the data flow direction during task execution. When there are multiple request times included in the request information, the order in which each operator performs corresponding operations is determined based on the generation time corresponding to the metric data, and the generated playback view also follows the time order followed during task execution during the playback task execution process.

[0057] Optionally, generating a playback view based on the metric data includes: generating a list for describing the performance of the operator according to the static attribute information; and generating a ray for indicating the data flow between multiple lists according to the dynamic interaction information, where the data flow is used to indicate the flow direction and transformation process of data between multiple operators.

[0058] In this embodiment, in the playback view generated based on the metric data, a list or other type of diagram for describing the performance of each operator is generated according to the static attribute information of each operator to detail the performance of the operator during task execution in the playback view. At the same time, the playback view also shows a ray for indicating the data flow direction generated according to the dynamic interaction information between multiple operators, and the ray intuitively shows the flow and transformation of data between operators through arrows. The static attribute information in the embodiments of the present application includes various information describing the performance of the operator from different aspects, such as the percentage of the execution task duration in the entire task processing duration (i.e., the busy time ratio Busy_Time), the percentage of the duration without processing activities in the entire task processing duration (i.e., the idle time ratio Idel_Time), the percentage of the memory resources consumed by the operator during execution in all available memory resources of the circular buffer (i.e., the memory ratio Mem), the number of data transactions or events processed by the operator per second (TPS), the rate or amount of data received by the operator from the upstream operator or data source (In_W), the rate or amount of data sent by the operator to the downstream operator or data receiver (Out_W), the amount of data or number of events read by the operator from the upstream per second (In_R), the amount of data or number of events sent by the operator to the downstream per second (Out_R), the queue load percentage of the operator receiving data (In_Q), the queue load percentage of the operator sending data (Out_Q), etc. The dynamic interaction information between operators includes: the amount of data transmitted between operators, the speed of data transmission between operators, the delay when data is transmitted between operators, and other information. Figure 5 is a schematic diagram of the playback view, Figure 5The playback view of the shown stream processing system at a specified moment shows a task processing process involving four operators (processworld), namely Axxxxxxx, Bxxxxxxx1, Cxxxxxxx1, and Bxxxxxxx2, as Figure 5 shown. Each operator has a corresponding list that records the static attribute information of the operator. The rays between multiple lists indicate the flow and transformation of data during the task processing through the arrows carried by the rays. Figure 5As can be seen from the playback view shown, at the playback moment corresponding to this playback view, the busy time percentage Busy_Time of operator Axxxxxxx is 71%, the idle time percentage Idel_Time is 4%, the memory percentage Mem is 30%, the number of data transactions processed per second (TPS) is 1018, the amount of data received from the upstream operator or data source (In_W) is 0, the amount of data sent to the downstream operator or data receiver (Out_W) is 1578 bits (b), the amount of data read from the upstream per second (In_R) is 0, the amount of data sent to the downstream per second (Out_R) is 202, the queue load percentage for receiving data (In_Q) is 0%, and the queue load percentage for sending data (Out_Q) is 0%; the busy time percentage Busy_Time of operator Bxxxxxxx1 is 1%, the idle time percentage Idel_Time is 24%, the memory percentage Mem is 10%, the number of data transactions processed per second (TPS) is 509, the amount of data received from the upstream operator or data source (In_W) is 789b, the amount of data sent to the downstream operator or data receiver (Out_W) is 789b, the amount of data read from the upstream per second (In_R) is 101, the amount of data sent to the downstream per second (Out_R) is 101, the queue load percentage for receiving data (In_Q) is 0%, and the queue load percentage for sending data (Out_Q) is 100%; the busy time percentage Busy_Time of operator Cxxxxxxx1 is 100%, the idle time percentage Idel_Time is 1%, the memory percentage Mem is 10%, the number of data transactions processed per second (TPS) is 291, the amount of data received from the upstream operator or data source (In_W) is 234b, the amount of data sent to the downstream operator or data receiver (Out_W) is 0, the amount of data read from the upstream per second (In_R) is 59, the amount of data sent to the downstream per second (Out_R) is 0, the queue load percentage for receiving data (In_Q) is 100%, and the queue load percentage for sending data (Out_Q) is 0%; the busy time percentage Busy_Time of operator Bxxxxxxx2 is 1%, the idle time percentage Idel_Time is 24%, the memory percentage Mem is 10%, the number of data transactions processed per second (TPS) is 509, the amount of data received from the upstream operator or data source (In_W) is 789b, the amount of data sent to the downstream operator or data receiver (Out_W) is 789b, the amount of data read from the upstream per second (In_R) is 101, the amount of data sent to the downstream per second (Out_R) is 101, the queue load percentage for receiving data (In_Q) is 0%, and the queue load percentage for sending data (Out_Q) is 100%;During the task execution process, the data output by operator Axxxxxx during task execution flows to operator Bxxxxxxx1 and operator Bxxxxxxx2 respectively. The data output by operator Bxxxxxxx1 during task execution flows to operator Cxxxxxxx1, and the data output by operator Bxxxxxxx2 during task execution also flows to operator Cxxxxxxx1. Through the above method provided by this embodiment, the flow and transformation of data between operators can be intuitively displayed.

[0059] Through the above steps, by collecting and storing the performance index data of operators in real time, an accurate playback view can be provided at any target moment requested by the user. It not only displays the static attribute information of the operators, but also clearly presents the flow direction and transformation process of data between the operators, greatly improving the efficiency and accuracy of data playback. At the same time, through the use of a circular buffer and data compression processing, the storage space is effectively saved, ensuring the stable operation of the system. On the user interaction interface, the target moment is selected by dragging the progress bar, and the operation is simple and intuitive, improving the user experience; the playback function at the operator level of the stream processing system can be realized.

[0060] Figure 6 It is an architecture diagram of a stream processing system provided according to an embodiment of the present application, as Figure 6 shown, the stream processing system includes: a work processing node TaskMnager, a log sampler (LogSampler) and a log collector (LogCollector) embedded in TaskMnager; as Figure 6As shown in the figure, each TaskManager in the stream processing system contains multiple operators (processorA, processorB, processorC). The TaskManager is used to provide resources for the multiple operators it contains. Each operator is a functional unit in the stream processing system. A log collector (LogSampler) is embedded in each operator, which is used to collect the performance metric data of each operator in real time and write the performance metric data into both the log collector (LogCollector) and the storage system connected to the stream processing system. In this embodiment, the log collector (LogCollector) is used to receive the performance metric data. The log collector (LogCollector) stores the received performance metric data in a circular buffer, and when the memory margin in the circular buffer is less than or equal to the preset memory margin, it compresses the data stored in the circular buffer. The log collector sends the compressed data file (i.e., the second type of file) composed of the results obtained from the compression process to the storage system, and the storage system uses the compressed data file to replace some of the pass-through files stored in the storage system. Among them, the pass-through file replaced by the compressed data file is a pass-through file containing the same metric data as the compressed data file. Both the pass-through file and the compressed data file contain the metric data collected from the operators in the stream processing system. Therefore, the files stored in the storage system can also be called metric files. The above circular buffer is an array with a fixed memory, and the memory margin is used to indicate the proportion of the free memory in the circular buffer relative to the total memory of the circular buffer (i.e., the free memory ratio). Still as Figure 6 shown, the stream processing system also includes an interactive interface (Flink Web UI) connected to the storage system. This interactive interface is used to receive the user's playback request (i.e., the request information) and display the playback view generated for the playback request.

[0061] Figure 7 is a schematic diagram of a display device for data playback provided according to an embodiment of the present application. As Figure 7 shown, the display device for data playback includes: a receiving module 70, which is used to receive the request information. Among them, the request information includes at least one target time, and the request information is used to request the playback of the operations performed by the stream processing system at the target time; a reading module 72, which is used to read the metric data corresponding to the target time from the storage system. Among them, the metric data includes: the performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; a generating module 74, which is used to generate a playback view according to the metric data. Among them, the playback view records the static attribute information of each operator at the target time and the dynamic interaction information between multiple operators.

[0062] When the display device for data playback executes the data playback display method provided in the embodiments of the present application, the receiving module 70 receives request information for requesting an operation to be performed by the playback stream processing system at a specified moment. Among them, the above-mentioned specified moment (i.e., the target moment) is included in the request information and can be selected by dragging the progress bar on the interactive interface (FlinkWebUI) or directly input by the user on the interactive interface. By detecting the user's dragging operation on the progress bar, it is detected whether the duration of the user's stop operation exceeds a preset duration (for example, 1 second). Once it is detected that the duration of the user's stop operation exceeds the preset duration, the receiving module 70 will record the time point corresponding to the current position of the progress bar as the target moment and construct request information, where the request information includes necessary parameters such as the target moment. After receiving the request information, the reading module 72 will read the metric data corresponding to the target moment from the storage system. These metric data include the performance metric data of each operator, such as processing speed, number of written data items, number of output data items, size of written data volume, size of output data volume, idle time, busy time, etc. The generating module 74 will process these metric data to generate a playback view according to the metric data. The playback view not only includes the static attribute information of each operator at the target moment, such as the type and configuration parameters of the operator, but also includes the dynamic interaction information between operators, such as data transfer rate, latency, and data queue length, etc. The performance of each operator in the stream processing system and the flow and transformation of data between various operators are simultaneously displayed in the playback view.

[0063] It should be noted that Figure 7 The preferred implementation manners of the illustrated embodiments can be referred to Figure 2 the relevant descriptions of the illustrated embodiments and will not be elaborated here.

[0064] The embodiments of the present application also provide a non-volatile storage medium, in which a computer program is stored. Among them, the device where the non-volatile storage medium is located executes the above data playback display method by running the computer program.

[0065] The above non-volatile storage medium is used to store a program for performing the following functions: receiving request information, where the request information includes at least one target moment, and the request information is used to request an operation to be performed by the playback stream processing system at the target moment; reading the metric data corresponding to the target moment from the storage system, where the metric data includes: the performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; generating a playback view according to the metric data, where the playback view records the static attribute information of each operator at the target moment and the dynamic interaction information between multiple operators.

[0066] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute the above-described method for displaying data playback through the computer program.

[0067] The processor in the above electronic device is used to run a program that performs the following functions: receiving request information, where the request information includes at least one target time, and the request information is used to request an operation to be performed by the playback stream processing system at the target time; reading metric data corresponding to the target time from a storage system, where the metric data includes: performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; generating a playback view based on the metric data, where the playback view records static attribute information of each operator at the target time and dynamic interaction information between multiple operators.

[0068] An embodiment of the present application further provides a computer program product, including computer instructions, which implement the steps of the above-described method for displaying data playback when executed by a processor.

[0069] It should be noted that each module in the above-described device for displaying data playback may be a program module (for example, a set of program instructions that implement a specific function), or a hardware module. For the latter, it may be presented in the following forms, but is not limited thereto: the presentation form of each of the above modules is a processor, or the functions of each of the above modules are implemented by a processor.

[0070] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0071] In the above embodiments of the present application, the descriptions of each embodiment have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0072] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0073] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0075] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the related technology, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks or optical discs that can store program codes.

[0076] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A display method for data playback, characterized in that, Including: Receiving request information, where the request information includes at least one target time, and the request information is used to request the replay stream processing system to perform an operation at the target time; Reading the metric data corresponding to the target time from the storage system, where the metric data includes: performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; Generating a replay view according to the metric data, where the replay view records the static attribute information of each operator at the target time and the dynamic interaction information among multiple operators.

2. The method according to claim 1, wherein Reading the metric data corresponding to the target time from the storage system, where the storage system stores data in the following manner: Receiving the real-time data generated by the stream processing system when performing an operation at the current time, and storing the real-time data as a first type of file, where the real-time data includes: real-time metric data, a first timestamp corresponding to the real-time metric data, and a metric name of the real-time metric data; When receiving a second type of file composed of compressed data, determining a target first type of file corresponding to the second type of file among multiple first type of files, and deleting the target first type of file, where the compressed data is generated by compressing historical data generated by the stream processing system when performing operations at multiple historical times, and the historical data includes: historical metric data, a second timestamp corresponding to the historical metric data, and a metric name of the historical metric data.

3. The method according to claim 2, wherein Determining the target first type of file corresponding to the second type of file includes: Determining the second timestamp included in the second type of file, where the second type of file includes multiple second timestamps; Determining a target first timestamp indicating the same time as the second timestamp among multiple first timestamps; Determining the first type of file including the target first timestamp as the target first type of file corresponding to the second type of file.

4. The method according to claim 2, wherein The compressed data is generated by the following method: Receiving the real-time data generated by the stream processing system when performing an operation at the current time, and storing the real-time data in a circular buffer, where the circular buffer is an array with a fixed memory; After each storage of the real-time data into the circular buffer, determining the memory margin of the circular buffer, where the memory margin is used to indicate the proportion of free memory in the circular buffer; When the memory margin is less than or equal to a preset memory margin, performing compression processing on the data stored in the circular buffer to obtain the compressed data.

5. The method according to claim 4, characterized in that, Storing the real-time data in the circular buffer includes: Determining the storage order according to the first timestamp included in the real-time data; Storing the real-time data according to the storage order, where when storing, the first timestamp and the metric name in the real-time data are stored as keys, and the real-time metric data is stored as a value.

6. The method according to claim 1, wherein Read the metric data corresponding to the target time from the storage system, where the metric data corresponding to the target time is determined by the following method: Determine a target file containing a target timestamp among multiple files stored in the storage system, where the target timestamp is a timestamp indicating the same time as the target time indication, and each file consists of a file header, an index block, a file tail, and a data block, and the index block records the time range corresponding to the file, and the data block is used to store the metric data; Query the target metric data having a key-value relationship with the target timestamp in the target file; Determine the target metric data as the metric data corresponding to the target time.

7. The method according to claim 1, wherein Generate a playback view according to the metric data, including: Generate a list for describing the performance of the operator according to the static attribute information; and, Generate a ray for indicating the data flow between multiple lists according to the dynamic interaction information, where the data flow is used to indicate the flow direction and conversion process of data between multiple operators.

8. The method according to claim 1, wherein The target time is determined by the following method: Detect the operations performed by the target object on the interaction interface; When it is detected that the operation is dragging the progress bar and the duration for which the target object stops the operation is greater than a preset duration, determine the moment selected by the target object in the progress bar as the target time.

9. A stream processing system, characterized in that, Include: A work processing node TaskMnager, a log collector and a log aggregator embedded in the TaskMnager, where The TaskMnager is used to provide resources for multiple operators, and each operator is a functional unit in the stream processing system; The log collector is used to collect the performance metric data of each operator in real time, and write the performance metric data into both the log aggregator and a storage system connected to the stream processing system; The log aggregator is used to receive the performance metric data, store the performance metric data in a circular buffer, and perform compression processing on the data stored in the circular buffer when the memory margin of the circular buffer is less than or equal to a preset memory margin, where the circular buffer is an array with a fixed memory, and the memory margin is used to indicate the proportion of free memory in the circular buffer.

10. A display device for data playback, characterized in that, Include: A receiving module, configured to receive request information, where the request information includes at least one target time, and the request information is used to request a playback of the operations performed by the stream processing system at the target time; A reading module, configured to read the metric data corresponding to the target time from the storage system, where the metric data includes: the performance metric data of each operator in the stream processing system, and each operator is a functional unit in the stream processing system; A generating module, configured to generate a playback view according to the metric data, where the playback view records the static attribute information of each operator at the target time and the dynamic interaction information between multiple operators.

11. A non-volatile storage medium, characterized in that, A computer program is stored in the non-volatile storage medium. Among them, on the device where the non-volatile storage medium is located, the display method of data playback described in any one of claims 1 to 8 is executed by running the computer program.

12. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to execute the display method of data playback described in any one of claims 1 to 8 through the computer program.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, the steps of the display method of data playback described in any one of claims 1 to 8 are implemented.

Images