A method and system for visualizing Flink status information
By analyzing the Flink checkpoint file, obtaining the status handle and selecting matching acquisition methods according to the type, the visualization of Flink state information is realized, solving the problems of low visualization and high usage threshold in the existing technology, and reducing user acquisition and understanding Difficulty of status information.
Patent Information
- Application Number
- CN202411098316.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-12
Smart Images

Figure CN118626348B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for visualizing Flink state information. Background Art
[0002] As a data processing engine in the big data field, Flink's checkpoint mechanism is the core function for realizing the fault tolerance of Flink tasks. It periodically generates a snapshot of each operator state of the Flink task, thereby regularly persisting and storing these state data to ensure that the Flink program can be restored through these snapshots when it crashes unexpectedly, thereby correcting the interruption of program data status caused by the failure.
[0003] However, the checkpoint file content persisted by the Flink task is encoded by Flink's internal custom serialization mechanism and is displayed as garbled code. Therefore, the file content cannot be directly viewed or understood, and the degree of visualization is also low.
[0004] To solve this problem, Flink introduced the State Processor API, which can parse checkpoint files to obtain their internal contents. The State Processor API allows developers to read, write, and modify the states in Flink savepoints and checkpoints. However, it also has certain limitations. It can only obtain values based on the metadata information of the provided checkpoint state, such as state name, state type, and serialization method. In actual scenarios, data warehouse personnel will not know what states are in the Flink program, and developers familiar with the Flink source code can only know the status of the current Flink task definition after viewing the source code. Therefore, the usage threshold is relatively high. Summary of the invention
[0005] The purpose of the present invention is to provide a method and system for visualizing Flink state information, so as to solve the problems in the prior art that the visualization level of Flink state information is low and the usage threshold is high.
[0006] To achieve the above objectives, this application adopts the following technical solutions:
[0007] A Flink state information visualization method of the present application includes the following steps:
[0008] Obtain a checkpoint file, and parse the checkpoint file to obtain a status handle;
[0009] Determine a corresponding state type according to the state handle, where the state type includes a first state type and a second state type;
[0010] Selecting a state metadata acquisition method that matches the state type according to the state type, and extracting metadata information of the state from the state handle based on the metadata acquisition method;
[0011] The status data is acquired from the checkpoint file according to the data reading method corresponding to the status type and the metadata information, and the status data is visually displayed.
[0012] Preferably, the first state type is an operator state, and the second state type is a keyed state.
[0013] Preferably, the step of parsing the checkpoint file to obtain a status handle comprises:
[0014] Load the checkpoint file using the state processing application program interface to obtain a CheckpointMetadata class for storing checkpoint metadata information;
[0015] Each state in the CheckpointMetadata class is traversed to obtain a state handle, where the state handle includes an operator state handle and a keyed state handle.
[0016] Preferably, selecting a state metadata acquisition method matching the state type according to the state type, and extracting metadata information of the state from the state handle based on the metadata acquisition method, includes:
[0017] When the state type is an operator state, the operator state handle is restored through BackendRestorerProcedure and an operator state backend class is constructed, in which a registeredOperatorStates field for storing operator state metadata information is provided;
[0018] The registeredOperatorStates field is used to obtain metadata information of the operator state.
[0019] Preferably, the acquiring the state data from the checkpoint file according to the data reading method corresponding to the state type and the metadata information includes:
[0020] Traversing metadata information of the operator state, the metadata information includes a state value type, and the state value type includes an operator list state, a joint list state, and a broadcast state;
[0021] Convert the metadata information into a state descriptor, and determine the interface in the operator state backend class to be called according to the state value type;
[0022] The state descriptor is passed into the interface to obtain an iterator corresponding to the state value type data, and the state data of the operator state in the checkpoint file is obtained by iterating according to the iterator.
[0023] Preferably, selecting a state metadata acquisition method matching the state type according to the state type, and extracting metadata information of the state from the state handle based on the metadata acquisition method, includes:
[0024] When the state type is a key-controlled state, the byte information of the key-controlled state is obtained by using the openInputStream method preset in the key-controlled state handle;
[0025] Deserializing the byte information to obtain metadata information of the keyed state, the metadata information including a key name and a serialization type;
[0026] Construct a streamOperatorStateContext interface according to the checkpoint metadata information and the serialization type, and obtain a keyed state backend through the streamOperatorStateContext interface;
[0027] The metadata information of the keyed state is extracted from the keyed state backend using a reflection mechanism.
[0028] Preferably, the acquiring the state data from the checkpoint file according to the data reading method corresponding to the state type and the metadata information includes:
[0029] Obtain a key iterator through the keyed state backend, and construct a state for storing a single-value type according to metadata information of the keyed state;
[0030] Iterate the key iterator to obtain the value of each key in the checkpoint file, and determine whether there is a next key;
[0031] When there is a next key, the current key is updated through the setCurrentKey interface preset in the keyed state backend, and the state data of the keyed state is determined according to the state for storing the single-value type.
[0032] A Flink status information visualization system, including:
[0033] A parsing module, used for obtaining a checkpoint file and parsing the checkpoint file to obtain a status handle;
[0034] A judgment module, used for determining a corresponding state type according to the state handle, wherein the state type includes a first state type and a second state type;
[0035] An extraction module, configured to select a state metadata acquisition method matching the state type according to the state type, and extract metadata information of the state from the state handle based on the metadata acquisition method;
[0036] A visualization module is used to obtain status data from the checkpoint file according to the data reading method corresponding to the status type and the metadata information, and to visualize the status data.
[0037] An electronic device includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a Flink state information visualization method as described in any one of the above.
[0038] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement a Flink state information visualization method as described in any one of the above when executed.
[0039] The present invention has the following beneficial effects:
[0040] The present invention can directly obtain metadata analysis of Operator State and Keyed State through checkpoint files, without requiring users to know what state information is available, thus lowering the usage threshold. At the same time, after information visualization, it can also help data warehouses and developers quickly and intuitively understand the status information of Flink tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a first flow chart of a Flink state information visualization method provided by an embodiment of the present invention;
[0042] Figure 2 is a second flow chart of a Flink state information visualization method provided by an embodiment of the present invention;
[0043] Figure 3 is a third flow chart of a Flink state information visualization method provided by an embodiment of the present invention;
[0044] Figure 4 It is a structural diagram of a Flink state information visualization system provided by an embodiment of the present invention;
[0045] Figure 5 It is a schematic diagram of an electronic device for implementing a method for visualizing Flink state information provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] To make the technical solution of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "first", "second", etc. in the claims and specification of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is only to describe the distinction method adopted when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that a process, method, system, product or device containing a series of units is not necessarily limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices.
[0047] Example 1
[0048] like Figure 1 As shown in FIG. 1 , a method for visualizing Flink state information includes the following steps:
[0049] S110, obtaining a checkpoint file, and parsing the checkpoint file to obtain a status handle;
[0050] S120, determining a corresponding state type according to the state handle, where the state type includes a first state type and a second state type;
[0051] S130, selecting a state metadata acquisition method that matches the state type, and extracting state metadata information from the state handle based on the metadata acquisition method;
[0052] S140. Acquire status data from the checkpoint file according to the data reading method and metadata information corresponding to the status type, and visualize the status data.
[0053] In Example 1, the automatically generated checkpoint file is first obtained from the execution environment of the Flink job. Then, the checkpoint file is parsed using the State Processor API to obtain the CheckpointMetadata class for storing checkpoint metadata information. Each state in the CheckpointMetadata class is traversed to obtain the state handle. In view of the fact that in Flink, the state is mainly divided into two types: operator state, i.e., the first state type, and keyed state, i.e., the second state type. Operator State means one state for one operator, and Keyed State means that different data keys correspond to different states, and the state handle corresponds to the state one by one. Therefore, the state handle obtained here may be either an operator state handle (OperatorStateHandle) or a keyed state handle (KeyedStateHandle). Then, the corresponding specific state type is determined according to the obtained State Handle, i.e., Operator State or Keyed State.
[0054] This embodiment also provides different metadata acquisition methods and data reading methods for different status types. Therefore, after determining the specific status type, a method matching the specific status type is selected from the provided metadata acquisition methods to extract metadata information. After obtaining the metadata information, a method matching the specific status type is selected from the provided data reading methods. The status data can be obtained from the checkpoint file in combination with the obtained metadata information. Finally, the obtained status data is displayed in a chart, graph or other intuitive form.
[0055] This embodiment directly reads and parses the checkpoint file of the Flink job, thereby overcoming the difficulty in the prior art in directly obtaining and visualizing Flink status information, and provides strong support for monitoring, debugging, and optimization of the Flink job.
[0056] Example 2
[0057] like Figure 2 As shown in the figure, a method for visualizing Flink status information includes:
[0058] S210, obtaining a checkpoint file, and parsing the checkpoint file to obtain a status handle;
[0059] S220, determining a corresponding state type according to the state handle, where the state type includes a first state type and a second state type, wherein the first state type is an operator state, and the second state type is a key-controlled state;
[0060] S230, when the state type is the operator state, restore the operator state handle through BackendRestorerProcedure and construct an operator state backend class, in which a registeredOperatorStates field for storing operator state metadata information is set;
[0061] S240. Use the registeredOperatorStates field to obtain metadata information of the operator state.
[0062] S250, traversing metadata information of operator states, where the metadata information includes state value types, wherein the state value types include operator list states, joint list states, and broadcast states;
[0063] S260, converting the metadata information into a state descriptor, and determining the interface in the operator state backend class to be called according to the state value type;
[0064] S270: The state descriptor is passed into the interface to obtain an iterator corresponding to the state value type data, and the state data of the operator state in the checkpoint file is obtained according to the iterator iteration.
[0065] In Example 2, when it is determined that the specific state type corresponding to the State Handle is Operator State, the State Handle is actually OperatorStateHandle. The OperatorStateHandle is first restored through BackendRestorerProcedure, because OperatorStateHandle provides the location and content of Operator State in persistent storage, such as it provides the name of each state of the Operator and the byte offset of its data in the file and the handle of the file, and then the operator state backend (OperatorStateBackend) class is constructed. Specifically, BackendRestorerProcedure provides a CreateAndRestore interface, which can construct an OperatorStateBackend class according to the information of OperatorStateHandle. A key field RegisteredOperatorStates is also set inside the OperatorStateBackend class. RegisteredOperatorStates internally stores the state metadata information stored in the file path pointed to by the current OperatorStateHandle. Therefore, RegisteredOperatorStates can be used to obtain the Operator. The metadata information of State includes the state name (StateName), serialization type (SerializerType) and state value type (StateType), among which StateType is divided into three types: operator list state (List State), union list state (Union List State) and broadcast state (Broadcast State). Then, traverse the obtained metadata information and convert it into Flink's state descriptor (StateDescriptor). According to its StateType, choose whether to call the interface getUnion ListState or getList State in the OperatorStateBackend class, and pass StateDescriptor as a parameter to the interface to get the iterator corresponding to the StateType data. Finally, iterate the iterator to obtain the state data of the operator state in the checkpoint file, and visualize the obtained state data.
[0066] This embodiment can directly obtain the metadata of the Operator State through the checkpoint file. The user does not need to know what state information is available, which lowers the usage threshold. At the same time, after the information is visualized, it can help data warehouses and developers quickly and intuitively understand the status information of Flink tasks.
[0067] Example 3
[0068] like Figure 3 As shown in the figure, a method for visualizing Flink status information includes:
[0069] S310, obtaining a checkpoint file, and parsing the checkpoint file to obtain a status handle;
[0070] S320, determining a corresponding state type according to the state handle, where the state type includes a first state type and a second state type, wherein the first state type is an operator state, and the second state type is a key-controlled state;
[0071] S330, when the state type is the key-controlled state, using the openInputStream method preset in the key-controlled state handle to obtain the byte information of the key-controlled state;
[0072] S340, deserialize the byte information to obtain metadata information of the keyed state, where the metadata information includes the name of the key and the serialization type;
[0073] S350, construct a streamOperatorStateContext interface according to the checkpoint metadata information and the serialization type, and obtain the keyed state backend through the streamOperatorStateContext interface;
[0074] S360, extracting metadata information of the keyed state from the keyed state backend using a reflection mechanism;
[0075] S370, obtaining a key iterator through the keyed state backend, and constructing a state for storing a single-value type according to metadata information of the keyed state;
[0076] S380, iterate the key iterator to obtain the value of each key in the checkpoint file, and determine whether there is a next key;
[0077] S390. When the next key exists, the current key is updated through the setCurrentKey interface preset in the keyed state backend, and the state data of the keyed state is determined according to the state used to store the single-value type.
[0078] In Example 3, when the specific state type corresponding to the State Backend is Keyed State, the StateHandle is actually KeyedStateHandle. The KeyedStateHandle holds the state byte stream handle, and the byte information can be obtained through the openInputStream method, and then the byte information is read and deserialized through the KeyedBackendSerializationProxy#read api to obtain the metadata information of the Keyed State. The metadata information includes the name of the key and the key serialization type. At this time, the KeyedStateBackend maintains all the Keyed State information in the checkpoint file. Then, the checkpoint metadata information and the key serialization type are passed into the StreamTaskStateInitializer to construct the streamOperatorStateContext interface. The StreamOperatorStateContext interface provides the KeyedStateBackend api, based on which the KeyedStateBackend can be directly obtained, and then the metadata information of the KeyedState can be obtained from the KeyedStateBackend through the java reflection mechanism.
[0079] Next, get the key iterator of the state through the KeyedStateBackend#getKeys api. KeyedStateBackend#getKeys can get a Stream object through the state metadata information, and the stream object provides an iterator interface. The Iterator interface is an important interface in the Java collection framework. It provides a method to traverse collections such as List and Set without understanding the internal structure of the collection. The iterator can be obtained through the iterator interface, and a state ValueState for storing single-value types is constructed according to the obtained Keyed State metadata information. The iterator is iterated to obtain the value of each key in the checkpoint file and determine whether there is a next key. If so, the current key is updated through KeyedStateBackend#setCurrentKey, and then the state data can be obtained through ValueState. Finally, the obtained state data is visualized.
[0080] This embodiment can directly obtain the metadata of Keyed State through the checkpoint file. The user does not need to know what state information is available, which lowers the usage threshold. At the same time, after the information is visualized, it can help data warehouses and developers quickly and intuitively understand the status information of Flink tasks.
[0081] Example 4
[0082] like Figure 4 As shown, a Flink state information visualization system includes a parsing module, a judgment module, an extraction module and a visualization module, wherein:
[0083] The parsing module is used to obtain the checkpoint file and parse the checkpoint file to obtain the status handle;
[0084] A judgment module, used to determine the corresponding state type according to the state handle, where the state type includes a first state type and a second state type;
[0085] An extraction module, used to select a state metadata acquisition method that matches the state type according to the state type, and extract the state metadata information from the state handle based on the metadata acquisition method;
[0086] The visualization module is used to obtain the status data from the checkpoint file according to the data reading method corresponding to the status type and the metadata information, and to visualize the status data.
[0087] The above system can execute the methods provided by all the aforementioned embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. For technical details not fully described in this embodiment, please refer to the methods provided by all the aforementioned embodiments of the present invention.
[0088] Example 5
[0089] like Figure 5 As shown, an electronic device includes a memory 501 and a processor 502, wherein the memory 501 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 502 to implement the above-mentioned Flink state information visualization method.
[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0091] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement a Flink state information visualization method as described above when executed.
[0092] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 501 and executed by the processor 502, and the input interface 505 and the output interface 506 complete the I / O interface transmission of data to complete the present invention. The one or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.
[0093] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a memory 501 and a processor 502. Those skilled in the art may understand that this embodiment is only an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include an input device 507, a network access device, a bus, etc.
[0094] The processor 502 may be a central processing unit (CPU), or other general-purpose processors 502, digital signal processors 502 (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 502 may be a microprocessor 502 or the processor 502 may also be any conventional processor 502, etc.
[0095] The memory 501 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The memory 501 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 501 may also include both an internal storage unit of the computer device and an external storage device. The memory 501 is used to store computer programs and other programs and data required by the computer device. The memory 501 may also be used to temporarily store in the output device 508, and the aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory ROM503, a random access memory RAM504, a disk or an optical disk.
[0096] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A method for visualizing Flink state information, characterized in that: The following steps are involved: Obtain a checkpoint file, and parse the checkpoint file to obtain a status handle; Determine a corresponding state type according to the state handle, the state type including a first state type and a second state type, the first state type is an operator state, and the second state type is a keyed state; Selecting a state metadata acquisition method that matches the state type according to the state type, and extracting metadata information of the state from the state handle based on the metadata acquisition method, including: When the state type is an operator state, the operator state handle is restored through BackendRestorerProcedure and an operator state backend class is constructed, in which a registeredOperatorStates field for storing operator state metadata information is provided; Use the registeredOperatorStates field to obtain metadata information of the operator state; Acquire status data from the checkpoint file according to the data reading method corresponding to the status type and the metadata information, and visualize the status data; Wherein, obtaining the state data from the checkpoint file according to the data reading method corresponding to the state type and the metadata information includes: Traversing metadata information of the operator state, the metadata information includes a state value type, and the state value type includes an operator list state, a joint list state, and a broadcast state; Convert the metadata information into a state descriptor, and determine the interface in the operator state backend class to be called according to the state value type; The state descriptor is passed into the interface to obtain an iterator corresponding to the state value type data, and the state data of the operator state in the checkpoint file is obtained by iteration according to the iterator.
2. A method for visualizing Flink state information according to claim 1, characterized in that: The step of parsing the checkpoint file to obtain a status handle includes: Load the checkpoint file using the state processing application program interface to obtain a CheckpointMetadata class for storing checkpoint metadata information; Each state in the CheckpointMetadata class is traversed to obtain a state handle, where the state handle includes an operator state handle and a keyed state handle.
3. A method for visualizing Flink state information according to claim 2, characterized in that: The step of selecting a state metadata acquisition method matching the state type according to the state type, and extracting metadata information of the state from the state handle based on the metadata acquisition method includes: When the state type is a key-controlled state, the byte information of the key-controlled state is obtained by using the openInputStream method preset in the key-controlled state handle; Deserializing the byte information to obtain metadata information of the keyed state, the metadata information including a key name and a serialization type; Construct a streamOperatorStateContext interface according to the checkpoint metadata information and the serialization type, and obtain a keyed state backend through the streamOperatorStateContext interface; The metadata information of the keyed state is extracted from the keyed state backend using a reflection mechanism.
4. A method for visualizing Flink state information according to claim 3, characterized in that: The acquiring the state data from the checkpoint file according to the data reading method corresponding to the state type and the metadata information includes: Obtain a key iterator through the keyed state backend, and construct a state for storing a single-value type according to metadata information of the keyed state; Iterate the key iterator to obtain the value of each key in the checkpoint file, and determine whether there is a next key; When there is a next key, the current key is updated through the setCurrentKey interface preset in the keyed state backend, and the state data of the keyed state is determined according to the state for storing the single-value type.
5. A Flink state information visualization system, characterized in that: include: A parsing module, used for obtaining a checkpoint file and parsing the checkpoint file to obtain a status handle; A judgment module, used for determining a corresponding state type according to the state handle, wherein the state type includes a first state type and a second state type, wherein the first state type is an operator state and the second state type is a key-controlled state; An extraction module, used to select a state metadata acquisition method matching the state type according to the state type, and extract metadata information of the state from the state handle based on the metadata acquisition method, including: When the state type is an operator state, the operator state handle is restored through BackendRestorerProcedure and an operator state backend class is constructed, in which a registeredOperatorStates field for storing operator state metadata information is provided; Use the registeredOperatorStates field to obtain metadata information of the operator state; A visualization module, used to obtain status data from the checkpoint file according to the data reading method corresponding to the status type and the metadata information, and to visualize the status data; Wherein, obtaining the state data from the checkpoint file according to the data reading method corresponding to the state type and the metadata information includes: Traversing metadata information of the operator state, the metadata information includes a state value type, and the state value type includes an operator list state, a joint list state, and a broadcast state; Convert the metadata information into a state descriptor, and determine the interface in the operator state backend class to be called according to the state value type; The state descriptor is passed into the interface to obtain an iterator corresponding to the state value type data, and the state data of the operator state in the checkpoint file is obtained by iteration according to the iterator.
6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a Flink state information visualization method according to any one of claims 1 to 4.
7. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to implement a Flink state information visualization method according to any one of claims 1 to 4 when executed.
