State reading method and device of distributed stream processing framework, electronic equipment and storage medium

Abstract

The present disclosure provides a state reading method and device of a distributed stream processing framework, electronic equipment and storage medium, and relates to the technical field of data processing. The method comprises: finding class information of a class having a program entry method in task data of a stream processing task to obtain a class information set, and determining a starting class in the class information set; finding class information of all used classes in the class information set according to classes introduced by the starting class to obtain a class information subset; determining a state of a target operator in the class information subset; generating a reading task according to the type of the state, and running the reading task to read the state of the target operator. The present disclosure realizes the state reading of the target operator, so that the specific content in the read operator state can be displayed to facilitate the user to find out the root cause of the problem through the query.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for reading the state of a distributed stream processing framework. Background Technology

[0002] The stream processing task framework for handling large-scale data streams can process real-time data streams and batch data with low latency and high throughput. In addition, it can provide a powerful state management mechanism to save the intermediate state of the job and support stateful stream processing.

[0003] Stream processing task frameworks provide monitoring capabilities for jobs, tasks, and operators, and support viewing task execution status and conditions through a user interface. However, this user interface does not allow direct viewing of the specific details within the operator status. Nevertheless, the specific details of the operator status are helpful for quickly locating and resolving problems in a job. For example, if the status of an operator does not meet expectations, viewing the specific details can help identify the root cause of the problem. Therefore, how to enable viewing of the specific details of the operator status is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this disclosure is to provide a state reading method, apparatus, electronic device and storage medium for a distributed stream processing framework, which can specifically solve existing problems.

[0005] Based on the above objectives, in a first aspect, this disclosure proposes a state reading method for a distributed stream processing framework, comprising: searching for class information of classes with program entry methods in the task data of a stream processing task to obtain a set of class information, and determining the startup class in the set of class information; searching for class information of all used classes in the set of class information based on the classes introduced by the startup class to obtain a subset of class information; determining the state of a target operator in the subset of class information; generating a reading task based on the type of state, and running the reading task to read the state of the target operator.

[0006] Secondly, a state reading device for a distributed stream processing framework is also provided, comprising: a first search unit configured to search for class information of classes with program entry methods in the task data of a stream processing task, obtain a set of class information, and determine the startup class in the set of class information; a second search unit configured to search for class information of all used classes in the set of class information based on the classes introduced by the startup class, to obtain a subset of class information; a determination unit configured to determine the state of a target operator in the subset of class information; and a generation unit configured to generate a reading task based on the type of state and run the reading task to read the state of the target operator.

[0007] Thirdly, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the method of the first aspect.

[0008] Fourthly, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program being executed by a processor to implement the method of any one of the first aspects.

[0009] Fifthly, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor to implement any one of the first aspects.

[0010] In summary, this disclosure offers at least the following advantages: it enables the reading of the state of a target operator, thereby displaying the specific content of the read operator state, allowing users to find the root cause of the problem through querying. Furthermore, by using a startup class, this disclosure can filter potentially usable class information from a class information set to obtain a subset of class information, allowing for the targeted identification of stateful target operator classes. This enables the generation of state reading tasks for stateful target operator classes, facilitating subsequent querying and other uses of the target operator class's state. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this disclosure and should not be construed as limiting the scope of this disclosure.

[0012] Figure 1 A flowchart is shown of a state reading method for a distributed stream processing framework according to an embodiment of the present disclosure;

[0013] Figure 2 Another flowchart of a state reading method for a distributed stream processing framework according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram of a state reading device for a distributed stream processing framework according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram of a storage medium provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0017] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] The scenarios involved in the embodiments of this disclosure will be described below.

[0020] This disclosure relates to a visualized state query scenario. Visualized state query refers to the server querying the state of a target operator in a stream processing task using a pre-deployed state query framework. Based on the state type, a read task is generated to read the state of the target operator, and the read state can be sent to the client for visualization. Visualized state query can be applied, for example, to server-side user data processing. Currently, the mainstream state query framework is a distributed stream processing framework (e.g., the Flink framework).

[0021] Currently, there are no tools to visualize the status of query stream processing tasks. Therefore, if the status of an operator does not meet expectations, the user interface cannot display the specific content of the operator status, making it impossible for users to intuitively identify the cause of the problem. Thus, how to enable viewing the specific content of the operator status is a pressing technical problem that needs to be solved in this field.

[0022] In view of this, this disclosure provides a method for reading the state of a distributed stream processing framework. Using the technical solution of this disclosure, a subset of class information that may be used can be filtered from a set of class information through a startup class. This subset of class information is then used to specifically identify stateful target operator classes. Therefore, this disclosure can generate state reading tasks for stateful target operator classes, realizing the reading of the target operator's state and thus displaying the specific content of the read operator state.

[0023] Figure 1 A state reading method for the distributed stream processing framework of this disclosure is illustrated. This method can be used on a server-side application. In embodiments of this disclosure, the method includes:

[0024] Step S101: In the task data of the stream processing task, search for the class information of the class with the program entry method to obtain the class information set, and determine the startup class in the class information set.

[0025] In this embodiment, the execution entity of the state reading method of the distributed stream processing framework can search for the class information of the class containing the target entry method in the task data of the stream processing task. This class information can be various information related to the class.

[0026] Stream processing tasks can be diverse, such as Flink tasks. Flink is an open-source distributed stream processing framework used for processing and analyzing real-time data streams. The task data is at least one of the real-time data stream processing or analysis tasks. In this case, the program entry point method is the entry point method for the Flink program execution, such as the main method.

[0027] The aforementioned execution entity can obtain the class information of the startup classes in the set using various methods. For example, it can directly find the class information of a unique startup class. Alternatively, if the startup class includes elements not used to launch the target operator, the execution entity can determine the class information of the startup class based on a preset identifier, such as the package name. Furthermore, the execution entity can output the class information or identifier information of each startup class so that the user's terminal can display this class information or identifier information and receive the identifier information of the startup class selected by the user. The execution entity can also receive the identifier information of the class returned by the terminal and determine the corresponding class information.

[0028] Step S102: Based on the classes introduced by the startup class, search for the class information of the classes used in the class information set to obtain a subset of class information.

[0029] In this embodiment, the execution entity can search for the class information of all used classes in the class information set based on the classes introduced by the startup class. Specifically, if the startup class in the pre-determined class information set is content to be parsed, the startup class can be parsed first, and the used classes can be found. The used classes can refer to the various classes that are started using the startup class, that is, the various classes introduced by the startup class. In some cases, the found used classes can also refer to some classes among the classes introduced by the startup class, such as classes marked with a specific identifier.

[0030] Step S103: Determine the state of the target operator within the class information subset.

[0031] In this embodiment, the execution entity can search for the target operator, i.e., the state of the target operator, within the subset of class information. The target operator here can be, for example, a class selected by the user based on class information or identifier information, or a preset class.

[0032] The state of an operator refers to the data maintained in a stream processing job for computing and storing intermediate results. These states can be keyed states or operator states, and the stream processing job is responsible for managing their storage, updates, and fault recovery.

[0033] Step S104: Generate a read task based on the type of state, and run the read task to read the state of the target operator.

[0034] In this embodiment, the aforementioned execution entity can generate a task to read the state of the target operator based on the type of state, and then run the task. The task can instruct the execution entity to read the state of the target operator.

[0035] This embodiment enables the reading of the state of a target operator, thereby displaying the specific content of the read operator state, allowing users to find the root cause of the problem through querying. Furthermore, by using a startup class, this embodiment can filter potentially usable class information from the class information set to obtain a subset of class information, allowing for the targeted identification of stateful target operator classes. This enables the generation of state reading tasks for stateful target operator classes, facilitating subsequent querying and other uses of the target operator class's state.

[0036] In some optional implementations of any embodiment of this disclosure, determining the state of the target operator in a subset of class information includes: sending a subset of class information to the terminal so that the terminal displays the state of the classes in the subset of class information; and determining the state of the class to be queried received from the terminal as the state of the target operator.

[0037] In these optional implementations, the aforementioned execution entity can send a subset of class information to the terminal. In this way, the terminal will display the state of the class in the subset of class information, allowing the user to select the state to be queried from the displayed states, thereby ensuring that the state of a specific class is read in a targeted manner.

[0038] In some optional implementations of any embodiment of this disclosure, the above-mentioned determination of the state of the target operator includes: searching for a class of stateful operators that implements the first interface function, and searching for an attribute in the class of stateful operators that implements the second interface function; and determining the state of the target operator from the attribute search results.

[0039] In these optional implementations, the execution entity can use Java reflection to find the class of the stateful operator that implements the first interface, and then find the attribute in that class that implements the second interface, for example, by using Java reflection. State can include both classes and attributes.

[0040] For example, the aforementioned execution entity can use Java reflection to find the Class of a stateful operator that implements the RichFunction interface. Furthermore, it can use Java reflection to find the attribute within the stateful operator that implements the State interface. The Class that implements the RichFunction interface is the operator.

[0041] These implementations can leverage Java reflection to locate classes and attributes that implement specific interfaces, thus narrowing down the range of possible states for the target operator. This is particularly beneficial when the user selects which operator state to read, thereby reducing the user's choices and improving efficiency.

[0042] In some optional implementations of any embodiment of this disclosure, the method further includes: determining the type of state, creating an operator state table corresponding to the state based on the type; and running a read task to read the state of the target operator, including: running a read task to read the state from the operator state table.

[0043] In these optional implementations, the aforementioned execution entity can create an operator state table based on the type of the target operator's state. The reading process completed by the reading task is to read the state of the target operator from the operator state table.

[0044] For example, the aforementioned execution entity can determine the table structure of the operator state table based on the type of the target operator's state using a preset model and a corresponding relationship table.

[0045] These implementations can create table structures that closely match the state type. This allows for better recording and storage of operator states through the operator state table.

[0046] Optionally, a read task is generated based on the type of state, including: generating a state read operator, a state write operator for writing to the target database, and a state read startup class based on the type.

[0047] The aforementioned execution entity can generate state read operators, state write operators, and state read startup classes based on state types. The state write operator is a sink operator used for writing to the target database. Furthermore, a read startup class can also be generated based on the state type.

[0048] These optional implementations can dynamically generate state reading tasks based on the state of the target operator, thereby improving the targeting and accuracy of the reading tasks.

[0049] In some optional application scenarios, a read task is run to read the state from the operator state table, including: reading the operator by state and reading the state from the operator state table; the method also includes: writing the read state to a target database by writing the operator by state; the target database is used to output the state in a visual manner in response to a query command for the state.

[0050] The aforementioned execution entity can display on the display of this device or send it to other electronic devices when a user queries, so that the other electronic devices can display the status of the operator on their displays.

[0051] Queries can be automatic or triggered by the user. Users can enter query keywords to prompt the electronic device to query the status of the target operator in the SQL database and display it to the user.

[0052] These application scenarios can use dedicated operators for reading and writing to read and write the state, and present it to the user in a visual way so that the state is clear at a glance.

[0053] In some optional implementations of any embodiment of this disclosure, the above-mentioned creation of the operator state table corresponding to the state according to the type includes: if the type is an operator type, determining that the state table field of the operator state table corresponding to the state includes an attribute of the state value type; if the type is an input type, determining that the state table field of the operator state table corresponding to the state includes an attribute of the state value type and an attribute of the key type of the input keying stream.

[0054] In these implementations, the naming convention for the operator state table can be: Operator Class Name + Operator UID + State Field Name. If the state is of type Operator State, the state table fields only contain attributes of state value type. If the state is of type Keyed State, the table fields contain both attributes of key type for the input keyed stream and attributes of state value type.

[0055] These implementations refine the structure of the operator state table by using different state types, making the structure of the operator state table more consistent with the operator state, which helps to store information more comprehensively and accurately.

[0056] In some optional implementations of any embodiment of this disclosure, the stream processing task is a Flink task; the method further includes: in response to receiving a data packet of a Flink task uploaded by a terminal, parsing the data packet; and using the parsing result of the data packet as task data.

[0057] To facilitate understanding, the following examples illustrate the state reading method of the distributed stream processing framework disclosed herein.

[0058] Figure 2 This document illustrates a method for reading the state of a distributed stream processing framework according to an embodiment of this disclosure. In this method, the stream processing task can be a Flink task. The document addresses how to view the specific content of the operator state within the stream processing task in this scenario. Figure 2 As shown, the state reading method of this distributed stream processing framework may include:

[0059] 1. Configure the address information for the stream processing task:

[0060] (1) Configure the address information (e.g., SavePoint) of the stream processing task and test the connectivity between the execution entity and the device corresponding to the address information.

[0061] 2. Configure the target database:

[0062] (1) Select the type of target database;

[0063] (2) Configure the connection information of the target database and test the connectivity between the terminal and the target database.

[0064] It should be noted that the above-mentioned process of configuring the address information of the stream processing task and configuring the target database are preparatory work before executing the status query of this example. By configuring the type and connection information of the target database, data and program basis, or support for callable parameters, are provided for the execution of the Flink task in this disclosure.

[0065] Furthermore:

[0066] 3. Upload data packets for the streaming processing task:

[0067] (1) Upload the data packets (e.g., JAR files) for the streaming task;

[0068] (2) Based on the ASM database in the target database, parse the data packet to find the set of all classes with the main method. The ASM database is a Java library used to generate and transform Java bytecode.

[0069] (3) Display the task's startup class for the user to select.

[0070] It should be understood that the startup class (Class) is used to initiate streaming data tasks, and other classes can be introduced through the startup class. Therefore, the startup class can be used to narrow down the range of data used to determine the operator state. During the query process, the startup class can be filtered out from the class information set first, thereby reducing the amount of information read and improving the accuracy of the read.

[0071] 4. For selected stateful operators, visualize the state:

[0072] (1) Based on ASM parsing, the startup class is found in the class information set, and all the used class information is found to obtain a subset of class information;

[0073] (2) Based on Java reflection, search for a class (Class) that implements the RichFunction interface functionality (i.e., the first interface functionality in the above embodiments) from a subset of class information. The RichFunction interface functionality refers to rich function functionality. It allows users to access the runtime context within functions and defines the lifecycle methods of functions.

[0074] (3) Based on Java reflection, in the class Class of stateful operators, find the attribute in the operator that implements the State interface functionality (i.e., the second interface functionality in the above embodiment). The State interface functionality refers to the state interface functionality.

[0075] (4) Use the results found in (2) and (3) as the state of the target operator, so that the user can select the state to be used from the state of the target operator. In this way, the target operator to be processed in the stream processing task can be found through a specific interface.

[0076] 5. Create operator state table

[0077] (1) Determine the table structure of the operator state table according to the type of operator state to be used. The specific rules are as follows:

[0078] ① The naming convention for tables is: Operator Class Name + Operator UID + Status Field Name;

[0079] ② If the operator state is of type Operator State, the fields in the operator state table only contain attributes of the state value type;

[0080] ③ If the operator state is a Keyed State type state, the table fields contain both key type attributes of the input keyed stream and state value type attributes.

[0081] (2) Initialize the operator state table and then create the operator state table.

[0082] 6. State reading and structured storage:

[0083] (1) Based on the type of state, dynamically generate Java source code for the state reading task based on Flink-state-processor-api, which is used to execute the reading task in the above embodiment. The source code is an executable JAR file (JavaArchive). Flink-state-processor-api is Flink's state processing API, which is a set of interfaces and libraries that allow developers to create and manipulate states in Flink applications to achieve stateful stream processing.

[0084] ① Generate state reading operators based on state type;

[0085] ② Based on the state type, generate a state write operator, which is also the Sink operator of the target database;

[0086] ③ Based on the state type, read the startup class.

[0087] (2) Execute the above source code to run the reading task, read the state through the above state reading operator, and write the state to the target database table through the above state writing operator.

[0088] 7. Visual query of status:

[0089] (1) Input the status query SQL;

[0090] (2) Users can perform visual queries on the status of the target database table.

[0091] This embodiment uses a specific interface to find the target operator that needs to be processed in the stream processing task, and through stateful operators, finds the state with query value, thereby effectively filtering the state of the operator to reduce the amount of state to be read and improve the reading speed.

[0092] It should be understood that Figure 2 This is merely an exemplary state reading method according to an embodiment of this disclosure, and does not constitute a limitation on the state reading methods of the distributed stream processing framework of this disclosure. In other embodiments, for reading from other non-Flink tasks, this disclosure may include methods that are more advanced than those described above. Figure 2 The illustrated process may have more or fewer implementation steps, and may involve database type, program type, operator programming language, etc., all of which can be compared with... Figure 2 The examples shown may differ, but this is not a limitation.

[0093] In summary, this disclosure offers at least the following beneficial effects: it enables the reading of the state of a target operator, thereby displaying the specific content of the read operator state, allowing users to find the root cause of the problem through querying. Furthermore, by using a startup class, this disclosure can filter potentially usable class information from the class information set to obtain a subset of class information, allowing for the targeted identification of stateful target operator classes. This enables the generation of state reading tasks for stateful target operator classes, facilitating subsequent querying and other uses of the target operator class's state.

[0094] This disclosure provides a state reading device for a distributed stream processing framework. This device is used to execute the state reading method for the distributed stream processing framework described in the above embodiments, such as... Figure 3 As shown, the device includes: a first search unit 301, configured to search for class information of classes with program entry methods in the task data of a stream processing task, obtain a set of class information, and determine the startup class in the set of class information; a second search unit 302, configured to search for class information of all used classes in the set of class information based on the classes introduced by the startup class, to obtain a subset of class information; a determination unit 303, configured to determine the state of the target operator in the subset of class information; and a generation unit 304, configured to generate a reading task based on the type of state, and run the reading task to read the state of the target operator.

[0095] Optionally, the determining unit 303 is further configured to determine the state of the target operator in the class information subset as follows: sending the class information subset to the terminal so that the terminal displays the state of the classes in the class information subset; and determining the state of the class to be queried received from the terminal as the state of the target operator.

[0096] Optionally, the determining unit 303 is further configured to determine the state of the target operator in the following manner: find the class of stateful operators that implement the first interface function, and find the attribute in the class of stateful operators that implements the second interface function; determine the state of the target operator in the attribute search results.

[0097] Optionally, the device is also configured to: determine the type of state, and create an operator state table corresponding to the state based on the type; the generation unit 304 is further configured to execute a read task to read the state of the target operator in the following manner: execute a read task to read the state from the operator state table.

[0098] Optionally, generation unit 304 is further configured to perform the generation of read tasks based on the type of state as follows: based on the type, generate a state read operator, a state write operator for writing to the target database, and a state read initiation class.

[0099] Optionally, the generation unit 304 is further configured to perform a read operation task to read a state from an operator state table in the following manner: reading a state from an operator state table via a state read operator; the device is also configured to write the read state to a target database via a state write operator; the target database is used to output the state in a visual manner in response to a query instruction for the state.

[0100] Optionally, the device is further configured such that: if the type is operator type, the state table field of the operator state table corresponding to the determined state includes an attribute of the state value type; if the type is input type, the state table field of the operator state table corresponding to the determined state includes an attribute of the state value type and an attribute of the key type of the input keying stream.

[0101] The distributed stream processing framework state reading device and the distributed stream processing framework state reading method provided in the above embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0102] This disclosure also provides an electronic device corresponding to the state reading method of the distributed stream processing framework provided in the foregoing embodiments, for executing the state reading method of the distributed stream processing framework described above. This disclosure does not limit the scope of the embodiments.

[0103] Please refer to Figure 4 This illustrates a schematic diagram of an electronic device provided by some embodiments of the present disclosure. For example... Figure 4 As shown, the electronic device 40 includes: a processor 400, a memory 401, a bus 402 and a communication interface 403. The processor 400, the communication interface 403 and the memory 401 are connected through the bus 402. The memory 401 stores a computer program that can run on the processor 400. When the processor 400 runs the computer program, it executes the method provided in any of the foregoing embodiments of this disclosure.

[0104] The memory 401 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 403 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0105] Bus 402 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 401 is used to store programs. After receiving execution instructions, processor 400 executes the program. The state reading method of the distributed stream processing framework disclosed in any of the foregoing embodiments of this disclosure can be applied to processor 400, or implemented by processor 400.

[0106] The processor 400 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 400 or by instructions in software form. The processor 400 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 401. The processor 400 reads the information in memory 401 and, in conjunction with its hardware, completes the steps of the above method.

[0107] The electronic device provided in this disclosure and the state reading method of the distributed stream processing framework provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0108] This disclosure also provides a computer-readable storage medium corresponding to the state reading method of the distributed stream processing framework provided in the foregoing embodiments. Please refer to... Figure 5 The computer-readable storage medium shown is an optical disc 50, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the state reading method of the distributed stream processing framework provided in any of the aforementioned embodiments.

[0109] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0110] The computer-readable storage medium provided in the above embodiments of this disclosure and the state reading method of the distributed stream processing framework provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0111] It should be noted that:

[0112] In the foregoing text, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in this disclosure is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this disclosure.

[0114] The embodiments of this disclosure have been described above with reference to the accompanying drawings. These are merely specific implementations of this disclosure, but this disclosure is not limited to the specific implementations described above. The specific implementations described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this disclosure without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this disclosure.

Claims

1. A method for reading the state of a distributed stream processing framework, characterized in that, include: In the task data of the stream processing task, search for the class information of the class with the program entry method to obtain the class information set, and determine the startup class in the class information set; Based on the classes introduced by the startup class, the class information of the classes used is searched in the class information set to obtain a subset of class information; Within the aforementioned subset of information, determine the state of the target operator; A read task is generated based on the type of the state, and the read task is run to read the state of the target operator.

2. The method according to claim 1, characterized in that, Determining the state of the target operator within the subset of information includes: Send the subset of class information to the terminal so that the terminal can display the status of the classes in the subset of class information; The state of the class to be queried received from the terminal is determined as the state of the target operator.

3. The method according to claim 1, characterized in that, Determining the state of the target operator includes: Locate the class of the stateful operator that implements the first interface function, and then locate the attribute in the class of the stateful operator that implements the second interface function; The state of the target operator is determined from the attribute lookup results.

4. The method according to claim 1, characterized in that, The method further includes: Determine the type of the state, and create an operator state table corresponding to the state based on the type; Running the reading task to read the state of the target operator includes: Run the read task to read the state from the operator state table.

5. The method according to claim 4, characterized in that, The step of generating a read task based on the type of the state includes: Based on the type, generate a state read operator, a state write operator for writing to the target database, and a state read startup class.

6. The method according to claim 5, characterized in that, Running the read task to read the state from the operator state table includes: The state is read from the operator state table by reading the state; The method further includes: The read state is written to the target database using the state write operator; the target database is used to output the state in a visual manner in response to a query command for the state.

7. The method according to claim 4, characterized in that, The step of creating the operator state table corresponding to the state according to the type includes: If the type is an operator type, the state table field of the operator state table corresponding to the state includes the attribute of the state value type; If the type is an input type, the state table fields of the operator state table corresponding to the state include attributes of the state value type and attributes of the key type of the input keying stream.

8. A state reading device for a distributed stream processing framework, characterized in that, include: The first search unit is configured to search for class information of classes with program entry methods in the task data of the stream processing task, obtain a set of class information, and determine the startup class in the set of class information. The second search unit is configured to search for the class information of all used classes in the class information set based on the classes introduced by the startup class to obtain a subset of class information. The determining unit is configured to determine the state of the target operator within a subset of class information; The generation unit is configured to generate a reading task based on the type of the state, and run the reading task to read the state of the target operator.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-7.

Images