Agent-based Flink data flow recording method and system
Through a proxy-based method, real-time recording of Flink data flow is realized, which solves the flexibility and invasiveness of data flow recording in the prior art, provides customized listening and high flexibility data capture capabilities, significantly improving Flink Timeliness and maintenance of tasks.
Patent Information
- Application Number
- CN202411464497.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-01-03
AI Technical Summary
The existing Flink data stream recording method requires modification of the source code, which leads to high invasiveness and poor flexibility, making it impossible to view or record each piece of data in real time, and the compiled and packaged source code cannot be dynamically modified.
Adopt a proxy-based approach, by creating a maven project, adding packaged plug-ins and dependency toolkits, defining proxy classes and premain static methods, listening to the output of the Flink Connector connector, printing data using Java reflection and log4j, modifying the Flink configuration file to start Javaagent, implements real-time recording of data flows.
Break through the limitations of Flink monitoring indicators, customize monitoring methods, capture every piece of data, improve timeliness and flexibility, and does not need to change the Flink source code, independently develop and package, and is easy to maintain.
Smart Images
Figure CN119248615A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of Flink data management, and in particular to a Flink data stream recording method and system based on an agent. Background Art
[0002] Flink is a distributed processing engine and framework for stateful computation of bounded and unbounded data streams. It is used to process data, but does not store data itself. Flink does not officially provide relevant functions to view or print each piece of data in the data stream in real time. You can only view indicators such as the cumulative number of data inflows and outflows and instantaneous traffic through Flink's indicator module, but cannot obtain the specific content of each piece of data. In order to allow users to see each piece of data in the data stream in real time during the running of Flink tasks, or write data to a third-party log system in real time. The existing method is to modify the Flink source code and embed the record of each piece of data in the specific method of the output operator. In actual applications, this method is too invasive to the Flink source code, and secondary development will destroy the source code. The compiled and packaged Flink source code cannot be dynamically modified, and has poor flexibility.
[0003] Therefore, how to provide a more secure and flexible Flink data real-time recording method has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] In view of this, in order to overcome the deficiencies of the prior art, the present invention aims to provide a Flink data stream recording method and system based on an agent.
[0005] According to a first aspect of the present invention, a proxy-based Flink data stream recording method is provided, the method comprising: Create a Maven project, add packaging plug-ins and dependency toolkits to the created Maven project, and create a proxy class; Define the premain static method in the created proxy class, and write the monitoring business in the defined premain static method according to the dependent toolkit; Use the packaging plug-in to package the Maven project into a jar package, upload the jar package to the server, and configure the startup parameters for the Java virtual machine by modifying the Flink configuration file; Start Flink job submission according to the configured startup parameters, and monitor and print the output data of the Flink data stream during the Flink job running through the written monitoring business.
[0006] Preferably, in the proxy-based Flink data flow recording method of the present invention, a Maven project is created, a packaging plug-in and a dependency toolkit are added to the created Maven project, and a proxy class is created, including: creating a Maven project in a Java environment, adding a packaging plug-in and a dependency toolkit in the pom file of the created Maven project, configuring the name of the proxy class in the added packaging plug-in, creating a proxy class according to the name of the proxy class, and the proxy class is loaded when the Java virtual machine is started.
[0007] Preferably, in the proxy-based Flink data flow recording method of the present invention, a premain static method is defined in the created proxy class, and a monitoring service is written in the defined premain static method according to a dependency toolkit, including: Use the byte buddy dependency toolkit to enable monitoring of the interface class inherited by the Flink Connector output end; The writeRecord methods of all subclasses under the monitored interface class are intercepted, and the Java reflection method is used in the interception callback method to capture the current method and calling class, and the log4j dependency toolkit is used to print the parameters, calling time and class name of the captured method.
[0008] Preferably, in the proxy-based Flink data flow recording method of the present invention, the premain static method written with the monitoring service is executed when starting the Flink job submission.
[0009] Preferably, in the agent-based Flink data flow recording method of the present invention, a packaging plug-in is used to package the Maven project into a jar package, and the jar package is uploaded to the server, including: using the packaging plug-in to package the Maven project into a jar package through a packaging command, and uploading the jar package to the lib directory of the hadoop cluster.
[0010] Preferably, in the agent-based Flink data flow recording method of the present invention, the startup parameters are configured for the Java virtual machine by modifying the Flink configuration file, including: locating the Flink configuration file, retrieving the startup parameters in the Flink configuration file, when the startup parameters exist in the Flink configuration file, adding the jar package upload path to the parameter value of the startup parameter, and setting the Javaagent prefix before the added jar package upload path.
[0011] Preferably, in the agent-based Flink data flow recording method of the present invention, the startup parameters are configured for the Java virtual machine by modifying the Flink configuration file, including: when the startup parameters do not exist in the Flink configuration file, creating the startup parameters, adding the jar package upload path to the parameter value of the created startup parameters, and setting the Javaagent prefix before the added jar package upload path.
[0012] Preferably, in the proxy-based Flink data flow recording method of the present invention, Flink job submission is started according to the configured startup parameters, and the output data of the Flink data flow in the running of the Flink job is monitored and printed by the written monitoring business, including: Submit the Flink job to the cluster according to the configured startup parameters. The cluster allocates resources to load all jar packages associated with the Flink job. When the data generated during the running of the Flink job is written into the target table through the Flink connector, the output data of the Flink data stream during the running of the Flink job is monitored and printed through the written monitoring business.
[0013] Preferably, in the agent-based Flink data flow recording method of the present invention, Flink job submission is started according to the configured startup parameters, and the output data of the Flink data flow during the running of the Flink job is monitored and printed through the written monitoring business, and also includes: after the Flink job is submitted, the Flink data flow recording log is viewed through the Flink web management page, and the Flink data flow recording log includes detailed information and time information of the data printed by the monitoring business.
[0014] According to a second aspect of the present invention, a proxy-based Flink data flow recording system is provided, the system includes a recording server, and the recording server is used to: create a Maven project, add a packaging plug-in and a dependency toolkit to the created Maven project, and create a proxy class; define a premain static method in the created proxy class, and write a monitoring business in the defined premain static method according to the dependency toolkit; use the packaging plug-in to package the Maven project into a jar package, upload the jar package to the server, and configure the startup parameters for the Java virtual machine by modifying the Flink configuration file; start the Flink job submission according to the configured startup parameters, and monitor and print the output data of the Flink data flow in the running Flink job through the written monitoring business.
[0015] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect of the present invention when executing the program.
[0016] The proxy-based Flink data stream recording method and system of the present invention have the following beneficial technical effects: 1. Break through the indicator limitations of Flink monitoring, customize monitoring methods, capture each piece of data, repackage it after modification, replace the jar package of the original path, and it will take effect the next time you submit a Flink task, significantly improving timeliness and flexibility; 2. Without changing the Flink source code, you can monitor the general Flink connector. There is no need to perform secondary development on Flink. Independent development and packaging do not affect each other, making maintenance easy. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 This is an example diagram of the architecture of a proxy-based Flink data stream recording system according to an embodiment of the present invention; Figure 2 A flowchart of a method for recording Flink data streams based on an agent according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of the device provided by the present invention. DETAILED DESCRIPTION
[0019] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0020] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.
[0021] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0022] Figure 1 FIG. 4 is an example diagram of the architecture of a proxy-based Flink data stream recording system according to an embodiment of the present invention. Figure 1 As shown, the system may include a recording server 101, a communication network 102 and / or one or more recording clients 103. Figure 1 The example shown is a plurality of recording clients 103 .
[0023] The recording server 101 can be any suitable server for storing information, data, programs and / or any other suitable type of content. In some embodiments, the recording server 101 can perform appropriate functions. For example, in some embodiments, the recording server 101 can record Flink data streams based on agents. As an optional example, in some embodiments, the recording server 101 can be used to implement recording of Flink data streams based on agents by writing monitoring services. For example, the recording server 101 can be used to: create a Maven project, add a packaging plug-in and a dependency toolkit to the created Maven project, create a proxy class; define a premain static method in the created proxy class, and write a monitoring service in the defined premain static method according to the dependency toolkit; use a packaging plug-in to package the Maven project into a jar package, upload the jar package to the server, and configure the startup parameters for the Java virtual machine by modifying the Flink configuration file; start the Flink job submission according to the configured startup parameters, and monitor and print the output data of the Flink data stream in the running Flink job through the written monitoring service.
[0024] As another example, in some embodiments, the recording server 101 may send the proxy-based Flink data stream recording method to the recording client 103 for the user to use according to the request of the recording client 103 .
[0025] As an optional example, in some embodiments, the recording client 103 is used to provide a visual recording interface, which is used to receive a user's selection input operation for recording a Flink data stream based on a proxy, and, in response to the selection input operation, obtain a recording interface corresponding to the option selected by the selection input operation from the recording server 101 and display the recording interface, wherein the recording interface at least displays information about recording the Flink data stream based on the proxy and operation options for the information about recording the Flink data stream based on the proxy.
[0026] In some embodiments, the communication network 102 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 102 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The recording client 103 can be connected to the communication network 102 via one or more communication links (e.g., communication link 104), and the communication network 102 can be linked to the recording server 101 via one or more communication links (e.g., communication link 105). The communication link can be any communication link suitable for transmitting data between the recording client 103 and the recording server 101, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.
[0027] The recording client 103 may include any one or more clients that present an interface related to recording Flink data streams based on a proxy in an appropriate form for user use and operation. In some embodiments, the recording client 103 may include any suitable type of device. For example, in some embodiments, the recording client 103 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, and / or any other suitable type of client device.
[0028] Although the recording server 101 is illustrated as one device, in some embodiments, any suitable number of devices may be used to perform the functions performed by the recording server 101. For example, in some embodiments, multiple devices may be used to implement the functions performed by the recording server 101. Alternatively, the functions of the recording server 101 may be implemented using a cloud service.
[0029] Based on the above system, an embodiment of the present invention provides a proxy-based Flink data stream recording method, which is described below through the following embodiments.
[0030] Figure 2is a flowchart of a method for recording Flink data streams based on an agent according to an embodiment of the present invention. The method for recording Flink data streams based on an agent in this embodiment can be executed on a recording server, such as Figure 2 As shown, the proxy-based Flink data stream recording method includes the following steps: Step S201: Create a Maven project, add a packaging plug-in and dependency toolkit to the created Maven project, and create a proxy class.
[0031] As an optional example, this embodiment creates a Maven project in a Java environment. In this embodiment, Maven is a tool for managing and building Java projects. A packaging plug-in and a dependency toolkit are added to the pom file of the created Maven project, the name of the proxy class is configured in the added packaging plug-in, and a proxy class is created according to the name of the proxy class. The proxy class is loaded when the Java virtual machine is started.
[0032] For example, this embodiment creates a Maven project in a Java environment, and adds a "maven-assembly-plugin" packaging plug-in to the pom file of the created Maven project. The packaging plug-in is used to package the Maven project. The name of the proxy class is configured in the packaging plug-in, and the dependency toolkits "byte-buddy" and "log4j" are added to the pom file of the Maven project. The dependency toolkits are used to implement specific services in the proxy class. Byte-buddy is a Java bytecode generation and manipulation library that allows Java classes to be created and modified at runtime. It is used for dynamic proxy, bytecode enhancement, AOP (aspect-oriented programming) and other applications in Java applications. Log4j is an open source logging framework for checking Java applications and log management.
[0033] Step S202: define the premain static method in the created proxy class, and write the monitoring service in the defined premain static method according to the dependent toolkit.
[0034] As an optional example, this embodiment uses the byte buddy dependency toolkit to enable monitoring of the interface class inherited by the output end of the Flink Connector connector; intercepts the writeRecord method of all subclasses under the monitored interface class, uses the Java reflection method to capture the current method and calling class in the interception callback method, and uses the log4j dependency toolkit to print the parameters, calling time and class name of the captured method, where the method input parameter is the output data in the Flink data stream. It should be noted that in this embodiment, the premain static method written with the monitoring service is executed when the Flink job submission is started.
[0035] For example, this embodiment uses the "byte-buddy" dependency toolkit to enable monitoring of the "org.apache.flink.api.common.io.OutputFormat" interface class. All output classes of the Flink connector inherit this interface class. By monitoring this interface class, all implementation classes can be monitored. Secondly, the "writeRecord" method of all subclasses under this interface class is intercepted, and the current method and calling class are captured by the Java reflection method in the interception callback method. The "log4j" dependency toolkit is used to print the parameters, call time, class name, etc. of the captured method. The method input parameter is the output data in the Flink data stream, thereby completing the specific code implementation of the monitoring service.
[0036] Step S203: Use the packaging plug-in to package the Maven project into a jar package, upload the jar package to the server, and configure startup parameters for the Java virtual machine by modifying the Flink configuration file.
[0037] As an optional example, this embodiment uses a packaging plug-in to package the Maven project into a jar package through a packaging command, and uploads the jar package to the lib directory of the hadoop cluster. For example, this embodiment uses the "maven-assembly-plugin" packaging plug-in to package the Maven project into a jar package through the command "mvn clean package", and uploads the jar package to the lib directory of the hadoop cluster. The lib directory is the directory for storing the jar packages required for the flink job. The hadoop of this embodiment is an open source distributed computing platform under the Apache Software Foundation.
[0038] The configuration of the Fink job is stored in the "flink-conf.yaml" file. The "flink-conf.yaml" file is the main configuration file of the Flink framework, which contains various key parameters, including the operating environment and behavior of Flink. This embodiment needs to modify the "flink-conf.yaml" configuration file. This embodiment locates the Flink configuration file, retrieves the startup parameters in the Flink configuration file, and when the startup parameters exist in the Flink configuration file, adds the jar package upload path to the parameter value of the startup parameters, and sets the Javaagent prefix before the added jar package upload path. When the startup parameters do not exist in the Flink configuration file, create the startup parameters, add the jar package upload path to the parameter value of the created startup parameters, and set the Javaagent prefix before the added jar package upload path. This embodiment takes the flink-1.18 version and standalone mode as an example. In this scenario, find the "env.java.opts.all" parameter in the configuration file. If this parameter does not exist, create one; add the path of the jar package to the value of the "env.java.opts.all" parameter, and add the prefix "-javaagent:" before this path. Each time you submit a Flink job, this jar package will be submitted together, and the Java agent monitoring will be enabled.
[0039] Step S204: Start submitting the Flink job according to the configured startup parameters, and monitor and print the output data of the Flink data flow during the running of the Flink job through the programmed monitoring business.
[0040] As an optional example, this embodiment submits the Flink job to the cluster according to the configured startup parameters, and the cluster allocates resources to load all jar packages associated with the Flink job; when the data generated during the operation of the Flink job is written into the target table through the Flinkconnector connector, the output data of the Flink data stream during the operation of the Flink job is monitored and printed through the written monitoring business. After the Flink job is submitted, the Flink data stream record log is viewed through the Flink web management page. The Flink data stream record log includes detailed information and time information of the data printed by the monitoring business. In this embodiment, Flink web is a visual development platform based on web pages. For example, after the Flink job is submitted normally, by opening the Flink web management page, the task-manager page of the job is found in the management page, and the log is viewed on the task manager page. The log prints the details and time of each data.
[0041] Flink's general Connector inherits the "org.apache.flink.api.common.io.OutputFormat" class at the output end. This embodiment uses this as a monitoring point to intercept, print each piece of data or store it in the log system without changing the Flink source code. This code will be packaged into an independent jar package, modify the configuration items in the flink-conf.yaml configuration file, specify the startup parameters of the Java virtual machine, add the path of the jar package, submit the Flink job, and you can view the record of each piece of data at the output end on the task-manager management page of the Flink web.
[0042] The proxy-based Flink data stream recording method and system according to this embodiment have the following beneficial technical effects: 1. Break through the indicator limitations of Flink monitoring, customize monitoring methods, capture each piece of data, repackage it after modification, replace the jar package of the original path, and it will take effect the next time you submit a Flink task, significantly improving timeliness and flexibility; 2. Without changing the Flink source code, you can monitor the general Flink connector. There is no need to perform secondary development on Flink. Independent development and packaging do not affect each other, making maintenance easy.
[0043] like Figure 3 As shown, the present invention further provides a device, including a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 implements the above-mentioned agent-based Flink data flow recording method by running the executable computer program.
[0044] Among them, the computer program in the memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0045] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected based on actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0046] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.
[0047] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A proxy-based Flink data stream recording method, characterized in that: The method comprises: Create a Maven project, add packaging plug-ins and dependency toolkits to the created Maven project, and create a proxy class; Define the premain static method in the created proxy class, and write the monitoring business in the defined premain static method according to the dependent toolkit; Use the packaging plug-in to package the Maven project into a jar package, upload the jar package to the server, and configure the startup parameters for the Java virtual machine by modifying the Flink configuration file; Start Flink job submission according to the configured startup parameters, and monitor and print the output data of the Flink data stream during the Flink job running through the written monitoring business.
2. The proxy-based Flink data stream recording method according to claim 1, characterized in that: Create a Maven project, add a packaging plug-in and a dependency toolkit to the created Maven project, and create a proxy class, including: creating a Maven project in a Java environment, adding a packaging plug-in and a dependency toolkit to a pom file of the created Maven project, configuring the name of the proxy class in the added packaging plug-in, and creating a proxy class according to the name of the proxy class, wherein the proxy class is loaded when the Java virtual machine is started.
3. The proxy-based Flink data stream recording method according to claim 1, characterized in that: Define the premain static method in the created proxy class, and write the monitoring business in the defined premain static method according to the dependent toolkit, including: Use the byte buddy dependency toolkit to enable monitoring of the interface class inherited by the Flink Connector output end; The writeRecord methods of all subclasses under the monitored interface class are intercepted, and the Java reflection method is used in the interception callback method to capture the current method and calling class, and the log4j dependency toolkit is used to print the parameters, calling time and class name of the captured method.
4. The proxy-based Flink data stream recording method according to claim 1, characterized in that: The premain static method with monitoring business is executed when the Flink job is started and submitted.
5. The proxy-based Flink data stream recording method according to claim 1, characterized in that: The packaging plug-in is used to package the Maven project into a jar package, and the jar package is uploaded to the server, including: the packaging plug-in is used to package the Maven project into a jar package through a packaging command, and the jar package is uploaded to the lib directory of the hadoop cluster.
6. The proxy-based Flink data stream recording method according to claim 1, characterized in that: Configure startup parameters for the Java virtual machine by modifying the Flink configuration file, including: locating the Flink configuration file, retrieving the startup parameters in the Flink configuration file, adding the jar package upload path to the parameter value of the startup parameter when the startup parameter exists, and setting the Javaagent prefix before the added jar package upload path.
7. The proxy-based Flink data stream recording method according to claim 6, characterized in that: Configure startup parameters for the Java virtual machine by modifying the Flink configuration file, including: creating startup parameters when the startup parameters do not exist in the Flink configuration file, adding the jar package upload path to the parameter value of the created startup parameters, and setting the Javaagent prefix before the added jar package upload path.
8. The proxy-based Flink data stream recording method according to claim 1, characterized in that: Start the Flink job submission according to the configured startup parameters, and monitor and print the output data of the Flink data stream during the Flink job running through the written monitoring business, including: Submit the Flink job to the cluster according to the configured startup parameters. The cluster allocates resources to load all jar packages associated with the Flink job. When the data generated during the running of the Flink job is written into the target table through the Flink connector, the output data of the Flink data stream during the running of the Flink job is monitored and printed through the written monitoring business.
9. The proxy-based Flink data stream recording method according to claim 8, characterized in that: Start Flink job submission according to the configured startup parameters, monitor and print the output data of Flink data flow during Flink job running through the written monitoring business, and also include: after the Flink job is submitted, view the Flink data flow record log through the Flink web management page, and the Flink data flow record log includes the detailed information and time information of the data printed by the monitoring business.
10. A proxy-based Flink data stream recording system, characterized in that: The system includes a recording server, which is used to: create a Maven project, add a packaging plug-in and a dependency toolkit to the created Maven project, and create a proxy class; define a premain static method in the created proxy class, and write a monitoring business in the defined premain static method according to the dependency toolkit; use the packaging plug-in to package the Maven project into a jar package, upload the jar package to the server, and configure startup parameters for the Java virtual machine by modifying the Flink configuration file; start Flink job submission according to the configured startup parameters, and monitor and print the output data of the Flink data flow in the running of the Flink job through the written monitoring business.