Data pipeline heterogeneous operator integration system and method based on message middleware

Abstract

The invention relates to a data pipeline heterogeneous operator integration system and method based on message middleware, belongs to the technical field of big data computing, and is applied to a public security environment. The method comprises the following steps: 1, carrying out heterogeneous operator development and packaging to obtain heterogeneous operator subsets; 2, visually constructing a topological graph of a new data pipeline in a dragging operator mode; 3, loading the operator into a container to operate so as to carry out container isolation on a heterogeneous operator operation environment; 4, selecting a mainstream message middleware Kafka message with the maximum length of 10MB as a reference quantity; 5, when the communication message data volume between the heterogeneous operators is smaller than or equal to the reference volume and larger than 0, communication between the operators is conducted through the message middleware; and 6, when the communication message data volume between the heterogeneous operators is greater than the reference volume, performing communication between the operators by using the shared file. According to the method, the operation environment isolation of the heterogeneous operators is realized through the container technology, the communication of the heterogeneous operators is realized through the message middleware, and the integrated operation of the heterogeneous operators is realized.

Description

Technical Field

[0001] The present invention relates to a data pipeline heterogeneous operator integration system and method based on message middleware, belonging to the technical field of big data computing and applied in a public safety environment. Background Art

[0002] 1. In the field of public security, the current status of data pipeline operator integration based on message middleware is as follows:

[0003] In the era of digitalization and informatization, data management and application in the field of public security have become key issues. As an important link connecting data collection, storage, analysis and application, the technological development of data pipelines is of great significance to decision-making and emergency response in the field of public security.

[0004] Data pipeline is a high-level abstraction of the big data processing process. It abstracts the data processing process into a series of operators and their connection relationships. By shielding the underlying data processing details of the operators, general users without programming knowledge can construct data pipelines and run them on the cloud platform by splicing and configuring operators in the visual interface, thereby lowering the threshold for system use on the basis of fully utilizing the data processing capabilities of the underlying computing architecture. In addition, since operators can be used in multiple data pipelines, the degree of reuse of system components is also improved. In order to enable operators developed in different languages ​​and different units to run under the same architecture, it is necessary to define the development and packaging methods of operators and save the operators in a unified way. The creation and operation of data pipelines are carried out in the data pipeline operating environment, which provides users with a visual development interface for data pipelines and a data pipeline operating environment based on the cloud platform.

[0005] Container technology is a kernel-lightweight operating system layer virtualization technology that can isolate processes and resources. It is one of the basic supporting technologies of cloud computing technology. Container is a portable software unit that can combine applications and all their dependencies into a software package. The software package is not restricted by the underlying host operating system, so there is no need to build a complex environment, which simplifies the process from application development to deployment and provides an independent operating environment for the process. The main features of container technology are: ① Simple porting: build once, deploy anywhere, flexible migration, improve development efficiency, and reduce deployment workload; ② Lightweight: Compared with traditional virtual machine technology, container images are small in size and have a short startup time; ③ Security isolation: virtualize CPU, memory, storage, and network resources, and isolate resources between containers; ④ Elastic scaling: dynamically increase or decrease the number of container instances according to application load; ⑤ Version differentiation control: container image version control, which can track and compare historical versions. Container technology has the characteristics of lightweight, secure isolation, and elastic scaling, and is an ideal choice for building a data pipeline heterogeneous operator operating environment.

[0006] Message middleware is a supporting software system that provides synchronous or asynchronous, reliable message transmission for application systems in a network environment based on queue and message passing technology. Message middleware uses efficient and reliable message passing mechanisms for platform-independent data exchange and integrates distributed systems based on data communication. By providing message passing and message queuing models, inter-process communication can be achieved in a distributed environment, which is a feasible way for communication between data pipeline operators in a distributed operating environment. Through message middleware, asynchronous message communication can be carried out between different services, which can improve the throughput and concurrency performance of the system and avoid blocking and performance bottlenecks caused by synchronous calls. Message middleware can decouple the message sender and the message receiver, so that they can be developed and deployed independently without directly relying on each other's implementation details. Message middleware can set flow control. When the message flow is too large, the message sending can be suspended and continued when the flow decreases, so as to smoothly handle the flow peak.

[0007] Message middleware usually consists of three parts: producers, consumers, and message queues. Producers are responsible for sending messages to message queues. Consumers get messages from message queues and process them. Message queues, as a bridge for message delivery, are responsible for storing and managing messages. Message middleware usually uses the publish / subscribe model for message sending. Producers publish messages to a topic, and consumers can subscribe to the topic and receive messages from it. The message queue broadcasts the message published by the producer to all consumers who have subscribed to the topic, so a message can be processed by multiple consumers at the same time. When subscribing to a topic, consumers can also specify the group they belong to. Messages are broadcast in different groups of the topic, and unicast in the same group of the topic, that is, they are consumed by only one consumer in the group. Therefore, consumers in the same group can achieve load balancing. When the message queue in the group accumulates, you only need to increase the number of consumers in the group to increase the message processing capacity.

[0008] 2. At present, in the public safety environment, there are the following deficiencies:

[0009] Data pipeline operators can be developed based on different development languages, and there are differences in the dependent environments such as system libraries required at runtime. How to integrate heterogeneous operators to run under a unified data pipeline architecture and achieve real-time and efficient communication between operators has become a problem that needs to be solved urgently.

[0010] 3. Therefore, how to achieve the integrated operation of heterogeneous operators in data pipelines in the field of public security has become an urgent problem to be solved. Summary of the invention

[0011] The purpose of the present invention is to solve the technical problem of integrated communication of heterogeneous operators of data pipelines in the field of public security, and to propose a data pipeline heterogeneous operator integration system and method based on message middleware.

[0012] The objective of the present invention is achieved through the following technical solutions:

[0013] A data pipeline heterogeneous operator integration method based on message middleware of the present invention comprises the following steps:

[0014] Step 1: Develop and package heterogeneous operators according to the development and packaging process of data pipeline operators to obtain a set of heterogeneous operators;

[0015] Step 2: For the heterogeneous operator set in step 1, visually construct the topology of the new data pipeline by dragging operators in the data pipeline construction interface. The directed edges between operators in the topology represent the communication relationship between operators. The two parties of the end-to-end message communication between operators are defined as the message sender and the message receiver. Then the directed edges between operators in the topology are directed from the message sender to the message receiver.

[0016] Step 3: Load heterogeneous operators into containers for operation, and isolate the operating environments of heterogeneous operators by containers, thereby eliminating differences in development languages ​​and operating environments between heterogeneous operators.

[0017] Step 4: Use the maximum message length of the distributed publish-subscribe messaging middleware, that is, the maximum message length of the mainstream messaging middleware Kafka is 10MB as a reference;

[0018] Step 5: When the amount of communication message data between the heterogeneous operators described in step 2 is less than or equal to the reference amount and greater than 0, the message middleware is used to communicate between operators;

[0019] Step 5.1: The pipeline runtime engine creates a message middleware topic for communication;

[0020] Step 5.2: The message receiver subscribes to the message from the message middleware topic;

[0021] Step 5.3: The message sender sends a communication message to the message middleware topic;

[0022] Step 5.4: The message receiver receives the communication message from the message middleware topic, thereby completing the communication between heterogeneous operators;

[0023] Step 6: When the amount of communication message data between the heterogeneous operators described in step 2 is greater than the reference amount, the shared file is used for communication between operators;

[0024] Step 6.1: The message sender writes the message data into the shared file;

[0025] Step 6.2: The message sender sends the path of the shared file to the message receiver through the message middleware;

[0026] Step 6.3: The message receiver receives the path of the shared file and then reads the content of the shared file, thereby completing the communication between heterogeneous operators.

[0027] The present invention discloses a data pipeline heterogeneous operator integration system based on message middleware, which is used to implement the above method. The present invention discloses a data pipeline heterogeneous operator integration system based on message middleware, which includes a pipeline editing interface module, an operator library management interface module, a pipeline management interface module, a pipeline operation monitoring interface module, a pipeline management module, an operator management module, and a pipeline operation engine module.

[0028] The pipeline editing interface module is used to visualize the construction of data pipelines by dragging operators, which will serve as the input of the pipeline management module;

[0029] The operator library management interface module is used to provide an interface for registering, updating, deleting, and versioning operators, and will serve as the input of the operator management module;

[0030] The pipeline management interface module is used to provide an interface for deploying and managing pipelines, which will serve as input to the pipeline management module;

[0031] The pipeline operation monitoring interface module is used to monitor the pipeline operation status after the pipeline is deployed, and will serve as the interaction of the pipeline operation engine module;

[0032] The pipeline management module is used to interact with the pipeline management interface module and will serve as the input of the pipeline operation engine module;

[0033] The operator management module is used to register and publish digital objects formed by operators and output them to the digital object library, as well as update, delete and version management of operators;

[0034] The pipeline execution engine module is used to obtain the definitions of data pipelines and operators in the pipeline management module and the operator management module, create containers for operator execution, and load and call data pipeline operator data processing logic.

[0035] Beneficial effects:

[0036] The container technology isolates the operator operating environment and realizes the integrated operation of heterogeneous data pipeline operators developed in different languages. The message middleware technology is used to decouple the operator communication relationship. The operator only needs to send messages to the specified message middleware topic without specifying the receiving operator of the message. Therefore, the communication relationship between operators can be dynamically specified when the pipeline is started. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention;

[0038] Figure 2 It is a schematic diagram of inter-operator communication based on message middleware of the present invention;

[0039] Figure 3 It is the data pipeline creation and editing interface of the present invention;

[0040] Figure 4 It is the data pipeline deployment interface of the present invention;

[0041] Figure 5 It is the data pipeline operation monitoring interface of the present invention. DETAILED DESCRIPTION

[0042] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0043] Example

[0044] like Figure 1 As shown, a data pipeline heterogeneous operator integration method based on message middleware of the present invention is specifically implemented in the following steps:

[0045] Step 1: Develop and package heterogeneous operators according to the development and packaging process of data pipeline operators to obtain a set of heterogeneous operators;

[0046] In the embodiment, the data pipeline is defined by the operators that make up the data pipeline and the connection relationship between the operators. The data pipeline operator is a data processing algorithm implemented in a certain programming language and packaged into an executable module according to certain development and packaging specifications. The development process of the data pipeline operator is divided into: operator source code writing, operator executable module generation, operator local debugging, operator packaging and packaging; further,

[0047] (1) The operator source code must be written in accordance with the operator development specification, which defines the initialization function and message processing function interface and related data structures that the operator needs to implement, as well as the interface function used by the operator to send messages. Operators can be developed in programming languages ​​such as Java / Python / Javascript and implement the specified interface functions, including initialization functions, destruction functions, and message processing functions.

[0048] (2) Operator packaging and packaging must be carried out in accordance with the operator packaging specification, which defines the operator metadata content and format, the operator executable module compilation method, and the format and steps for packaging the operator metadata file, operator executable module, operator dependent dynamic library, configuration file and other files required for operator operation into an operator packaging package. The operator metadata file contains information such as operator type, operator name, operator development unit / time, operator entry class / file, operator image, etc.

[0049] Step 2: For the heterogeneous operator set in step 1, visually construct the topology of the new data pipeline by dragging operators in the data pipeline construction interface. The directed edges between operators in the topology represent the communication relationship between operators. The two parties of the end-to-end message communication between operators are defined as the message sender and the message receiver. Then the directed edges between operators in the topology are directed from the message sender to the message receiver.

[0050] Step 3: Load heterogeneous operators into containers for operation, and isolate the operating environments of heterogeneous operators by containers, thereby eliminating differences in development languages ​​and operating environments between heterogeneous operators.

[0051] In the embodiment, the data pipeline operator is developed by multiple cooperative units. There are differences in development language, operating environment, data requirements, etc. between the operators. It is necessary to solve the operator integration problems such as how to efficiently operate the heterogeneous operators that constitute the data pipeline, how to distribute communication, and how to share data. The advantage of container technology is that it can isolate the operating environment, dynamically allocate computing resources, and can be lightweight deployed and started. Therefore, it is the best technical approach to achieve distributed and efficient operation of heterogeneous operators. The specific implementation steps are as follows:

[0052] (1) The operator runtime environment is built in the form of an operator container image. The operating system and programming language runtime environment required for the operator are placed in the operator container image. The operator executable module, operator base program, runtime dependency library module, and data files required for operation are placed in a shared directory. The operator base program is provided by the data pipeline runtime environment and runs in the container. It is responsible for loading the operator and acting as an agent for the operator to communicate with other operators. The operator obtains input messages from the operator base program and sends output messages to the operator base program. Different languages ​​correspond to different operator base programs. The operator container image is stored in the container image library. Each language runtime version corresponds to a container image. For example, the Java language container image includes JDK8, JDK11, JDK13 and other versions, the Python language container image includes Python2.7, Python3.6, Python3.7 and other versions, and the JavaScript language container image includes Nodejs11, Nodejs13 and other versions.

[0053] (2) When the data pipeline runtime environment is started, for each operator in the data pipeline, according to the operator development language and runtime environment metadata, select the corresponding container image to create the operator runtime container. For example, for an operator developed in Java language and with JDK11 as the dependent environment, select the container image containing the JDK11 runtime environment to create the operator runtime container. The parameters for creating the operator runtime container include: ① shared directory mount path. Mount the shared directory to a specified directory in the file system in the container. ② entry program path. The operator base program is responsible for loading the operator and sets its path as the entry of the container. ③ environment variables. The operating parameters of the operator base program are passed in through the container environment variables. The operating parameters include: operator executable module path, message middleware server address, receiving message topic name, sending message topic name, operator running parameters, etc.

[0054] (3) The operator base program is set as the container entry, so the operator base program will be automatically started after the operating system in the container is successfully started. After the operator base program is started, it searches for and loads the operator executable module and its dependent modules according to the operator executable module path set in the environment variable. After loading successfully, it reads the operator running parameters in the environment variable, calls the operator initialization function, and passes the running parameters to the initialization function.

[0055] Step 4: Use the maximum message length of the distributed publish-subscribe messaging middleware, that is, the maximum message length of the mainstream messaging middleware Kafka is 10MB as a reference;

[0056] Step 5: When the amount of communication message data between the heterogeneous operators described in step 2 is less than or equal to the reference amount and greater than 0, the message middleware is used to communicate between operators;

[0057] Step 5.1: The pipeline runtime engine creates a message middleware topic for communication;

[0058] Step 5.2: The message receiver subscribes to the message from the message middleware topic;

[0059] Step 5.3: The message sender sends a communication message to the message middleware topic;

[0060] Step 5.4: The message receiver receives the communication message from the message middleware topic, thereby completing the communication between heterogeneous operators;

[0061] In the embodiment, Figure 2 As shown in the figure, the operators of the data pipeline run in different containers. When the length of the communication message between operators is less than the reference amount, the communication between operators is realized through the message middleware. The specific implementation steps are as follows:

[0062] (1) When the data pipeline operating environment is started, for each pair of communication relationships between operators A and B in the data pipeline, a topic T is created in the message middleware service, and the topic name T is passed as the message sending topic name parameter to the operator base program of the source operator A, and the topic name T is passed as the message receiving topic name parameter to the operator base program of the target operator B.

[0063] (2) After the base program of operator B is started, it subscribes to the message receiving topic T from the message middleware according to the message middleware server address and message receiving topic name parameters set in the environment variables. When operator A needs to send a message, it calls the message sending interface of the base program of operator A, and the base program of operator A sends the message to the topic T of the message middleware according to the sending message topic name parameter; since the base program of operator B has subscribed to the message of topic T, it will receive the message and call the message processing function of operator B, thereby realizing message communication between operators.

[0064] Step 6: When the amount of communication message data between the heterogeneous operators described in step 2 is greater than the reference amount, the shared file is used for communication between operators;

[0065] Step 6.1: The message sender writes the message data into the shared file;

[0066] Step 6.2: The message sender sends the path of the shared file to the message receiver through the message middleware;

[0067] Step 6.3: The message receiver receives the path of the shared file and then reads the content of the shared file, thereby completing the communication between heterogeneous operators.

[0068] In an embodiment, when the amount of message data that an operator needs to send is large, it may not be able to be sent through the message middleware. At this time, it is necessary to use a shared file method for message communication: first, a distributed file system needs to be deployed in the container running environment; when the operator running container is started, the directory of the distributed file system is mounted to the container file system; when operator A needs to send a large amount of data message to operator B, the data is first saved in a shared file of the distributed file system, and the shared file path is sent to operator B through the message middleware; when operator B receives the message, it reads the shared file content from the distributed file system, thereby realizing large amount of data message communication between operators.

[0069] The present invention discloses a data pipeline heterogeneous operator integration system based on message middleware, which is used to implement the above method. The present invention discloses a data pipeline heterogeneous operator integration system based on message middleware, which includes a pipeline editing interface module, an operator library management interface module, a pipeline management interface module, a pipeline operation monitoring interface module, a pipeline management module, an operator management module, and a pipeline operation engine module.

[0070] The pipeline editing interface module is used to visualize the construction of data pipelines by dragging operators, which will serve as the input of the pipeline management module;

[0071] The operator library management interface module is used to provide an interface for registering, updating, deleting, and versioning operators, and will serve as the input of the operator management module;

[0072] The pipeline management interface module is used to provide an interface for deploying and managing pipelines, which will serve as input to the pipeline management module;

[0073] The pipeline operation monitoring interface module is used to monitor the pipeline operation status after the pipeline is deployed, and will serve as the interaction of the pipeline operation engine module;

[0074] The pipeline management module is used to interact with the pipeline management interface module and will serve as the input of the pipeline operation engine module;

[0075] The operator management module is used to register and publish digital objects formed by operators and output them to the digital object library, as well as update, delete and version management of operators;

[0076] The pipeline execution engine module is used to obtain the definitions of data pipelines and operators in the pipeline management module and the operator management module, create containers for operator execution, and load and call data pipeline operator data processing logic.

[0077] To further illustrate the heterogeneous operator integrated communication in a public safety environment of the present invention, a visual interface is used for illustration below.

[0078] like Figure 3 The figure shows the data pipeline creation and editing interface of the present invention. In this interface, the left panel is a list of heterogeneous operators, and the right panel is a data pipeline topology diagram. The user can drag the operator icon from the left panel to the right panel and establish a connection relationship between the operator icons to construct a new data pipeline topology diagram. The directed edges between the operators in the topology diagram point from the message sender to the message receiver; click the "Save" button in the toolbar to save the newly created data pipeline; click the "Deploy" button in the toolbar to enter the data pipeline deployment interface.

[0079] like Figure 4 The data pipeline deployment interface of the present invention is shown. In this interface, the user can enter the name of the deployed data pipeline instance, select the deployment target cluster, and enter the data pipeline operation parameters in the pipeline parameter configuration panel; click the "OK" button to start the deployment of the data pipeline instance.

[0080] like Figure 5The data pipeline operation monitoring interface of the present invention is shown. In this interface, the topological diagram of the data pipeline is displayed in the upper panel. The operator name and operation status are displayed above the operator icon in the figure, and the total number of messages read and written by the operator is displayed below; the operation log of the data pipeline, the operator operation status and the operation result information are displayed in the panel below the interface.

[0081] The results show that the present invention realizes a data pipeline heterogeneous operator integration system and method based on message middleware.

Claims

1. A data pipeline heterogeneous operator integration method based on message middleware, characterized by: The following steps are included: Step 1: Develop and package heterogeneous operators according to the development and packaging process of data pipeline operators to obtain a set of heterogeneous operators; Step 2: For the heterogeneous operator set in step 1, visually construct the topology of the new data pipeline by dragging operators in the data pipeline construction interface. The directed edges between operators in the topology represent the communication relationship between operators. The two parties of the end-to-end message communication between operators are defined as the message sender and the message receiver. Then the directed edges between operators in the topology are directed from the message sender to the message receiver. Step 3: Load heterogeneous operators into containers for operation, and isolate the operating environments of heterogeneous operators by containers, thereby eliminating differences in development languages ​​and operating environments between heterogeneous operators. Step 4: Use the maximum message length of the distributed publish-subscribe messaging middleware, that is, the maximum message length of the mainstream messaging middleware Kafka is 10MB as a reference; Step 5: When the amount of communication message data between the heterogeneous operators described in step 2 is less than or equal to the reference amount and greater than 0, the message middleware is used to communicate between operators; Step 5.1: The pipeline runtime engine creates a message middleware topic for communication; Step 5.2: The message receiver subscribes to the message from the message middleware topic; Step 5.3: The message sender sends a communication message to the message middleware topic; Step 5.4: The message receiver receives the communication message from the message middleware topic, thereby completing the communication between heterogeneous operators; Step 6: When the amount of communication message data between the heterogeneous operators described in step 2 is greater than the reference amount, the shared file is used for communication between operators; Step 6.1: The message sender writes the message data into the shared file; Step 6.2: The message sender sends the path of the shared file to the message receiver through the message middleware; Step 6.3: The message receiver receives the path of the shared file and then reads the content of the shared file, thereby completing the communication between heterogeneous operators.

2. A data pipeline heterogeneous operator integration system based on message middleware that implements the method as claimed in claim 1, characterized in that: It includes pipeline editing interface module, operator library management interface module, pipeline management interface module, pipeline operation monitoring interface module, pipeline management module, operator management module, and pipeline operation engine module; The pipeline editing interface module is used to visualize the construction of data pipelines by dragging operators, which will serve as the input of the pipeline management module; The operator library management interface module is used to provide an interface for registering, updating, deleting, and versioning operators, and will serve as the input of the operator management module; The pipeline management interface module is used to provide an interface for deploying and managing pipelines, which will serve as input to the pipeline management module; The pipeline operation monitoring interface module is used to monitor the pipeline operation status after the pipeline is deployed, and will serve as the interaction of the pipeline operation engine module; The pipeline management module is used to interact with the pipeline management interface module and will serve as the input of the pipeline operation engine module; The operator management module is used to register and publish digital objects formed by operators and output them to the digital object library, as well as update, delete and version management of operators; The pipeline execution engine module is used to obtain the definitions of data pipelines and operators in the pipeline management module and the operator management module, create containers for operator execution, and load and call data pipeline operator data processing logic.

Images