Plug-in type data communication middleware design implementation method based on ZeroMQ

Abstract

The invention discloses a plug-in type data communication middleware design implementation method and device based on ZeroMQ, and the method comprises the steps: constructing a ZeroMQ asynchronous communication core through an asynchronous I / O multiplexing and message serialization optimization algorithm, and achieving the direct communication of decentralized nodes; a plug-in management framework is constructed by a plug-in dependency analysis and sandbox isolation algorithm, and module modularization and dynamic expansion are supported; building a visual protocol adaptation interface by adopting an intelligent field mapping and real-time conversion preview algorithm, generating conversion rules and scripts, packaging the conversion rules and scripts into adaptation plug-ins, and loading the adaptation plug-ins to a communication core; and integrating a script engine, realizing user-defined logic and conversion rule collaboration through security sandbox execution and a dynamic compiling optimization algorithm, and generating a modular plug-in which is dynamically plugged into a communication core. According to the method, the problems that a traditional integrated interface is tedious, high in coupling and difficult to adapt can be solved, and rapid seamless integration of the heterogeneous system is achieved.

Description

Technical Field

[0001] This application relates to the field of distributed system communication technology, and in particular to a design and implementation method and apparatus for a pluggable data communication middleware based on ZeroMQ. Background Technology

[0002] When integrating or expanding existing information systems, the primary approach is point-to-point direct interface development, which presents the following technical problems: High interface development workload: A dedicated interface needs to be developed for each new system, resulting in high code duplication; Complex protocol adaptation: Different systems use different data formats and communication protocols, making adaptation cumbersome; High system coupling: Interface changes have far-reaching consequences, impacting system stability; Difficult maintenance: The number of interfaces grows exponentially with system expansion, leading to high maintenance costs; Performance bottlenecks: Traditional communication methods struggle to meet the demands of high-concurrency, low-latency real-time data exchange. Existing middleware technologies have the following limitations: Existing message middleware such as ActiveMQ and RabbitMQ suffer from the following shortcomings: Centralized architecture: Reliance on dedicated message broker servers poses a single point of failure risk; Limited protocol support: Insufficient support for diverse industrial and military-specific protocols; Complex configuration: Cumbersome and inflexible deployment and configuration processes; Performance limitations: Insufficient throughput and high latency in high-concurrency scenarios. Summary of the Invention

[0003] The main purpose of this application is to provide a design and implementation method and apparatus for a pluggable data communication middleware based on ZeroMQ, which aims to achieve rapid and seamless integration of heterogeneous systems through a visual protocol adaptation and dynamic pluggable loading mechanism.

[0004] To achieve the above objectives, this application provides a design and implementation method for a pluggable data communication middleware based on ZeroMQ, including: In distributed heterogeneous systems: An asynchronous communication core based on ZeroMQ is constructed using asynchronous I / O multiplexing algorithms and message serialization optimization algorithms to enable direct communication between decentralized heterogeneous node systems. A plug-in management framework is built based on plug-in dependency resolution algorithm and plug-in sandbox isolation algorithm to modularize, pluggable and dynamically expand intermediate communication plug-ins; A visual protocol adaptation interface is constructed using an intelligent field mapping algorithm and a real-time conversion preview algorithm. The graphical configuration is then used to generate protocol conversion rules and adaptation scripts through the visual protocol adaptation interface. The protocol conversion rules and adaptation scripts are encapsulated as adaptation plugins through a plug-in management framework and loaded into the ZeroMQ communication core. A script engine integrating a secure sandbox execution algorithm and a dynamic compilation optimization algorithm is used to write, verify, and execute custom protocol conversion logic. The custom logic and the protocol conversion rules of the visual protocol adaptation interface work together to generate modular communication plugins. These modular intermediate plugins are then dynamically plugged into the ZeroMQ communication core to achieve rapid and seamless integration of heterogeneous systems.

[0005] Optionally, the execution steps of the asynchronous I / O multiplexing algorithm include: Create a poller object to monitor I / O events for multiple sockets; Register the sockets that need to be listened to into the poller and monitor the POLLIN event; In the loop, wait for the event to occur using the poller's poll method and set a preset timeout. When an event occurs, iterate through all ready sockets. If it is a POLLIN event, receive the message in a non-blocking manner and put it into the message queue. Process messages in the message queue.

[0006] Optionally, the execution steps of the message serialization optimization algorithm include: Choose Protocol Buffers, MessagePack, or JSON serialization method based on the configuration; If Protocol Buffers is selected, message serialization is performed using pre-compiled proto structures; If MessagePack is selected, the MessagePack library will be called to perform message serialization; If JSON is selected, the JSON library will be used for message serialization.

[0007] Optionally, the plugin dependency resolution algorithm is a topological sorting algorithm, whose execution steps include: Construct a dependency graph with each intermediate plugin as a node and the dependency relationship as a directed edge, with the edge pointing from the dependent party to the dependent party; Calculate the in-degree of each node; Add nodes with an in-degree of 0 to the queue; Remove a node from the queue and add it to the loading order list. Decrement the in-degree of all its neighboring nodes by 1. If the in-degree of a neighboring node becomes 0, add it to the queue. Repeat the above steps until the queue is empty. If the length of the loading order list is equal to the number of nodes, the loading order is determined; otherwise, a circular dependency exception is thrown.

[0008] Optionally, the execution steps of the plug-in sandbox isolation algorithm include: Create a sandbox environment with restricted builtins and security APIs, and set resource limits for the sandbox environment related to memory, execution time, and file descriptors; Compile the plugin code and perform syntax checks to obtain the compiled code; The compiled code is executed in a sandbox environment with a timeout set; if a timeout or memory overflow exception occurs during execution, an exception is thrown.

[0009] Optionally, the execution steps of the intelligent field mapping algorithm include: Iterate through each field in the target protocol and all fields in the source protocol; Calculate the name similarity and type compatibility between the source and target fields. The name similarity is calculated using a weighted average of edit distance and word vector similarity. Type compatibility is determined based on a pre-defined type conversion table. A total score is obtained by combining name similarity and type compatibility. The source field with the highest score is selected as the mapping and the confidence level is recorded.

[0010] Optionally, the execution steps of the real-time conversion preview algorithm include: Obtain the first 10 source data samples. For each sample, perform type conversion and verification field by field according to the currently configured mapping rules and conversion script. Show the user the transformed data.

[0011] Optionally, the execution steps of the security sandbox execution algorithm include: Create a secure global environment that restricts built-in functions and modules; Remove dangerous functions such as __import__, open, and exec; Inject input data and security APIs into the global environment; Compile the script code and perform syntax analysis; Set an execution timeout to execute the compiled code in a sandbox environment; capture exceptions during execution and return the execution result or error information.

[0012] Optionally, the execution steps of the dynamic compilation optimization algorithm include: Choose the appropriate compiler based on the scripting language; Perform syntax analysis on the script to generate an abstract syntax tree (AST); optimize the AST by constant folding, dead code elimination, and inlining. Compile the optimized AST into bytecode or machine code; If the bytecode or machine code supports just-in-time compilation, then further optimization is performed from scratch.

[0013] Furthermore, to achieve the above objectives, this application also provides a design and implementation apparatus for a pluggable data communication middleware based on ZeroMQ, comprising: Applications of distributed heterogeneous systems: The asynchronous core building block is used to construct an asynchronous communication core based on ZeroMQ, using asynchronous I / O multiplexing algorithms and message serialization optimization algorithms, so as to enable direct communication between decentralized heterogeneous node systems; The plugin management module is used to build a plugin-based management framework based on plugin dependency resolution algorithms and plugin sandbox isolation algorithms, so as to modularize, pluggable, and dynamically extend intermediate communication plugins. The protocol visualization adaptation module is used to build a visual protocol adaptation interface using intelligent field mapping algorithm and real-time conversion preview algorithm. Through the visual protocol adaptation interface, the graphical configuration is generated into protocol conversion rules and adaptation scripts. The protocol conversion rules and adaptation scripts are encapsulated as adaptation plugins through a plug-in management framework and loaded into the ZeroMQ communication core. The script integration module is used to integrate the script engine with the security sandbox execution algorithm and dynamic compilation optimization algorithm, and to write, verify and execute custom protocol conversion logic through the script engine; The custom logic and the protocol conversion rules of the visual protocol adaptation interface work together to generate modular communication plugins. These modular intermediate plugins are then dynamically plugged into the ZeroMQ communication core to achieve rapid and seamless integration of heterogeneous systems.

[0014] This application proposes a design and implementation method and apparatus for a pluggable data communication middleware based on ZeroMQ. The method includes constructing an asynchronous communication core based on ZeroMQ in a distributed heterogeneous system using asynchronous I / O multiplexing algorithms and message serialization optimization algorithms to enable direct communication between decentralized heterogeneous systems; constructing a pluggable management framework based on pluggable dependency resolution algorithms and pluggable sandbox isolation algorithms to modularize, pluggable, and dynamically extend intermediate communication pluggear; constructing a visual protocol adaptation interface using intelligent field mapping algorithms and real-time conversion preview algorithms, generating protocol conversion rules and adaptation scripts from graphical configurations through the visual protocol adaptation interface; wherein the protocol conversion rules and adaptation scripts are encapsulated as adaptation pluggears through the pluggable management framework and loaded into the ZeroMQ communication core; integrating a script engine using secure sandbox execution algorithms and dynamic compilation optimization algorithms, writing, verifying, and executing custom protocol conversion logic through the script engine; wherein the custom logic and the protocol conversion rules of the visual protocol adaptation interface are executed collaboratively to generate modular communication pluggears, and the modular intermediate pluggears are dynamically plugged into the ZeroMQ communication core to achieve rapid and seamless integration of heterogeneous systems. This application addresses the problems of high workload, complex protocol adaptation, and high system coupling in traditional system integration by constructing a high-performance ZeroMQ communication core, designing a pluggable management framework, providing a visual protocol adaptation interface, and an integration script engine. This middleware supports multiple communication modes, features high throughput and low latency, and achieves flexible system expansion and rapid integration through its pluggable architecture. It is particularly suitable for scenarios requiring high concurrency and real-time data communication, such as military information systems and industrial automation. Attached Figure Description

[0015] Figure 1 This application provides a flowchart of a method for designing and implementing a pluggable data communication middleware based on ZeroMQ, as an example. Figure 2 This is a schematic diagram of the code implementation of the protocol adapter designer class provided in an embodiment of the pluggable data communication middleware design and implementation method based on ZeroMQ in this application. Figure 3 This is a schematic diagram of the code implementation of the script engine class provided in an embodiment of the pluggable data communication middleware design and implementation method based on ZeroMQ in this application. Figure 4 This is a schematic diagram of the code implementation of the radar data adaptation plugin provided in an embodiment of the plug-in data communication middleware design and implementation method based on ZeroMQ in this application. Figure 5 This is a schematic diagram of the code implementation of the command processing plugin provided in one embodiment of the pluggable data communication middleware design and implementation method based on ZeroMQ in this application.

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0018] Reference Figure 1 The first embodiment of this application provides a design and implementation method for a pluggable data communication middleware based on ZeroMQ. This method may include: S101. In a distributed heterogeneous system, an asynchronous I / O multiplexing algorithm and a message serialization optimization algorithm are used to build an asynchronous communication core based on ZeroMQ, enabling direct communication between decentralized heterogeneous systems. In one embodiment of this application, the execution steps of the asynchronous I / O multiplexing algorithm may include: creating a poller object to monitor I / O events of multiple sockets; registering the sockets to be monitored into the poller and paying attention to the POLLIN event; waiting for the event to occur in a loop using the poller's poll method and setting a preset timeout; when an event occurs, iterating through all ready sockets, and if it is a POLLIN event, receiving the message non-blockingly and putting it into a message queue; and processing the messages in the message queue.

[0019] The `poller` object is a core component in ZeroMQ for efficiently managing multiple socket events. It can simultaneously monitor the state changes of multiple sockets, avoiding the resource waste caused by creating a separate thread for each socket. Specifically, in the radar information system, sockets from subsystems such as radar, commands, and simulators are all registered in the same `poller` for centralized monitoring of data reception events. For example, the preset timeout can be set to 1000 milliseconds, ensuring both rapid response to events and avoiding unnecessary CPU idling.

[0020] In one embodiment of this application, the execution steps of the message serialization optimization algorithm may include: selecting Protocol Buffers, MessagePack, or JSON serialization method according to the configuration; if Protocol Buffers is selected, message serialization is performed using a pre-compiled proto structure; if MessagePack is selected, the MessagePack library is called for message serialization; if JSON is selected, the JSON library is used for message serialization.

[0021] Specifically, the choice of serialization method should be based on the requirements of the actual application scenario. For example, instruction sets have high requirements for data transmission efficiency and compression ratio, so Protocol Buffers serialization is configured, using pre-compiled proto structures to compress data into a compact binary stream; while simulation training systems require ease of debugging and compatibility, so JSON serialization is chosen, using a JSON library to convert data into an easy-to-read text format. Optionally, users can also flexibly configure a suitable serialization method based on factors such as inter-system communication bandwidth and data complexity.

[0022] Clearly, asynchronous I / O multiplexing algorithms can efficiently manage I / O events of multiple sockets, avoiding concurrency bottlenecks caused by traditional blocking communication. Combined with message serialization optimization algorithms for efficient data encoding, the ZeroMQ-based communication core can achieve decentralized direct communication between nodes without relying on a dedicated message broker server. This eliminates the risk of single points of failure and significantly improves data transmission throughput and real-time performance, meeting the high-concurrency, low-latency communication requirements of distributed heterogeneous systems.

[0023] For example, taking a radar information system dataset as an example, the steps for system initialization and communication architecture establishment are as follows: The purpose of system initialization and communication architecture establishment is to create a decentralized, high-performance communication network connecting various subsystems. Its input is the system configuration file `system_config.json`, which contains the network addresses, ports, and communication modes of each subsystem (radar, command, simulator). The output is a stable, multiplexed asynchronous communication network. Runtime objects are used for receiving, sending, and serializing / deserializing messages.

[0024] refer to Figure 2 The specific steps for initialization and communication architecture establishment are as follows: Step 1.1 Communication Core Initialization: (1) The middleware starts and reads system_config.json. For example, it configures the display radar system to act as a publisher (PUB) to publish target data at tcp: / / 192.168.1.100:5555; and the command and simulation systems to connect as subscribers (SUB).

[0025] (2) Call the initialize_communication_core(config) function to create an independent ZeroMQContext and Socket for each logical connection.

[0026] (3) Create a PUB Socket for the radar data channel and bind it to port 5555; create a SUB Socket for the command and simulator system, connect it to the radar address respectively, and set a subscription filter (such as "TARGET_UPDATE").

[0027] Step 1.2 Asynchronous I / O Multiplexing Settings: (1) Create an AsyncIOMultiplexer instance.

[0028] (2) Register all the above sockets (including those that may be created by future plugins) into the poller and listen for the zmq.POLLIN (data is readable) event.

[0029] (3) Start an independent event loop thread and run the event_loop() method. This thread continuously calls poller.poll(timeout=1000). When any data arrives at any socket, it immediately retrieves the message and puts it into the internal message queue for subsequent processing by the module. This step achieves low-latency response under high concurrency and avoids CPU consumption while waiting.

[0030] Step 1.3 Message serialization strategy configuration: (1) Based on the configuration, set Protocol Buffers (efficient and high compression ratio) to be used between the instruction system and the middleware, and JSON (easy to debug and compatible) to be used between the instruction system and the simulation training system.

[0031] (2) When the event loop thread receives the raw byte stream of the radar system, it calls MessageSerializer.optimize_serialization() to deserialize it and convert it into an internal unified data object.

[0032] (3) Before being sent to the instruction system, the internal data object is serialized into a compact binary format by serialize_with_protobuf(); when sent to the simulation system, it is serialized by json.dumps().

[0033] S102. A plug-in management framework is constructed based on the plug-in dependency resolution algorithm and the plug-in sandbox isolation algorithm to modularize, pluggable and dynamically expand intermediate communication plug-ins.

[0034] In one embodiment of this application, the plugin dependency resolution algorithm is a topological sorting algorithm, and its execution steps include: constructing a dependency graph, with each intermediate plugin as a node and the dependency relationship as a directed edge, the direction of the edge pointing from the dependent party to the dependent party; calculating the in-degree of each node; adding nodes with an in-degree of 0 to a queue; removing nodes from the queue and adding them to the loading order list, decrementing the in-degree of all its neighboring nodes by 1, and adding them to the queue if the in-degree of a neighboring node becomes 0; repeating the above steps until the queue is empty, and if the length of the loading order list is equal to the number of nodes, the loading order is determined; otherwise, a circular dependency exception is thrown.

[0035] The construction of the dependency graph relies on plugin configuration files, which explicitly record the other plugins that the plugin depends on. Specifically, when loading the `asterix_decoder` plugin, its `plugin_manifest.yaml` file is read, revealing its dependency on the `data_validator` plugin. Therefore, the constructed dependency graph contains directed edges from `data_validator` to `asterix_decoder`. It's important to note that the topological sorting algorithm strictly guarantees the correctness of the plugin loading order, preventing plugin initialization failures due to missing dependencies.

[0036] In one embodiment of this application, the execution steps of the plugin sandbox isolation algorithm may include: creating a sandbox environment containing restricted builtins and security APIs, and setting resource limits related to memory, execution time, and file descriptors of the sandbox environment; compiling the plugin code and performing syntax checks to obtain the compiled code; executing the compiled code in the sandbox environment and setting a timeout; and throwing an exception if a timeout or memory overflow exception occurs during execution.

[0037] Specifically, the sandbox environment only includes whitelisted built-in functions and secure APIs, such as `math.sqrt` and `json.loads`, while functions that may compromise system security, such as `__import__` and `open`, are strictly prohibited. For example, the maximum memory usage of a plugin can be set to 512MB, and the maximum execution time to 30 seconds, to prevent plugins from excessively consuming system resources. Understandably, plugin code undergoes syntax checking before execution to ensure its validity and prevent framework crashes due to syntax errors.

[0038] For example, the purpose of the plug-in management framework is to modularize the functionality for handling radar-specific binary protocols (such as ASTERIX) and dynamically integrate it into the system. The input is the asterix_decoder plugin package (containing plugin_manifest.yaml and decoder.py). The user issues a "Load radar plugin" command through the management interface. The output includes: the data_validator and asterix_decoder plugin instances running normally in the sandbox. Raw radar data is decoded into structured target information in real time.

[0039] refer to Figure 3 The steps for dynamically loading and running protocol adapter plugins include the following: Step 2.1 Plugin Discovery and Dependency Resolution: (1) PluginManager scans the specified plugin directory and finds the asterix_decoder plugin. It reads its manifest.yaml and finds that it depends on a basic data_validator plugin.

[0040] (2) Call DependencyResolver.resolve_dependencies() to build the dependency graph: {'asterix_decoder': ['data_validator']}.

[0041] (3) Execute Kahn's topological sorting algorithm: Calculate the in-degree: {'asterix_decoder': 1, 'data_validator': 0}.

[0042] Initial queue: ['data_validator'].

[0043] The sorting result is: ['data_validator', 'asterix_decoder']. This step ensures the correct loading order and avoids initialization failure due to missing dependencies.

[0044] Step 2.2 Safe Loading and Sandbox Initialization: (1) First, load the data_validator plugins in sequence. PluginManager calls the load_plugin() method.

[0045] (2) Use importlib to dynamically import the plugin module and call its initialization function. Before initialization, create a PluginSandbox for the plugin.

[0046] (3) The create_sandbox_environment() method constructs a restricted global dictionary that contains only functions on the whitelist (such as math.sqrt, json.loads) and removes dangerous functions such as __import__, open, and eval.

[0047] (4) Instantiate the plugin in the sandbox environment and register its declared message handlers (such as validate_target_data) to the system's message dispatcher via register_message_handlers().

[0048] Step 2.3 Plug-in Hot-Swapping and Communication: (1) After the asterix_decoder plugin is loaded, it declares that it is interested in the message type "RAW_ASTERIX_FRAME".

[0049] (2) When the communication core receives the radar raw byte stream and identifies it as this type, the message dispatcher will asynchronously call asterix_decoder.handle_message().

[0050] (3) The plugin runs in a sandbox, decodes binary ASTERIX data into target information in JSON format, and publishes a new “TARGET_UPDATE” message to the internal bus.

[0051] (4) If a plugin malfunctions, ResourceMonitor will detect timeouts or memory overruns, triggering plugin isolation and restart to ensure that a single plugin failure does not cause the entire middleware to crash.

[0052] The plugin dependency resolution algorithm ensures the orderliness and correctness of plugin loading, while the plugin sandbox isolation algorithm ensures the security and independence of plugin operation. The plugin-based management framework built by the combination of the two enables the intermediate communication plugin to achieve modular development, pluggable deployment, and dynamic expansion. This reduces the coupling between plugins, ensures the stability of the entire system, and avoids the impact of a single plugin failure on the overall system.

[0053] S103. A visual protocol adaptation interface is constructed using an intelligent field mapping algorithm and a real-time conversion preview algorithm. The graphical configuration is then used to generate protocol conversion rules and adaptation scripts through the visual protocol adaptation interface. In one embodiment of this application, the execution steps of the intelligent field mapping algorithm may include: traversing each field in the target protocol and all fields in the source protocol; calculating the name similarity and type compatibility between the source field and the target field, wherein the name similarity is a weighted average of edit distance and word vector similarity; determining type compatibility according to a preset type conversion table; obtaining a total score by combining name similarity and type compatibility, selecting the source field with the highest score as the mapping, and recording the confidence level.

[0054] The name similarity calculation combines edit distance and word vector similarity, providing a more comprehensive measure of the relevance of field names. For example, the source field "CmdID" and the target field "command_id" have an edit distance similarity of 0.75, a word vector similarity of 0.82, and a weighted average name similarity of 0.84. Both are string types, with a type compatibility of 1.0, resulting in a comprehensive score of 0.904, exceeding the preset threshold of 0.7. The system automatically recommends this mapping relationship. Specifically, the preset type conversion table clearly defines the compatibility between different data types; for example, int is compatible with float, and string is compatible with char.

[0055] In one embodiment of this application, the execution steps of the real-time conversion preview algorithm may include: obtaining the first 10 source data samples; for each sample, performing type conversion and verification field by field according to the currently configured mapping rules and conversion script; and displaying the converted data to the user.

[0056] For example, when a user configures the mapping relationship between XML format commands and internal JSON format commands in the command system, the system will extract 10 real XML commands from historical logs as source data samples, apply the currently configured mapping rules and conversion scripts to perform the conversion, and display the data before and after conversion side by side. Optionally, for the conversion of complex fields, users can write conversion scripts in the interface, and the system will update the preview results in real time, making it convenient for users to verify the correctness of the conversion logic.

[0057] For example, the purpose of the visual protocol adapter interface includes: creating a new data access interface for the command system, converting its proprietary XML-formatted commands into an internally unified JSON format without writing code. Inputs include: a command system data format specification (XSD Schema file); and internally common command JSON Schema definitions. Outputs include: a set of mapping rules completed in a visual configuration interface; an automatically generated and loaded CmdXMLAdapter plugin instance; and real-time streaming command data converted to the internal format.

[0058] refer to Figure 4 The processing steps for the visual protocol adaptation interface are as follows: Step 3.1 Protocol Parsing and Smart Mapping Recommendation: (1) Users upload the command.xsd of the command system and the internal command_schema.json in the visual interface.

[0059] (2) The backend ProtocolAdapterDesigner calls parse_xml_schema() and parse_json_schema() to extract the source fields (such as ``) and target fields (such as {"target": {"id": ""}}).

[0060] (3) The system starts IntelligentFieldMapper.auto_map_fields(): Calculate field similarity: For the source field "CmdID" and the target field "command_id", the algorithm calculates the edit distance similarity (0.75), word vector similarity (0.82) and word root similarity (1.0), and the weighted result is a name similarity of 0.84.

[0061] Computation type compatibility: The compatibility between "string" and "string" is 1.0.

[0062] Overall score: 0.84 * 0.6 + 1.0 * 0.4 = 0.904. This exceeds the threshold of 0.7, so the system automatically recommends mapping these two scores together.

[0063] Step 3.2 User Confirmation and Conversion Logic Definition: (1) Users can view and fine-tune the system-recommended mapping relationship by dragging and dropping the interface.

[0064] (2) For fields that require complex conversion (such as converting the “Location / Grid” string “NK123456” in the source XML to the target {“lat”: 39.12, “lon”: 116.45”}), users can write a small Python conversion function in the “Script Editor” on the right side of the interface.

[0065] (3) When the user clicks “Preview”, the system executes RealTimeTransformationPreview.generate_preview(), extracts several real XML commands from the historical log as samples, applies the currently configured mapping and script, and generates and displays the data before and after the transformation side by side in real time for the user to verify.

[0066] Step 3.3 Configuration Generation and Plugin Packaging: (1) After the user confirms that everything is correct, click “Generate Adapter”.

[0067] (2) ProtocolAdapterDesigner.generate_adapter_plugin(“CmdXMLAdapter”) is called, which populates the user-defined mapping rules and conversion scripts into a pre-defined plugin template, generating a complete, executable Python plugin class CmdXMLAdapter.

[0068] (3) The plugin class is automatically packaged and dynamically loaded into the running middleware by calling PluginManager.load_plugin(), taking effect immediately. The XML command flow of the instruction system can then be understood and processed through this new plugin.

[0069] The intelligent field mapping algorithm automatically matches the fields of the source and target protocols, reducing the workload and error rate of manual configuration. The real-time conversion preview algorithm allows users to intuitively view the conversion effect and adjust the configuration in a timely manner. The visual protocol adaptation interface built by these two technologies simplifies and graphically represents the complex protocol conversion configuration process, lowers the technical threshold, improves configuration efficiency, and enables the rapid generation of protocol conversion rules and adaptation scripts.

[0070] S104. Among them, the protocol conversion rules and adaptation scripts are encapsulated into adaptation plugins through a plug-in management framework and loaded into the ZeroMQ communication core.

[0071] Specifically, the protocol conversion rules and adaptation scripts generated by the visual protocol adaptation interface are encapsulated by the plug-in management framework according to preset plug-in templates, generating executable plug-in classes. For example, the generated CmdXMLAdapter plug-in class contains field mapping relationships and conversion scripts, which can convert XML format commands of the instruction system into the internally unified JSON format.

[0072] The purpose of the integrated script engine is to allow experimenters to temporarily insert a rule: filter out all targets with a height below 100 meters and highlight targets with specific numbers in red. Inputs include: the filtering and marking script `filter_script.py` written by the instructor in the management interface; the target data stream flowing through the middleware in real time; and the script bytecode that has undergone security verification and performance optimization, the target data stream filtered and enhanced in real time according to the script logic, and runtime performance monitoring reports.

[0073] refer to Figure 5 The integrated script engine processing steps are as follows: Step 4.1 Script security verification and compilation optimization: (1) The director submits the script. ScriptEngine first calls preprocess_and_validate_script().

[0074] (2) Syntax and security scanning: The validate_ast() function analyzes the script's AST and detects any attempts to call os.system (a dangerous operation), immediately rejecting execution and issuing a warning. After modification, the script contains only pure data logic.

[0075] (3) Dynamic compilation and optimization: DynamicCompiler.compile_python_script() compiles valid scripts.

[0076] ConstantFolder directly folds the expression 100 * 0.3048 (feet to meters) into the constant 30.48.

[0077] DeadCodeEliminator detects and eliminates a piece of debug code outside a loop whose result is not being used.

[0078] Finally, optimized bytecode is generated.

[0079] Step 4.2 Safe execution within the sandbox: (1) When a target data {"id": "TGT001", "alt_ft": 250} arrives, the system triggers the execution of the filtering script.

[0080] (2) SecureScriptSandbox.execute_script_safely() creates a secure environment and passes the target data as input_data.

[0081] (3) Execute optimized bytecode in the sandbox under the monitoring of the timeout(30) context manager and ResourceMonitor.

[0082] (4) The script calculates the height (250 * 0.3048 = 76.2 meters), determines it to be less than 100 meters, and therefore returns None (indicating filtering). For another target with sufficient height, the script adds the field {"highlight": "red"} to it.

[0083] Step 4.3 Runtime Monitoring and JIT Optimization: (1) PerformanceMonitor continuously monitors the execution of the script and finds that it is frequently called (hotspot function).

[0084] (2) The system automatically triggers the JIT compiler to perform deeper just-in-time compilation optimization on the bytecode of the script, converting it into machine code. After that, the execution speed of the filtering logic is increased by tens of times, meeting the millisecond-level response requirements of system communication.

[0085] The plug-in management framework uses the `load_plugin` method to dynamically load encapsulated adaptation plugins into the ZeroMQ communication core. During the loading process, a plugin dependency resolution algorithm checks for dependencies; if dependencies exist, they are loaded sequentially. It's worth noting that after loading, the adaptation plugin is registered in the communication core's message handler. When the communication core receives data corresponding to the protocol, it automatically calls the plugin for protocol conversion.

[0086] Clearly, by encapsulating the protocol conversion rules and adaptation scripts into adaptation plugins and loading them into the communication core through a plug-in management framework, the protocol adaptation function is decoupled from the communication core. The adaptation plugins can be developed, deployed, and updated independently without modifying the communication core code, which improves the system's flexibility and scalability while ensuring the stability of the communication core.

[0087] S105 employs a script engine that integrates a secure sandbox execution algorithm and a dynamic compilation optimization algorithm. This script engine is used to write, verify, and execute custom protocol conversion logic. The custom logic and the protocol conversion rules of the visual protocol adaptation interface are executed collaboratively to generate modular communication plugins. These modular intermediate plugins are dynamically plugged into the ZeroMQ communication core to achieve rapid and seamless integration of heterogeneous systems.

[0088] In one embodiment of this application, the execution steps of the secure sandbox execution algorithm may include: creating a secure global environment and restricting built-in functions and modules; removing dangerous functions such as __import__, open, and exec; injecting input data and secure APIs into the global environment; compiling script code and performing syntax analysis; setting an execution timeout and executing the compiled code in the sandbox environment; capturing exceptions during execution and returning execution results or error information.

[0089] Specifically, the secure global environment created by the script engine only includes safe built-in functions such as `len`, `str`, and `int`, prohibiting access to low-level system resources and dangerous operations. For example, if the filtering script written by the instructor contains the `os.system` function, the system will detect this dangerous function during the syntax analysis phase, immediately refuse execution, and issue an alert. For instance, the script execution timeout is set to 30 seconds to prevent the script from getting stuck in an infinite loop and consuming system resources.

[0090] In one embodiment of this application, the execution steps of the integrated script engine may include: selecting a corresponding compiler according to the script language; performing syntax analysis on the script to generate an abstract syntax tree (AST); optimizing the AST by constant folding, dead code elimination, and inlining; compiling the optimized AST into bytecode or machine code; and if the bytecode or machine code supports just-in-time compilation, performing further optimization from scratch.

[0091] Finally, the script engine executes custom data filtering logic. Its purpose includes: the experimenters need to temporarily insert a rule to filter out all targets with a height below 100 meters and to add a red highlight mark to targets with specific numbers.

[0092] Input includes: the filtering and tagging script `filter_script.py` written by the director in the management interface; and the target data stream flowing through the middleware in real time. Output includes: the script bytecode that has undergone security verification and performance optimization; and the target data stream that has been filtered and enhanced in real time according to the script logic.

[0093] Referring to Figure 6, the processing steps of the integrated script engine may include: Step 4.1 Script security verification and compilation optimization: (1) The director submits the script. ScriptEngine first calls preprocess_and_validate_script().

[0094] (2) Syntax and security scanning: The validate_ast() function analyzes the script's AST and detects any attempts to call os.system (a dangerous operation), immediately rejecting execution and issuing a warning. After modification, the script contains only pure data logic.

[0095] (3) Dynamic compilation and optimization: DynamicCompiler.compile_python_script() compiles valid scripts.

[0096] ConstantFolder directly folds the expression 100 * 0.3048 (feet to meters) into the constant 30.48.

[0097] DeadCodeEliminator detects and eliminates a piece of debug code outside a loop whose result is not being used.

[0098] Finally, optimized bytecode is generated.

[0099] Step 4.2 Safe execution within the sandbox: (1) When a target data {"id": "TGT001", "alt_ft": 250} arrives, the system triggers the execution of the filtering script.

[0100] (2) SecureScriptSandbox.execute_script_safely() creates a secure environment and passes the target data as input_data.

[0101] (3) Execute optimized bytecode in the sandbox under the monitoring of the timeout(30) context manager and ResourceMonitor.

[0102] (4) The script calculates the height (250 * 0.3048 = 76.2 meters), determines it to be less than 100 meters, and therefore returns None (indicating filtering). For another target with sufficient height, the script adds the field {"highlight": "red"} to it.

[0103] Step 4.3 Runtime Monitoring and JIT Optimization: (1) PerformanceMonitor continuously monitors the execution of the script and finds that it is frequently called (hotspot function).

[0104] (2) The system automatically triggers the JIT compiler to perform deeper just-in-time compilation optimization on the bytecode of the script, converting it into machine code. After that, the execution speed of the filtering logic is increased by tens of times, meeting the millisecond-level response requirements of system communication.

[0105] It can be said without a doubt that the secure sandbox execution algorithm provides a secure running environment for custom scripts, preventing malicious code from damaging the system, while the dynamic compilation optimization algorithm improves the execution efficiency of the scripts. The integrated script engine supports users in writing, verifying, and executing custom protocol conversion logic, meeting personalized needs in complex scenarios.

[0106] In other words, users can use custom protocol conversion logic written in the script engine to work in conjunction with the conversion rules generated by the visual protocol adaptation interface to complete the data protocol conversion. For example, the field mapping rules configured in the visual interface are responsible for basic field matching, while the custom script is responsible for complex field conversion and data filtering. The two together form a complete protocol adaptation logic, which is then encapsulated as a modular communication plugin.

[0107] For example, in radar information systems, radar data adaptation plugins, command processing plugins, and other modular plugins are generated in this way. These plugins can be dynamically loaded into the ZeroMQ communication core through a plugin management framework and can also be uninstalled at any time as needed, achieving hot-swapping of plugins. It should be noted that the modular intermediate plugins interact with other subsystems through the communication core, completing data transmission and protocol conversion between heterogeneous systems.

[0108] Clearly, the collaborative work of custom logic and visual configuration conversion rules can cover various complex protocol adaptation scenarios. The generated modular communication plugins are integrated into the communication core through dynamic plug-and-play, enabling rapid and seamless integration of heterogeneous systems without large-scale modifications to existing systems. This significantly reduces the workload and maintenance costs of system integration and solves the problems of complex protocol adaptation and high system coupling in traditional system integration.

[0109] It is worth noting that the method also includes a message routing engine for content-based message routing, supporting the configuration of complex routing rules; quality of service assurance can provide message persistence, retry mechanisms and failover functions; and the monitoring and management interface can provide a RESTful API for middleware status monitoring and management.

[0110] After testing and verification, the present invention has the following significant technical effects: 1. Improved development efficiency: The integration time for the new system has been reduced from several weeks to several days, resulting in a 60% increase in efficiency. 2. Significantly improved performance: message throughput reaches 10,000 messages / second, latency is less than 1 millisecond. 3. System Stability: The plug-in isolation mechanism ensures that a single point of failure does not affect the overall system, achieving 99% availability. 4. Protocol Compatibility: Supports 20+ industrial protocols and military-specific protocols, including Modbus, OPC UA, and DDS. 5. Resource utilization: Memory usage reduced by 60%, CPU utilization increased by 40%. 6. Cross-platform support: Stable operation on operating systems such as Windows, Linux, and Kylin. Reference Figure 2 The second embodiment of this application provides Subject Matter Two, which may include: Based on the above embodiments, the second embodiment of this application also provides a design and implementation apparatus for a pluggable data communication middleware based on ZeroMQ, which may include: Applications of distributed heterogeneous systems: The asynchronous core building block is used to construct an asynchronous communication core based on ZeroMQ, using asynchronous I / O multiplexing algorithms and message serialization optimization algorithms, so as to enable direct communication between decentralized heterogeneous node systems; The plugin management module is used to build a plugin-based management framework based on plugin dependency resolution algorithms and plugin sandbox isolation algorithms, so as to modularize, pluggable, and dynamically extend intermediate communication plugins. The protocol visualization adaptation module is used to build a visual protocol adaptation interface using intelligent field mapping algorithm and real-time conversion preview algorithm. Through the visual protocol adaptation interface, the graphical configuration is generated into protocol conversion rules and adaptation scripts. Among them, the protocol conversion rules and adaptation scripts are encapsulated into adaptation plugins through a plug-in management framework and loaded into the ZeroMQ communication core; The script integration module is used to integrate the script engine with the security sandbox execution algorithm and dynamic compilation optimization algorithm, and to write, verify and execute custom protocol conversion logic through the script engine; The custom logic and the protocol conversion rules of the visual protocol adaptation interface work together to generate modular communication plugins. These modular intermediate plugins are then dynamically plugged into the ZeroMQ communication core to achieve rapid and seamless integration of heterogeneous systems.

[0111] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A design and implementation method for a pluggable data communication middleware based on ZeroMQ, characterized in that, include: In distributed heterogeneous systems: An asynchronous communication core based on ZeroMQ is constructed using asynchronous I / O multiplexing algorithms and message serialization optimization algorithms to enable direct communication between decentralized heterogeneous systems. A plug-in management framework is built based on plug-in dependency resolution algorithm and plug-in sandbox isolation algorithm to modularize, pluggable and dynamically expand intermediate communication plug-ins; A visual protocol adaptation interface is constructed using an intelligent field mapping algorithm and a real-time conversion preview algorithm. The graphical configuration is then used to generate protocol conversion rules and adaptation scripts through the visual protocol adaptation interface. The protocol conversion rules and adaptation scripts are encapsulated as adaptation plugins through a plug-in management framework and loaded into the ZeroMQ communication core. A script engine integrating a secure sandbox execution algorithm and a dynamic compilation optimization algorithm is used to write, verify, and execute custom protocol conversion logic. The custom logic and the protocol conversion rules of the visual protocol adaptation interface work together to generate modular communication plugins. These modular intermediate plugins are then dynamically plugged into the ZeroMQ communication core to achieve rapid and seamless integration of heterogeneous systems.

2. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The execution steps of the asynchronous I / O multiplexing algorithm include: Create a poller object to monitor I / O events for multiple sockets; Register the sockets that need to be listened to into the poller and monitor the POLLIN event; In the loop, wait for the event to occur using the poller's poll method and set a preset timeout. When an event occurs, iterate through all ready sockets. If it is a POLLIN event, receive the message in a non-blocking manner and put it into the message queue. Process messages in the message queue.

3. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The execution steps of the message serialization optimization algorithm include: Choose Protocol Buffers, MessagePack, or JSON serialization method based on the configuration; If Protocol Buffers is selected, message serialization is performed using pre-compiled proto structures; If MessagePack is selected, the MessagePack library will be called to perform message serialization; If JSON is selected, the JSON library will be used for message serialization.

4. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The plugin dependency resolution algorithm is a topological sorting algorithm, and its execution steps include: Construct a dependency graph with each intermediate plugin as a node and the dependency relationship as a directed edge, with the edge pointing from the dependent party to the dependent party; Calculate the in-degree of each node; Add nodes with an in-degree of 0 to the queue; Remove a node from the queue and add it to the loading order list. Decrement the in-degree of all its neighboring nodes by 1. If the in-degree of a neighboring node becomes 0, add it to the queue. Repeat the above steps until the queue is empty. If the length of the loading order list is equal to the number of nodes, the loading order is determined; otherwise, a circular dependency exception is thrown.

5. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The execution steps of the plug-in sandbox isolation algorithm include: Create a sandbox environment with restricted builtins and security APIs, and set resource limits for the sandbox environment related to memory, execution time, and file descriptors; Compile the plugin code and perform syntax checks to obtain the compiled code; The compiled code is executed in a sandbox environment with a timeout set; if a timeout or memory overflow exception occurs during execution, an exception is thrown.

6. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The execution steps of the intelligent field mapping algorithm include: Iterate through each field in the target protocol and all fields in the source protocol; Calculate the name similarity and type compatibility between the source and target fields. The name similarity is calculated using a weighted average of edit distance and word vector similarity. Type compatibility is determined based on a pre-defined type conversion table. A total score is obtained by combining name similarity and type compatibility. The source field with the highest score is selected as the mapping and the confidence level is recorded.

7. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The execution steps of the real-time conversion preview algorithm include: Obtain the first 10 source data samples. For each sample, perform type conversion and verification field by field according to the currently configured mapping rules and conversion script. Show the user the transformed data.

8. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The execution steps of the security sandbox execution algorithm include: Create a secure global environment that restricts built-in functions and modules; Remove dangerous functions such as __import__, open, and exec; Inject input data and security APIs into the global environment; Compile the script code and perform syntax analysis; Set an execution timeout to execute the compiled code in a sandbox environment; capture exceptions during execution and return the execution result or error information.

9. The design and implementation method of the pluggable data communication middleware based on ZeroMQ as described in claim 1, characterized in that, The execution steps of the dynamic compilation optimization algorithm include: Choose the appropriate compiler based on the scripting language; Perform syntax analysis on the script to generate an abstract syntax tree (AST); optimize the AST by constant folding, dead code elimination, and inlining. Compile the optimized AST into bytecode or machine code; If the bytecode or machine code supports just-in-time compilation, then further optimization is performed from scratch.

10. A design and implementation device for a pluggable data communication middleware based on ZeroMQ, characterized in that, include: Applied to distributed heterogeneous systems: The asynchronous core building block is used to build an asynchronous communication core based on ZeroMQ, using asynchronous I / O multiplexing algorithms and message serialization optimization algorithms, so as to enable direct communication between decentralized heterogeneous systems. The plugin management module is used to build a plugin-based management framework based on plugin dependency resolution algorithms and plugin sandbox isolation algorithms, so as to modularize, pluggable, and dynamically extend intermediate communication plugins. The protocol visualization adaptation module is used to build a visual protocol adaptation interface using intelligent field mapping algorithm and real-time conversion preview algorithm. Through the visual protocol adaptation interface, the graphical configuration is generated into protocol conversion rules and adaptation scripts. The protocol conversion rules and adaptation scripts are encapsulated as adaptation plugins through a plug-in management framework and loaded into the ZeroMQ communication core. The script integration module is used to integrate the script engine with the security sandbox execution algorithm and dynamic compilation optimization algorithm, and to write, verify and execute custom protocol conversion logic through the script engine; The custom logic and the protocol conversion rules of the visual protocol adaptation interface work together to generate modular communication plugins. These modular intermediate plugins are then dynamically plugged into the ZeroMQ communication core to achieve rapid and seamless integration of heterogeneous systems.

Images