Gateway data processing method based on plug-in mechanism, electronic equipment and medium

Through the gateway data processing method based on the plug-in mechanism, the gateway is divided into main programs and plug-ins, which solves the problems of high system pressure and poor scalability of traditional data acquisition tools, realizes lightweight and efficient data acquisition, supports multi-protocol and flexible expansion .

CN119071149APending Publication Date: 2024-12-03山东浪潮智能生产技术有限公司
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411355827.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-27
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing data acquisition tools and platforms have problems such as high system pressure, high hardware costs and insufficient expansion capabilities, especially traditional soft gateways are not light enough and have poor scalability.

Method used

Adopt the gateway data processing method based on the plug-in mechanism, and divide the gateway into the main program and extensible plug-in. The main program is responsible for device configuration and data process management. The plug-in is responsible for specific network protocol processing and data collection, and supports hot loading and custom script expansion. , improve data acquisition efficiency and scalability through message publishing and event-driven.

Benefits of technology

It achieves lightweight, improves data acquisition efficiency and scalability, reduces system pressure and hardware costs, supports multiple network protocols and data protocols, and enhances flexibility and wide application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071149A_ABST
    Figure CN119071149A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of software, and discloses a gateway data processing method based on a plug-in mechanism, electronic equipment and a storage medium, and the method comprises the steps: obtaining a network request for collecting data; starting a main program according to the network request, and loading a plug-in; configuring an event type according to the plug-in; and performing data processing according to the event type. According to the invention, the gateway is divided into the main program and the extensible plug-in, the main program only pays attention to the configuration of the equipment and the management of the data flow, the plug-in pays attention to the specific network protocol processing and data acquisition, and only related interfaces defined in the main program need to be realized, so that the implementation is convenient. The main program can load the plug-ins with the unlimited number in a hot loading mode, collection and data forwarding of various kinds of data can be met, the data collection efficiency is improved, light weight is achieved, and expansibility is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial Internet and Internet of Things, and specifically relates to a data processing method, electronic device and medium of a gateway based on a plug-in mechanism. Background Art

[0002] The Internet is rapidly extending from the consumer field to the production field, and the industrial economy is deeply expanding from digitalization to networking and intelligence, which in turn has given rise to the Industrial Internet. Data is the foundation for the development of the Industrial Internet, especially some perception data, such as equipment data, alarm data, etc. Therefore, there are very high requirements for data collection. Many data collection tools usually run on independent platforms or as part of the Internet of Things platform.

[0003] The data collection methods of traditional data collection tools or platforms (hereinafter referred to as data collection platforms) are generally divided into the following categories: 1. Directly collect device data through the Internet of Things platform; 2. Collect data through hardware gateways; 3. Collect data through soft gateways. Generally speaking, data collection platforms will support these categories at the same time, but there are some problems with these collection methods: the direct collection method has a great impact on the system pressure, especially when there are a large number of direct collection devices. If the concurrency is not handled well, it will affect the performance of the entire platform; the hard gateway method separates data collection and the platform, which can greatly reduce the pressure on the platform, but the biggest problem is that the hardware cost and maintenance cost need to be considered; the soft gateway method can reduce the pressure on the platform and control costs, but the usual soft gateway is not lightweight enough, and will pre-set various data collection drivers. The drivers cannot be dynamically added, deleted, and updated, so the expansion capability is insufficient. Summary of the invention

[0004] In view of this, the present invention provides a data processing method, an electronic device and a storage medium for a gateway based on a plug-in mechanism to solve the problems of insufficient lightness and insufficient expansion capability existing in the prior art.

[0005] In a first aspect, the present invention provides a data processing method for a gateway based on a plug-in mechanism, comprising: obtaining a network request for collecting data; starting a main program according to the network request and loading a plug-in; configuring an event type according to the plug-in; and performing data processing according to the event type.

[0006] In some implementations, starting a main program according to the network request and loading a plug-in specifically includes: automatically loading the plug-in when the main program starts; or manually placing the plug-in into a plug-in directory after the main program starts; and hot-loading the plug-in by the main program.

[0007] In some implementations, configuring the event type according to the plug-in specifically includes: configuring the collected information according to the collection scope; and packaging the collected information into the event type to complete the configuration of the event type.

[0008] In some embodiments, the data types include data acquisition, acquisition termination or initiation, and acquisition range update.

[0009] In some implementations, the data processing method of the gateway based on the plug-in mechanism further includes: loading a driver; after the driver is loaded, starting event monitoring so that different plug-ins are only associated with their own events.

[0010] In some embodiments, the data processing method of the gateway based on the plug-in mechanism also includes: the data collected by the plug-in is standardized and packaged and published to the main program through message publishing; after the data collected by the plug-in is collected by the main program, it is filtered according to the configuration information or batch processed according to the system concurrency.

[0011] In some implementations, the data processing method of the gateway based on the plug-in mechanism further includes: integrating the data collected by the plug-in through the gateway to obtain integrated data; and forwarding the integrated data to the platform or other programs.

[0012] In some implementations, the data processing method of the gateway based on the plug-in mechanism further includes: extending the plug-in through a custom script and real-time hot loading.

[0013] In a second aspect, the present invention also provides an electronic device, comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: obtain a network request for collecting data; start a main program and load a plug-in based on the network request; configure an event type based on the plug-in; and perform data processing based on the event type.

[0014] In a third aspect, the present invention provides a storage medium storing a computer program, configuration information, and data logs. When the computer program is executed by a processor, it implements any one of the data processing methods for a gateway based on a plug-in mechanism.

[0015] The beneficial effects of the present invention are:

[0016] The gateway is divided into a main program and extensible plug-ins. The main program only focuses on device configuration and data flow management, while the plug-ins focus on specific network protocol processing and data collection. It only needs to implement the relevant interfaces defined in the main program. The main program can load an unlimited number of plug-ins through hot loading, which can meet the collection and forwarding of various types of data, improve the efficiency of data collection, achieve lightweight, and improve scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a data processing method of a gateway based on a plug-in mechanism according to an embodiment of the present invention;

[0019] Figure 2 It is a flowchart of a data processing method of a gateway based on a plug-in mechanism according to another embodiment of the present invention;

[0020] Figure 3 A flowchart of plug-in loading of a data processing method for a gateway based on a plug-in mechanism according to an embodiment of the present invention;

[0021] Figure 4 The figure is a block diagram of an electronic device according to an embodiment of the present invention.

[0022] Description of reference numerals:

[0023] 110. Processor;

[0024] 120. Memory. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] Combine the following Figures 1 to 4 , describing an embodiment of the present invention.

[0027] like Figure 1 and Figure 2As shown, according to an embodiment of the present invention, on the one hand, a data processing method of a gateway based on a plug-in mechanism is provided, comprising the following steps:

[0028] Step S101: obtaining a network request for collecting data.

[0029] Step S103: Start the main program according to the network request and load the plug-in.

[0030] Step S105: configure the event type according to the plug-in.

[0031] Step S107: Perform data processing according to the event type.

[0032] In this embodiment, the gateway is a soft gateway. The gateway is divided into a main program and an extensible plug-in. The main program only focuses on the configuration of the device and the management of the data flow, while the plug-in focuses on the specific network protocol processing and data collection. It only needs to implement the relevant interfaces defined in the main program. The main program can load an unlimited number of plug-ins through hot loading, which can meet the collection and forwarding of various types of data, improve the efficiency of data collection, achieve lightweight, and improve scalability.

[0033] Among them, define the plug-in, define the plug-in, including plug-in definition information (plugin.properties) and plug-in standard structure. Through customization, multiple network protocols and data protocols can be supported, such as: Http, Tcp, Udp, CoAP, Modbus, Opc, Mqtt, Plc, Websocket, Kafka, DB, etc.

[0034] Further, based on step S103, the main program is started according to the network request, and the plug-in is loaded, which specifically includes the following steps:

[0035] Step S1031: Automatically load the plug-in when the main program starts. Or

[0036] Step S1033: After the main program is started, manually place the plug-in into the plug-in directory.

[0037] Step S1035: The main program hot-loads the plug-in.

[0038] In this embodiment, the plug-in is responsible for specific data collection. The plug-in can be placed in the plug-in directory in advance and automatically loaded when the gateway main program starts. After the main program is started, the plug-in can also be manually placed in the plug-in directory and hot-loaded by the main program; the plug-in supports manual loading and also supports loading by the platform. Through the hot loading plug-in method, the scalability and lightweight development and deployment of data collection are achieved. Through the custom classloader method, hot loading, hot unplugging and hot updating of plug-ins can be achieved. The use of interface programming, plug-in mechanism and event-driven methods improves the expansion capability of the product and can achieve high scalability. Through hot loading technology, the free combination of plug-ins can be met, which improves flexibility.

[0039] For example, Figure 3 As shown, the specified plug-in loading directory is the main program jar package, the plugins directory under the same directory, this directory is configurable.

[0040] Traverse or periodically poll the plugins in the directory to get the exact address of the plugin

[0041] Customize URLClassLoader to load plugin jar package

[0042] URLClassLoaderclassLoader=URLClassLoader.newInstance(new

[0043] URL[]{new URL(url)},ClassloaderUtil.class.getClassLoader())

[0044] Load the configuration information "plugin.properties" in the plug-in, and read the plug-in name, version, description, and main loading class information.

[0045] According to the main loading class information, the main class is loaded through the reflection mechanism, and the bean is initialized and loaded into the spring bean factory through spring's bean definition and bean factory BeanDefinitionBuilderbeanDefinitionBuilder=BeanDefinitionBuilder.genericBeanDefinition(clazz); SpringUtil.defaultListableBeanFactory.registerBeanDefinition(beanName,beanDefinitionBuilder.getBeanDefinition())

[0046] Furthermore, step S105, according to the plug-in configuration event type, specifically includes the following steps:

[0047] Step S1051: configure the collection information according to the collection range.

[0048] Step S1053: Pack the collected information into event types to complete the configuration of event types.

[0049] The scope of collection includes but is not limited to industrial equipment, files, etc. It is also necessary to define the collection source, collection content, driver plug-in, etc. These configurations are usually issued by the platform, and can also be defined directly on the gateway side. Taking industrial equipment as an example, the collection scope includes device information and measurement point information. The device information includes device name, device number, device type, driver plug-in, connection information, collection frequency, etc.; the measurement point information includes measurement point name, measurement point number, measurement point identification, data type, collection point, etc.

[0050] After configuring the collected information, the program will automatically retrieve and group it according to the plug-in, and package the collected information into event classes, sending different events according to the required plug-ins. Through the event mechanism, separation of concerns is achieved and the encapsulation of the module is improved.

[0051] Specifically, the data types include data collection, collection termination or start, and collection range update.

[0052] Event types mainly include: data collection, collection termination / startup, collection range update, etc. These are defined in the event type according to different attributes. After the plug-in receives the event it needs to process, it will perform different data processing according to the defined event type.

[0053] Among them, data collection: extract the definition information that needs to be collected from the event class, establish a connection, and collect data according to the definition content. The data may be collected through polling or subscription.

[0054] Collection termination / start: The entire plug-in collection can be terminated, or part of the collection content can be terminated. After termination, it can be started according to the command. After starting, the configuration will be updated to collect data.

[0055] Collection scope update: This will involve changes in the collection configuration or additions or deletions to the scope, and the data collection process needs to be updated based on the configuration information.

[0056] Furthermore, the data processing method of the gateway based on the plug-in mechanism also includes the following steps:

[0057] Step S201: Load the driver.

[0058] Step S203: After the driver is loaded, event monitoring is started so that different plug-ins are only associated with their own events.

[0059] In this embodiment, after the driver is loaded, event monitoring will be started, and different plug-ins only need to pay attention to their own events. When an event is received, a comparison is performed. If it is an event that needs to be paid attention to, the data is taken out for processing.

[0060] Among them, you can make full use of Spring Event custom events and inherit ApplicationEvent. The main contents include event type, data type, plug-in driver, event data, etc. The event generator publishes the corresponding event through ApplicationEventPublisher, encapsulates the relevant data, and waits for the event monitor to consume it after sending. Define event monitoring to implement ApplicationListener, and judge whether it is event information that needs to be processed by itself through the event type, data type, plug-in driver and other information in the event object.

[0061] In this example, based on the event-driven spring event, we make full use of the encapsulation of event publishing and monitoring by sping to simplify event operations. Custom events include: plug-in loading completion, plug-in startup, plug-in stop, data acquisition configuration changes, etc. Through event-driven, asynchronous execution of related operations is achieved, reducing time consumption and concurrency pressure.

[0062] Furthermore, the data processing method of the gateway based on the plug-in mechanism also includes the following steps:

[0063] Step S301: The data collected by the plug-in is packaged in a standardized manner and published to the main program through message publishing.

[0064] Step S303: After the data collected by the plug-in is collected by the main program, it is filtered according to the configuration information or batch processed according to the system concurrency situation.

[0065] In this embodiment, after the plug-in collects data, it will standardize and package the collected content and publish it to the main program through message publishing; and after the main program collects the plug-in collected data, it will filter it according to the configuration information or perform batch processing according to the system concurrency.

[0066] Specifically, based on the disruptor technology, CAS technology can be used to meet concurrency and thread safety. Ringbuffer can be used to buffer data and ensure the concurrent release of large amounts of data. Sequencer, as the brain of the disruptor, can be used to control data generation and consumption. Wait Strategy can be used to implement a processing strategy for overcapacity data.

[0067] Furthermore, the data processing method of the gateway based on the plug-in mechanism also includes the following steps:

[0068] Step S401: Integrate the data collected by the plug-in through the gateway to obtain integrated data.

[0069] Step S403: forwarding the integrated data to the platform or other programs.

[0070] When forwarding data, the gateway integrates plug-ins to collect data, which will eventually need to be forwarded to the platform or other programs. The gateway will forward it in a standard data format, including but not limited to MQTT, Websocket, etc. Through the production and consumption model + buffer + lock-free, the system's concurrency and data processing capabilities are improved.

[0071] In this embodiment, the disruptor open source framework can be used to achieve high concurrency and high throughput through asynchronous queues. The framework mainly uses technologies such as CAS, Ring Buffer, Sequencer, Event, and Wait Strategy to achieve high concurrency. The gateway uses the disruptor's multi-producer single-consumer scenario to aggregate the data acquisition plug-in data for unified processing and forwarding.

[0072] The gateway can be deployed in a distributed, multi-platform, and multi-mode manner, realizing distributed collection and centralized management.

[0073] Among them, distributed deployment can support single-machine multi-gateway deployment, multi-environment and mode deployment, and low system resource requirements. Multi-environment deployment can support windows and linux operating system deployment, support docker, k8s and other modes of deployment. A gateway can load one or more plug-ins to realize the collection of various types of data and realize the free combination of plug-ins.

[0074] Furthermore, the data processing method of the gateway based on the plug-in mechanism also includes: extending the plug-in through custom scripts and real-time hot loading.

[0075] Scripting technology can be used to meet the flexible configuration of functions. Among them, you can define the Groovy script loading tool class GroovyUtil, implement the GroovyShell class instance, load the script content to instantiate the script object. The script content can come from the configuration file or be entered through the interface. For scripts that need to be executed multiple times, the script object can be cached to avoid instantiation every time. When the script is executed, it can be executed without parameters or with running parameters, and finally the required functions can be realized.

[0076] Scripts can be hot-loaded. You can provide Groovy scripts to the gateway through configuration files and page configurations, and then they can be automatically executed. Scripts are mainly used as extensions of general plug-ins to meet the fine-tuning of plug-in data acquisition work and the supplementary operations of the gateway's main program on the data flow. Through custom scripts and real-time hot loading, flexible and dynamic function expansion is achieved.

[0077] like Figure 4 As shown, according to an embodiment of the present invention, on the other hand, an electronic device is provided, including: at least one processor 110; a memory 120 communicatively connected to the at least one processor, wherein the memory 120 stores instructions executable by the at least one processor 110, and the instructions are executed by the at least one processor 110, so that the at least one processor 110 can perform the following steps:

[0078] Step S101: obtaining a network request for collecting data.

[0079] Step S103: Start the main program according to the network request and load the plug-in.

[0080] Step S105: configure the event type according to the plug-in.

[0081] Step S107: Perform data processing according to the event type.

[0082] According to an embodiment of the present invention, on the other hand, a storage medium is provided, which stores a computer program, configuration information, and data logs, and when the computer program is executed by a processor, any one of the data processing methods of a gateway based on a plug-in mechanism is implemented.

[0083] The computer executes the instructions to perform the following steps:

[0084] Step S101: obtaining a network request for collecting data.

[0085] Step S103: Start the main program according to the network request and load the plug-in.

[0086] Step S105: configure the event type according to the plug-in.

[0087] Step S107: Perform data processing according to the event type.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0090] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0091] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0092] The beneficial effects of the present invention are:

[0093] 1. The gateway is decomposed into the main program and extension plug-ins, both of which are in the form of jar packages, with small size, simple deployment, and lightweight.

[0094] 2. Through plug-in and script mechanisms, flexible configuration and high scalability can be achieved.

[0095] 3. Through distributed and asynchronous operations, the concurrency of collection and data processing speed are greatly improved, and performance is improved.

[0096] 4. The overall process is clear, plug-in development is simple, and it can easily support existing and future data collection, increasing the breadth of application.

[0097] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0098] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data processing method for a gateway based on a plug-in mechanism, characterized in that: include: Get network requests for collected data; Start the main program according to the network request and load the plug-in; Configure event types according to the plugin; Data processing is performed according to the event type.

2. The data processing method of a gateway based on a plug-in mechanism according to claim 1, characterized in that: The main program is started according to the network request, and the plug-in is loaded, specifically including: Automatically load the plug-in when the main program starts; or After the main program is started, manually place the plug-in into the plug-in directory; The main program hot loads the plug-in.

3. The data processing method of a gateway based on a plug-in mechanism according to claim 1, characterized in that: According to the plug-in configuration event type, it specifically includes: Configure the collected information according to the collection scope; The collected information is packaged into the event type to complete the configuration of the event type.

4. The data processing method of the gateway based on the plug-in mechanism according to claim 3 is characterized in that: The data types include data collection, collection termination or start, and collection range update.

5. The data processing method of a gateway based on a plug-in mechanism according to any one of claims 1 to 4, characterized in that: Also includes: Load the driver; After the driver is loaded, event monitoring is started so that different plug-ins are only associated with their own events.

6. The data processing method of a gateway based on a plug-in mechanism according to any one of claims 1 to 4, characterized in that: Also includes: The data collected by the plug-in is packaged in a standardized manner and published to the main program through message publishing; After the data collected by the plug-in is collected by the main program, it is filtered according to the configuration information or batch processed according to the system concurrency.

7. The data processing method of a gateway based on a plug-in mechanism according to any one of claims 1 to 4, characterized in that: Also includes: Integrate the data collected by the plug-in through the gateway to obtain integrated data; The integrated data is forwarded to the platform or other programs.

8. The data processing method of a gateway based on a plug-in mechanism according to any one of claims 1 to 4, characterized in that: Also includes: Extend the plugin with custom scripts and live hot reloading.

9. An electronic device, characterized in that: include: at least one processor (110); a memory (120) communicatively coupled to the at least one processor (110); The memory (120) stores instructions executable by the at least one processor (110), and the instructions are executed by the at least one processor (110) to enable the at least one processor (110) to: Get network requests for collected data; Start the main program according to the network request and load the plug-in; Start, stop, update, and uninstall the plug-in based on the received action; Configure event types according to the plugin; Data processing is performed according to the event type and configuration information.

10. A storage medium, characterized in that: A computer program, configuration information and data log are stored, and when the computer program is executed by a processor, a data processing method for a gateway based on a plug-in mechanism as described in any one of claims 1 to 8 is implemented.

Citation Information

Cited By

  • Data processing system, method and equipment based on plug-in and medium

    CN119806843A

  • Plugin-based Data Processing System, Method, Device, and Medium

    CN119806843B