Plugin-based Data Processing System, Method, Device, and Medium

Abstract

The present disclosure provides a data processing system, method, device, and medium based on plugins. The data processing system includes a cloud end, an edge computing end, and a device end; the cloud end includes a human-computer interaction module; the edge computing end includes a data processing plugin, and the edge computing end runs a data acquisition service, a data processing service, and an edge gateway service; the data processing service is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plugin, and implement data processing process update operations, device offline debugging operations, reported data processing operations, and device operation control operations. The present disclosure improves the real-time performance and response speed of the data processing system; through the data processing plugin, data processing and conversion can be flexibly performed, supporting different device data formats and access methods, meeting the diverse customization needs of customers, reducing the cost and complexity of customized development, and improving development efficiency and debugging efficiency.

Description

Technical Field

[0001] The present disclosure relates to the technical field of the Internet of Things (IoT), and particularly to a plugin-based data processing system, method, device, and medium. Background Art

[0002] With the rapid development of the Internet of Things (IoT) technology, more and more intelligent devices are widely used in various fields, such as industrial automation, smart cities, healthcare, etc. The massive data generated by these devices needs to be analyzed and processed in real-time or near real-time to support rapid decision-making. The traditional cloud computing model uploads all data to the cloud for processing, which not only increases the network transmission latency but also may cause the cloud server to be overloaded due to the large amount of data. To solve these problems, edge computing emerged. By deploying computing resources close to the data source, it realizes local data processing, effectively reducing data transmission latency and relieving the pressure on the cloud.

[0003] However, the existing IoT device data processing solutions still face the following challenges:

[0004] 1. High hardware resource requirements: A large amount of physical resources (including computing resources, storage resources, and network resources) are required to support the access and data processing of a large number of IoT devices, resulting in high requirements for the hardware infrastructure.

[0005] 2. High data processing latency: The latency of data transmission and processing affects the real-time performance and response speed of the system. Especially in a low-bandwidth or high-latency network environment, the data transmission and processing processes need to be optimized.

[0006] 3. High customization development cost: Due to the different data formats and access methods of different devices, a large number of customization requirements are brought to process the data reported by the devices, and a flexible data processing and conversion mechanism needs to be supported. The traditional code implementation method or the streaming processing method based on Flink (an open-source distributed stream processing framework) both require quite professional developers to support, resulting in an increase in the overall project cost.

[0007] 4. Long development and debugging cycle: Traditional data processing development usually requires developers to modify the code and then update the program to the field to implement. When problems occur during debugging, this time-consuming process will be repeated, resulting in a long overall project development and debugging cycle. Summary of the Invention

[0008] The technical problem to be solved by the present disclosure is to overcome the defects of high hardware resource requirements, high data processing latency, high customization development cost, and long development and debugging cycle in the existing industrial Internet of Things device data processing, and to provide a plugin-based data processing system, method, device, and medium.

[0009] The present disclosure solves the above technical problems through the following technical solutions:

[0010] In a first aspect, a data processing system based on plugins is provided. The data processing system is applied to the industrial Internet of Things. The data processing system includes a cloud, an edge computing terminal, and a device terminal;

[0011] The edge computing terminal is communicatively connected to the cloud and the device terminal respectively;

[0012] The cloud includes a human-computer interaction module;

[0013] The edge computing terminal includes data processing plugins. The edge computing terminal runs a data acquisition service, a data processing service, and an edge gateway service;

[0014] The data processing plugins run depending on the data processing service;

[0015] The data processing service is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plugins, and implement data processing process update operations, device offline debugging operations, reported data processing operations, and device operation control operations.

[0016] Optionally, the human-computer interaction module is used to provide a device operation configuration interface;

[0017] The device operation configuration interface is used to provide an upgrade operation entry for the data processing plugins to obtain upgraded data processing plugins;

[0018] The human-computer interaction module is used to send the upgraded data processing plugins to the edge gateway service;

[0019] The edge gateway service is used to receive the upgraded data processing plugins and send the upgraded data processing plugins to the data processing service;

[0020] The data processing service is used to update the data processing process based on the upgraded data processing plugins and generate update feedback data to send to the edge gateway service;

[0021] The edge gateway service is used to receive the update feedback data and send the update feedback data to the cloud;

[0022] The cloud is used to obtain a plugin update result based on the update feedback data.

[0023] Optionally, the human-computer interaction module is used to provide a device debugging interface;

[0024] The device debugging interface is used to provide a debugging operation entry for the data processing plugin to obtain device debugging input data;

[0025] The human - machine interaction module is used to send the device debugging input data to the edge gateway service;

[0026] The edge gateway service is used to receive the device debugging input data and send the device debugging input data to the data processing service;

[0027] The data processing service is used to call the data processing plugin to process the device debugging input data, obtain device debugging feedback data, and send the device debugging feedback data to the edge gateway service;

[0028] The edge gateway service is used to receive the device debugging feedback data and send the device debugging feedback data to the cloud;

[0029] The cloud is used to obtain a device debugging result based on the device debugging feedback data.

[0030] Optionally, the data acquisition service is used to acquire the original operation data generated by the device end and send the original operation data to the data processing service;

[0031] The data processing service is used to call the data processing plugin to process the original operation data, obtain target operation data, and send the target operation data to the edge gateway service;

[0032] The edge gateway service is used to receive the target operation data and send the target operation data to the cloud.

[0033] Optionally, the data processing plugin includes device control logic;

[0034] The data acquisition service is used to acquire the original operation data generated by the device end and send the original operation data to the data processing service;

[0035] The data processing service is used to call the data processing plugin to analyze the original operation data to determine whether to trigger a device control command;

[0036] The data processing service is used to send the device control command to the data acquisition service when it is determined that the device control command needs to be triggered;

[0037] The data acquisition service is used to receive the device control command and send the device control command to the device end;

[0038] The device side is used to generate device control data based on the device control command;

[0039] The data acquisition service is further used to obtain the device control data and send the device control data to the data processing service;

[0040] The data processing service is used to obtain a device control result based on the device control data.

[0041] Optionally, the data processing service communicates with the data acquisition service and the edge gateway service based on the MQTT (Message Queuing Telemetry Transport) protocol.

[0042] Optionally, the cloud includes an EMQX (an open-source IoT message middleware) message middleware;

[0043] The cloud and the edge computing side communicate through the EMQX message middleware;

[0044] And / or, a device management service and a data storage service are running in the cloud;

[0045] And / or, the data processing service and the data processing plug-in are developed based on the Python (a machine language) language.

[0046] Optionally, the data processing plug-in includes an attribute reporting interface, an event reporting interface, an attribute writing interface, and a service call interface;

[0047] The data processing process update operation corresponds to the attribute writing interface and the service call interface;

[0048] The reported data processing operation corresponds to the attribute reporting interface and the event reporting interface;

[0049] The device offline debugging operation corresponds to the attribute writing interface and the service call interface;

[0050] The device operation control operation corresponds to the attribute reporting interface, the event reporting interface, the attribute writing interface, and the service call interface.

[0051] In a second aspect, a data processing method based on a plug-in is provided, and the data processing method is implemented by using the above-mentioned data processing system based on a plug-in;

[0052] The data processing method includes:

[0053] Running a data processing service and a data processing plug-in;

[0054] Implementing a data processing process update operation, a device offline debugging operation, a reported data processing operation, and a device operation control operation based on the data processing service and the data processing plug-in.

[0055] Optionally, the steps of implementing the data processing process update operation based on the data processing service and the data processing plugin include:

[0056] Obtain the upgraded data processing plugin corresponding to the data processing plugin;

[0057] Update the data processing process based on the upgraded data processing plugin and generate update feedback data;

[0058] Obtain the plugin update result based on the update feedback data;

[0059] Alternatively, the steps of implementing the device offline debugging operation based on the data processing service and the data processing plugin include:

[0060] Obtain the device debugging input data;

[0061] Call the data processing plugin to process the device debugging input data to obtain device debugging feedback data;

[0062] Obtain the device debugging result based on the device debugging feedback data;

[0063] Alternatively, the steps of implementing the reported data processing operation based on the data processing service and the data processing plugin include:

[0064] Obtain the original operation data generated by the device end;

[0065] Call the data processing plugin to process the original operation data to obtain the target operation data;

[0066] Alternatively, the steps of implementing the device operation control operation based on the data processing service and the data processing plugin include:

[0067] Obtain the original operation data generated by the device end;

[0068] Call the data processing plugin to analyze the original operation data to determine whether to trigger a device control command; when it is determined that the device control command needs to be triggered, send the device control command to the data acquisition service;

[0069] Receive the device control command and send the device control command to the device end;

[0070] Generate device control data based on the device control command and send the device control data to the data processing service;

[0071] Obtain the device control result based on the device control data.

[0072] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and adapted to run on the processor. When the processor executes the computer program, the above-described data processing method based on a plug-in is implemented.

[0073] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-described data processing method based on a plug-in is implemented.

[0074] In a fifth aspect, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the above-described data processing method based on a plug-in is implemented.

[0075] On the basis of conforming to the common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred examples of the present disclosure.

[0076] The positive and progressive effects of the present disclosure are as follows:

[0077] For the data processing system, method, device and medium based on a plug-in of the present disclosure, the data processing plug-in depends on the data processing service to run. The data processing service performs data processing based on the hot update technology and cache cleaning technology of the data processing plug-in, and realizes the update operation of the data processing process, the off-line debugging operation of the device, the reporting data processing operation and the device operation control operation; the present disclosure places the data processing service at the edge computing end, significantly reducing the data transmission delay and improving the real-time performance and response speed of the data processing system; relying on the hot update technology of the data processing plug-in, developers can update the plug-in code in real time without restarting the system, quickly respond to business requirements, and improve the development and debugging efficiency; through the data processing plug-in, data processing and conversion can be carried out flexibly, supporting the data formats and access methods of different devices, meeting the diverse customization needs of customers, reducing the cost and complexity of customized development, and providing an off-line debugging framework for device data, enabling developers to debug the data processing plug-in even without on-site devices, solving the problem of difficult coordination of on-site device startup, shortening the development cycle, and improving the development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 FIG. 1 is a first schematic structural diagram of a data processing system based on a plug-in provided in Embodiment 1 of the present disclosure;

[0079] Figure 2 FIG. 2 is a second schematic structural diagram of a data processing system based on a plug-in provided in Embodiment 1 of the present disclosure;

[0080] Figure 3Schematic diagram of the communication process between the data processing service, the data collection service, and the edge gateway service in the plugin-based data processing system provided in Embodiment 1 of the present disclosure;

[0081] Figure 4 Schematic diagram of implementing the data processing process update operation in the plugin-based data processing system provided in Embodiment 1 of the present disclosure;

[0082] Figure 5 Schematic diagram of implementing the device offline debugging operation in the plugin-based data processing system provided in Embodiment 1 of the present disclosure;

[0083] Figure 6 Schematic diagram of implementing the reported data processing operation in the plugin-based data processing system provided in Embodiment 1 of the present disclosure;

[0084] Figure 7 Schematic diagram of implementing the device operation control operation in the plugin-based data processing system provided in Embodiment 1 of the present disclosure;

[0085] Figure 8 The first process schematic diagram of the plugin-based data processing method provided in Embodiment 2 of the present disclosure;

[0086] Figure 9 The second process schematic diagram of the plugin-based data processing method provided in Embodiment 2 of the present disclosure;

[0087] Figure 10 Schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present disclosure. Detailed implementation manners

[0088] The present disclosure will be further described below by way of embodiments, but the present disclosure is not limited to the scope of the described embodiments.

[0089] In the embodiments of the present disclosure, prefix words such as "first" and "second" are only used to distinguish different described objects, and have no limiting effect on the position, order, priority, quantity, content, etc. of the described objects. The use of ordinal numbers and other prefix words for distinguishing described objects in the embodiments of the present disclosure does not constitute a limitation on the described objects. The description of the described objects refers to the description in the context of the embodiments, and should not constitute an unnecessary limitation due to the use of such prefix words. In addition, in the description of this embodiment, unless otherwise specified, the meaning of "a plurality" is two or more.

[0090] The following further explains the professional terms in the present disclosure:

[0091] Physical resources: include computing resources (CPU, GPU), storage resources (memory, hard disk), and network resources (bandwidth, latency).

[0092] Data transmission delay: It refers to the time interval from the generation of data to the completion of processing, which is particularly important for applications with high real-time requirements.

[0093] Data format: It refers to the organization and representation of data. Different devices may use different data formats, such as JSON, XML, CSV, etc.

[0094] Hot update: A plugin update technology that enables code changes to take effect without restarting the application service.

[0095] Edge computing: Bringing computing, storage, and network services closer to the data source or the user's location to reduce latency, improve response speed, and save bandwidth.

[0096] Node-RED: A flow-based development tool that allows developers to connect hardware devices by dragging and dropping and then process data.

[0097] Flink: An open-source distributed stream processing framework for real-time processing of large-scale data streams, supporting high-throughput and low-latency data processing.

[0098] Embodiment 1

[0099] This embodiment provides a plugin-based data processing system, which is applied to the industrial Internet of Things. As Figure 1 and Figure 2 shown, the data processing system includes a cloud 1, an edge computing side 2, and a device side 3;

[0100] The edge computing side 2 is respectively communicatively connected to the cloud 1 and the device side 3;

[0101] The cloud 1 includes a human-computer interaction module 11;

[0102] The edge computing side 2 includes a data processing plugin 21, and the edge computing side runs a data acquisition service 22, a data processing service 23, and an edge gateway service 24;

[0103] The data processing plugin 21 runs depending on the data processing service 23;

[0104] The data processing service 23 is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plugin 21, and implement data processing process update operations, device offline debugging operations, reported data processing operations, and device operation control operations.

[0105] The device side refers to the physical devices in the customer's factory, which can be any devices that can communicate with the edge computing side, such as sensors, injection molding machines, die casting machines, etc.

[0106] The cloud includes a human-computer interaction module, which can also be called a Web (webpage) console / management console, and includes functions such as device management, edge gateway management, template management, data query, and report display, facilitating relevant operations for developers, implementers, or customers.

[0107] The edge computing side can be an edge gateway (box), an industrial computer, an edge server, etc. The edge computing side mainly provides data collection services, data processing services, and edge gateway services. The data processing plugin depends on the data processing service to run.

[0108] The data collection service supports the collection of multiple data sources based on various industrial protocols. It can actively collect data from the device side or the device side can actively provide data to the data collection service. The data processing service mainly processes the collected device data and provides a data caching function to ensure the temporary storage and quick access of data. The edge gateway service mainly provides a unified gateway export service, serving as the only channel for data transmission to ensure data security and consistency, and supporting data encryption, data compression, and traffic control.

[0109] Since the data processing service is placed on the edge computing side to support edge computing, the data processing response speed is significantly faster than the solution of processing data on the cloud server, significantly reducing data transmission latency, controlling the data processing response time within 10 ms, and improving the real-time performance and response speed of the data processing system. The entire data processing service only occupies about 10 MB of memory and is suitable for deployment on a small edge gateway (i.e., the edge computing side).

[0110] In the data processing system based on plugins of the present disclosure, placing the data processing service on the edge computing side significantly reduces data transmission latency and improves the real-time performance and response speed of the data processing system. Since the data processing service has low resource occupancy, the hardware resource requirements are reduced; relying on the hot update technology of data processing plugins, developers can update the plugin code in real time without restarting the system, quickly respond to business requirements, and improve development and debugging efficiency; through data processing plugins, data processing and conversion can be flexibly performed, supporting different device data formats and access methods, meeting the diverse customization needs of customers, reducing the cost and complexity of customized development, and providing an offline debugging framework for device data, enabling developers to debug data processing plugins even without on-site devices, solving the problem of difficult coordination of on-site device startup, shortening the development cycle, and improving development efficiency.

[0111] In an optional implementation manner, as Figure 3 shown, the data processing service communicates with the data collection service and the edge gateway service based on the MQTT protocol.

[0112] In an optional implementation manner, asFigure 3 As shown in the figure, the data processing plug-in includes an attribute reporting interface, an event reporting interface, an attribute writing interface, and a service call interface;

[0113] The data processing process update operation corresponds to the attribute writing interface and the service call interface;

[0114] The reported data processing operation corresponds to the attribute reporting interface and the event reporting interface;

[0115] The device offline debugging operation corresponds to the attribute writing interface and the service call interface;

[0116] The device operation control operation corresponds to the attribute reporting interface, the event reporting interface, the attribute writing interface, and the service call interface.

[0117] The data processing service is an independent microservice that includes the framework for overall data processing and is deployed at the edge computing end, such as within an edge gateway. The data processing service communicates with the data collection service and the edge gateway service respectively through the MQTT protocol.

[0118] The data processing plug-in is a code segment that is required to implement some fixed interfaces. The data processing plug-in includes 4 standard interfaces: an attribute reporting interface, an event reporting interface, an attribute writing interface, and a service call interface, which are used to process the device data reported by the data collection service and sent from the cloud, implement the data processing process update operation, the reported data processing operation, the device offline debugging operation, and the device operation control operation, and meet the customer's customized data processing requirements and the edge device reverse control requirements.

[0119] The attribute reporting interface outputs and reports the collected attribute key-value pairs to the cloud after being processed by the plug-in script.

[0120] The event reporting interface outputs and reports the collected real-time events to the cloud after being processed by the plug-in script.

[0121] The attribute writing interface processes the attribute write command sent from the Web console (i.e., the aforementioned human-computer interaction module) through the plug-in script and then sends it to the device side, and returns the execution result to the cloud.

[0122] The service call processing processes the service call command sent from the Web console through the plug-in script and then sends it to the device side, and returns the execution result to the cloud.

[0123] Specifically, the data processing process update operation corresponds to the attribute writing interface and the service call interface; the reported data processing operation corresponds to the attribute reporting interface and the event reporting interface; the device offline debugging operation corresponds to the attribute writing interface and the service call interface; the device operation control operation corresponds to the attribute reporting interface, the event reporting interface, the attribute writing interface, and the service call interface.

[0124] In an optional implementation, the human-computer interaction module is used to provide a device operation configuration interface;

[0125] The device operation configuration interface is used to provide an upgrade operation entry for the data processing plugin to obtain the upgraded data processing plugin;

[0126] The human-computer interaction module is used to send the upgraded data processing plugin to the edge gateway service;

[0127] The edge gateway service is used to receive the upgraded data processing plugin and send the upgraded data processing plugin to the data processing service;

[0128] The data processing service is used to update the data processing process based on the upgraded data processing plugin and generate updated feedback data to send to the edge gateway service;

[0129] The edge gateway service is used to receive the updated feedback data and send the updated feedback data to the cloud;

[0130] The cloud is used to obtain the plugin update result based on the updated feedback data.

[0131] Specifically, as Figure 4 shown, the implementer modifies the plugin content of the data processing plugin of a certain device in the device operation configuration interface in the cloud, that is, updates the plugin content, and then initiates a synchronous plugin command to send / push the data processing plugin (i.e., the upgraded data processing plugin) to the edge gateway service via the industrial protocol, and then the edge gateway service forwards / pushes it to the data processing service. The data processing service applies the plugin hot reloading technology and the old module cache cleaning technology to perform plugin hot update. The data processing service will immediately reload all modules that use this plugin, clean up the old modules and reload the modules, and reload the processing logic. Without the need to republish the program corresponding to the data processing service or restart the program, for the subsequent reported data or issued commands, use the new plugin logic to calculate to achieve the update operation of the data processing process. At the same time, the feedback data is sent to the cloud via the edge gateway service in sequence, and the cloud obtains the plugin update result based on the updated feedback data, determines whether the plugin is successfully updated, realizes the update operation of the data processing process, and meets the customer's customized data processing requirements.

[0132] In an optional implementation, the human-computer interaction module is used to provide a device debugging interface;

[0133] The device debugging interface is used to provide a debugging operation entry for the data processing plugin to obtain device debugging input data;

[0134] The human-computer interaction module is used to send the device debugging input data to the edge gateway service;

[0135] The edge gateway service is used to receive device debugging input data and send the device debugging input data to the data processing service;

[0136] The data processing service is used to call a data processing plugin to process the device debugging input data, obtain device debugging feedback data, and send the device debugging feedback data to the edge gateway service;

[0137] The edge gateway service is used to receive the device debugging feedback data and send the device debugging feedback data to the cloud;

[0138] The cloud is used to obtain device debugging results based on the device debugging feedback data.

[0139] Specifically, as Figure 5 shown, the implementer writes a piece of simulated data in the device debugging interface of the cloud as the device debugging input data, uses the device debugging function to initiate a debugging command, sends the device debugging input data to the data processing service via the edge gateway service, and the data processing service will process the device debugging input data according to the established plugin processing logic to obtain the device debugging feedback data, and send the device debugging feedback data to the cloud via the edge gateway service. The cloud obtains the device debugging results based on the device debugging feedback data, and judges whether the offline debugging of the plugin is successful, so as to realize the device offline debugging operation.

[0140] In an optional implementation manner, the data acquisition service is used to obtain the original operation data generated by the device end and send the original operation data to the data processing service;

[0141] The data processing service is used to call a data processing plugin to process the original operation data, obtain the target operation data, and send the target operation data to the edge gateway service;

[0142] The edge gateway service is used to receive the target operation data and send the target operation data to the cloud.

[0143] The original operation data can be original operation attribute data and / or original operation data.

[0144] Property is used to describe the specific information and status during device operation. For example, the current ambient temperature read by an environmental monitoring device, the status of a smart light switch, the wind level of an electric fan, etc. Properties can be divided into two types: read-write and read-only. The read-write type supports reading and setting property values, and the read-only type only supports reading property values.

[0145] A service refers to instructions or methods that can be externally invoked by a device. Input and output parameters can be set during service invocation. Input parameters are the parameters during service execution, and output parameters are the results after service execution. Compared with attributes, a service can implement more complex business logics through one instruction, such as performing a specific task. Services are divided into two invocation methods: asynchronous and synchronous.

[0146] An event refers to information actively reported to the cloud when a device is running, generally including information, alarms, and faults that need to be perceived and processed externally. Multiple output parameters can be included in an event. For example, notification information after a certain task is completed; temperature and time information when a device fails; the operating status when a device alarms, etc. Events can be subscribed to and pushed.

[0147] Specifically, as Figure 6 shown, when the data processing service starts an asynchronous processing task, it will listen in real time for the original operation data reported by the data collection service through the MQTT protocol. The data can be actively collected from the device side or the device side can actively provide data to the data collection service. When the original operation data is received, before the original operation data flows to the edge gateway service, the corresponding data processing plugin will be called to process the data to obtain the target operation data, realizing customized processing of the reported attributes and events. The processed target operation data will be sent to the cloud via the edge gateway service for users to query or for meeting subsequent requirements such as report generation, realizing the operation of processing reported data.

[0148] In an optional implementation manner, the data processing plugin includes device control logic;

[0149] The data collection service is used to obtain the original operation data generated by the device side and send the original operation data to the data processing service;

[0150] The data processing service is used to call the data processing plugin to analyze the original operation data to determine whether to trigger a device control command;

[0151] The data processing service is used to send a device control command to the data collection service when it is determined that a device control command needs to be triggered;

[0152] The data collection service is used to receive the device control command and send the device control command to the device side;

[0153] The device side is used to generate device control data based on the device control command;

[0154] The data collection service is also used to obtain the device control data and send the device control data to the data processing service;

[0155] The data processing service is used to obtain device control results based on device control data.

[0156] Specifically, as Figure 7 shown, the implementer needs to write the device control logic (also known as the reverse control logic) in the plugin. Generally, a change in a certain attribute value is used as a signal to trigger a device control command. When the data processing service receives the original operation data reported by the data acquisition service, it immediately calls the corresponding plugin to make a trigger judgment to determine whether the trigger condition is met. If the trigger condition corresponding to the trigger logic is met, it directly initiates an attribute write or service call, and initiates a device control command at the edge computing end through the device control logic in the plugin, reducing the device control delay to less than 10 ms. At the same time, the device generates device control data based on the device control command. The data acquisition service obtains the device control data and sends the device control data to the data processing service. The data processing service obtains the device control result based on the device control data, and the data does not need to be transferred to the cloud, realizing the device operation control operation at the edge computing end.

[0157] In an optional implementation, as Figure 2 shown, the cloud includes the EMQX message middleware;

[0158] The cloud and the edge computing end communicate through the EMQX message middleware.

[0159] Using EMQX as the middleware for communication between the cloud and the edge computing end enables different devices at the device end to report data using independent topics (Topics), and they do not interfere with each other while reporting data with high concurrency.

[0160] In an optional implementation, as Figure 2 shown, the cloud runs a device management service and a data storage service.

[0161] The data storage service uses InfluxDB (an open-source time series database) to store the time series data reported by the edge computing end, stores the data content as the field (Field) data type, stores the device ID as the tag (Tag), and stores different data points (Points) in different measurement tables (Measurements) to accelerate data query.

[0162] The device management service supports full life cycle management such as device authentication, registration, configuration, monitoring, alarm, and data management.

[0163] In an optional implementation, the data processing service and the data processing plugin are developed based on the Python language.

[0164] The data processing service is an independent microservice developed in Python. It includes a framework for overall data processing and is deployed at the edge computing end, such as within an edge gateway. The data processing service communicates with the data acquisition service and the edge gateway service respectively through the MQTT protocol.

[0165] The data processing plugin is a Python code segment that requires implementing some fixed interfaces. The data processing plugin includes four standard interfaces: the attribute reporting interface, the event reporting interface, the attribute writing interface, and the service call interface, which are used to process the device data reported by the data acquisition service and sent from the cloud, and to implement operations such as updating the data processing process, reporting data processing operations, device offline debugging operations, and device operation control operations, meeting the customer's customized data processing requirements and the edge device reverse control requirements.

[0166] The data processing plugin code is implemented using the Python scripting language, which has a lower professional threshold and can thus reduce the development cost.

[0167] The data processing framework implemented in Python is much smaller in size than streaming processing frameworks such as Flink, reducing resource overhead.

[0168] Using the importlib (a function) module of Python to implement the hot update of the data processing plugin, developers can update the plugin code in real time without restarting the system, quickly respond to business requirements, and improve development and debugging efficiency.

[0169] The existing common ways to process industrial IoT device data include implementing through code customization development, implementing based on the Flink streaming framework, and implementing based on Node-Red.

[0170] This disclosure has the following advantages compared with the way of implementing through code customization development:

[0171] a. Lower development cost: Usually, backend developers use compiled languages such as C++ or Java to develop data processing services. The data processing plugin code of this disclosure is implemented using the Python scripting language, which has a lower professional threshold and can thus reduce the development cost.

[0172] b. Shorter development and debugging cycle: In the traditional code development mechanism, after developers modify the code, they need to package, deploy, and upgrade it to the site before further debugging, and this process will be repeated when problems occur. Thanks to the plugin hot update mechanism, this system supports users to modify the plugin code in the platform and then send it to the data processing service in real time and take effect immediately, thus reducing the development cycle.

[0173] This disclosure has the following advantages compared with the way of implementing based on the Flink streaming framework:

[0174] a. Lower resource occupancy: Usually, Flink is deployed on cloud servers and is suitable for IoT projects that collect and process data from servers with a large number of users. Its resource occupancy is very high, often occupying dozens or even hundreds of gigabytes of memory. For small and medium-sized industrial IoT projects, this is a complete waste of resources. The data processing service of the present disclosure only occupies about 10MB of memory, with a low resource occupancy rate.

[0175] b. Lower data processing latency: Since Flink is deployed on cloud servers, there is usually a physical latency of more than 100ms from data collection to sending to the server, which cannot meet the real-time requirements for some scenarios of device reverse control. The data processing service of the present disclosure is deployed at the edge computing end close to the device (such as an edge gateway). From collecting device data to completing data processing, it can be within 10ms, with lower data processing latency.

[0176] The present disclosure has the following advantages compared with the implementation method using the Node-RED processing framework:

[0177] a. Lower development cost: Node-RED requires developers to have the development ability of the JavaScript language, which is usually only familiar to front-end Web developers, imposing more restrictions on users. The data processing plugin of the present disclosure uses the Python language, which is more general and simple and can be directly written and implemented by implementers, thereby reducing the development cost.

[0178] b. Higher degree of customization: Due to the limitation of the pure drag-and-drop mode, Node-RED reduces its degree of customization and cannot meet some relatively complex data processing scenarios. The Python plugin of this system supports any form of extended code and supports synchronous and asynchronous device reverse control logic, making it more flexible and having a higher degree of customization.

[0179] Embodiment 2

[0180] This embodiment provides a plugin-based data processing method, and the data processing method is implemented using the plugin-based data processing system in Embodiment 1; as Figure 8 shown, the data processing method includes:

[0181] S1. Run the data processing service and the data processing plugin.

[0182] S2. Based on the data processing service and the data processing plugin, implement data processing process update operations, device offline debugging operations, reported data processing operations, and device operation control operations.

[0183] Specifically, the data processing system includes a cloud, an edge computing side, and a device side; the edge computing side is communicatively connected to the cloud and the device side respectively; the cloud includes a human-computer interaction module; the edge computing side includes a data processing plugin, and the edge computing side runs a data collection service, a data processing service, and an edge gateway service; the data processing plugin depends on the data processing service to run; the data processing service is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plugin, and implement data processing process update operations, device offline debugging operations, reported data processing operations, and device operation control operations.

[0184] The plugin-based data processing method of this embodiment is implemented by using the plugin-based data processing system in Embodiment 1. The working principle of the data processing method is the same as that of the data processing system. For specific details, please refer to the content disclosed in Embodiment 1, which will not be elaborated here.

[0185] In the plugin-based data processing method of this embodiment, the data processing plugin depends on the data processing service to run. The data processing service performs data processing based on the hot update technology and cache cleaning technology of the data processing plugin, and implements data processing process update operations, device offline debugging operations, reported data processing operations, and device operation control operations; placing the data processing service on the edge computing side significantly reduces data transmission latency, improves the real-time performance and response speed of the data processing system, and because the data processing service has low resource occupancy, it reduces the hardware resource requirements; relying on the hot update technology of the data processing plugin, developers can update the plugin code in real time without restarting the system, quickly respond to business requirements, and improve development and debugging efficiency; through the data processing plugin, data processing and conversion can be flexibly performed, supporting different device data formats and access methods, meeting the diverse customization needs of customers, reducing the cost and complexity of customized development, and providing a device data offline debugging framework, enabling developers to debug the data processing plugin even without on-site devices, solving the problem of difficult coordination of on-site device startup, shortening the development cycle, and improving development efficiency.

[0186] As Figure 9 shown, for the data processing process update operation, step S2 specifically includes:

[0187] S201. Obtain the upgraded data processing plugin corresponding to the data processing plugin.

[0188] S202. Update the data processing process based on the upgraded data processing plugin and generate update feedback data.

[0189] S203. Obtain the plugin update result based on the update feedback data.

[0190] Specifically, as Figure 4As shown in the figure, the implementer modifies the plug-in content of a data processing plug-in of a device in the device operation configuration interface on the cloud, that is, updates the plug-in content, and then initiates a synchronous plug-in command to send / push the data processing plug-in (i.e., the upgraded data processing plug-in) to the edge gateway service via the industrial protocol, and then the edge gateway service forwards / pushes it to the data processing service. The data processing service applies the plug-in hot reloading technology (Hot Reloading) and the old module cache cleaning technology (Invalidate Caches) to perform plug-in hot updates. The data processing service will immediately reload all modules that use this plug-in, clean up the old modules and reload the modules, and reload the processing logic. Without the need to republish the program corresponding to the data processing service or restart the program, for the subsequent reported data or issued commands, use the new plug-in logic to calculate, and implement the update operation of the data processing process. At the same time, the feedback data is sent to the cloud via the edge gateway service in sequence. The cloud obtains the plug-in update result based on the update feedback data, determines whether the plug-in is successfully updated, implements the update operation of the data processing process, and meets the customer's customized data processing requirements.

[0191] As Figure 9 shown, for the device offline debugging operation, step S2 specifically includes:

[0192] S204. Obtain device debugging input data.

[0193] S205. Call the data processing plug-in to process the device debugging input data to obtain device debugging feedback data.

[0194] S206. Obtain the device debugging result based on the device debugging feedback data.

[0195] Specifically, as Figure 5 shown, the implementer writes a piece of simulated data in the device debugging interface on the cloud as the device debugging input data, uses the device debugging function, initiates a debugging command, sends the device debugging input data to the data processing service via the edge gateway service. The data processing service will process the device debugging input data according to the established plug-in processing logic to obtain device debugging feedback data, and send the device debugging feedback data to the cloud via the edge gateway service. The cloud obtains the device debugging result based on the device debugging feedback data, and determines whether the plug-in offline debugging is successful, and implements the device offline debugging operation.

[0196] As Figure 9 shown, for the reported data processing operation, step S2 specifically includes:

[0197] S207. Obtain the original operation data generated by the device side.

[0198] S208. Invoke the data processing plugin to process the original operation data to obtain the target operation data.

[0199] Specifically, as Figure 6 shown, the data processing service starts an asynchronous processing task and will listen in real time for the original operation data reported by the data acquisition service through the MQTT protocol. It can actively collect data from the device side or the device side can actively provide data to the data acquisition service. When the original operation data is received, before the original operation data flows to the edge gateway service, the corresponding data processing plugin will be invoked to process the data to obtain the target operation data, realizing the customized processing of the reported attributes and events. The processed target operation data will be sent to the cloud via the edge gateway service for users to query or to meet subsequent requirements such as report generation, realizing the operation of processing the reported data.

[0200] As Figure 9 shown, for the device operation control operation, step S2 specifically includes:

[0201] S209. Obtain the original operation data generated by the device side.

[0202] S210. Invoke the data processing plugin to analyze the original operation data to determine whether a device control command is triggered; when it is determined that a device control command needs to be triggered, send the device control command to the data acquisition service.

[0203] S211. Receive the device control command and send the device control command to the device side.

[0204] S212. Generate device control data based on the device control command and send the device control data to the data processing service.

[0205] S213. Obtain the device control result based on the device control data.

[0206] Specifically, as Figure 7As shown in the figure, the implementer needs to write the device control logic (also known as the reverse control logic) in the plugin. Generally, a change in a certain attribute value is used as a signal to trigger a device control command. When the data processing service receives the original operation data reported by the data acquisition service, it immediately calls the corresponding plugin to make a trigger judgment to determine whether the trigger condition is met. If the trigger condition corresponding to the trigger logic is met, it directly initiates an attribute write or service call, and initiates a device control command at the edge computing end through the device control logic in the plugin, reducing the device control delay to less than 10 ms. At the same time, the device end generates device control data based on the device control command. The data acquisition service obtains the device control data and sends the device control data to the data processing service. The data processing service obtains the device control result based on the device control data, and the data does not need to be transferred to the cloud, realizing the device operation control operation at the edge computing end.

[0207] For the method embodiment, since it basically corresponds to the system embodiment, the relevant parts can refer to the partial description of the system embodiment, and the specific content will not be elaborated here.

[0208] Embodiment 3

[0209] Figure 10 The following is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored on the memory and configured to run on the processor. When the processor executes the computer program, it implements the data processing method based on the plugin in the above embodiment. Figure 10 The electronic device 80 shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0210] As Figure 10 shown, the electronic device 80 may be presented in the form of a general computing device, for example, it may be a server device. The components of the electronic device 80 may include, but are not limited to: at least one of the above processors 81, at least one of the above memories 82, and a bus 83 connecting different system components (including the memory 82 and the processor 81).

[0211] The bus 83 includes a data bus, an address bus, and a control bus.

[0212] The memory 82 may include volatile memory, such as random access memory (RAM) 821 and / or cache memory 822, and may further include read-only memory (ROM) 823.

[0213] The memory 82 may also include a program tool 825 (or utility) having a set (at least one) of program modules 824. Such program modules 824 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0214] The processor 81 executes various functional applications and data processing by running computer programs stored in the memory 82, such as the data processing method based on plug-ins provided in the above embodiments.

[0215] The electronic device 80 may also communicate with one or more external devices 84 (such as a keyboard, a pointing device, etc.). Such communication may be carried out through an input / output (I / O) interface 85. Moreover, the electronic device 80 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 86. As shown in the figure, the network adapter 86 communicates with other modules of the electronic device 80 through a bus 83. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in combination with the electronic device 80, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0216] It should be noted that, although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above may be embodied in one unit / modules. Conversely, the features and functions of one unit / modules described above may be further divided and embodied by multiple units / modules.

[0217] Embodiment 4

[0218] The embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the data processing method based on plug-ins provided in the above embodiments.

[0219] Among them, the more specific forms that the readable storage medium may adopt may include, but are not limited to: portable disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0220] Embodiment 5

[0221] Embodiments of the present disclosure also provide a computer program product, including a computer program which, when executed by a processor, implements the data processing method based on a plug-in provided in the above embodiments.

[0222] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on the user device, partially on the user device, executed as an independent software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0223] Although the specific embodiments of the present disclosure have been described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Without departing from the principles and essence of the present disclosure, those skilled in the art can make various changes or modifications to these embodiments, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A plug-in-based data processing system, characterized in that The described data processing system is applied to the industrial Internet of Things. The data processing system includes a cloud, an edge computing terminal, and a device terminal; The edge computing terminal is respectively communicatively connected to the cloud and the device terminal; The cloud includes a human-computer interaction module; The edge computing terminal includes a data processing plugin. The edge computing terminal runs a data acquisition service, a data processing service, and an edge gateway service; The data processing service is a microservice. The data processing service communicates with the data acquisition service and the edge gateway service respectively; The data processing plugin runs depending on the data processing service; The data processing service is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plugin, and realize data processing process update operations, device offline debugging operations, reported data processing operations, and device operation control operations; The human-computer interaction module is used to provide a device operation configuration interface; The device operation configuration interface is used to provide an upgrade operation entry for the data processing plugin to obtain an upgraded data processing plugin; The human-computer interaction module is used to send the upgraded data processing plugin to the edge gateway service; The edge gateway service is used to receive the upgraded data processing plugin and send the upgraded data processing plugin to the data processing service; The data processing service is used to update the data processing process based on the upgraded data processing plugin and generate update feedback data to send to the edge gateway service; The edge gateway service is used to receive the update feedback data and send the update feedback data to the cloud; The cloud is used to obtain a plugin update result based on the update feedback data to realize the data processing process update operation; The human-computer interaction module is used to provide a device debugging interface; The device debugging interface is used to provide a debugging operation entry for the data processing plugin to obtain device debugging input data; The human-computer interaction module is used to send the device debugging input data to the edge gateway service; The edge gateway service is used to receive the device debugging input data and send the device debugging input data to the data processing service; The data processing service is used to call the data processing plugin to process the device debugging input data, obtain device debugging feedback data, and send the device debugging feedback data to the edge gateway service; The edge gateway service is used to receive the device debugging feedback data and send the device debugging feedback data to the cloud; The cloud is used to obtain a device debugging result based on the device debugging feedback data to realize the device offline debugging operation; The data acquisition service is used to acquire the original operation data generated by the device terminal and send the original operation data to the data processing service; The data processing service is used to call the data processing plugin to process the original operation data, obtain target operation data, and send the target operation data to the edge gateway service; The edge gateway service is used to receive the target operation data and send the target operation data to the cloud to implement the reported data processing operation; The data processing plugin includes device control logic; The data acquisition service is used to obtain the original operation data generated by the device end and send the original operation data to the data processing service; The data processing service is used to call the data processing plugin to analyze the original operation data to determine whether to trigger a device control command; The data processing service is used to send the device control command to the data acquisition service when it is determined that the device control command needs to be triggered; The data acquisition service is used to receive the device control command and send the device control command to the device end; The device end is used to generate device control data based on the device control command; The data acquisition service is also used to obtain the device control data and send the device control data to the data processing service; The data processing service is used to obtain a device control result based on the device control data to implement the device operation control operation.

2. The data processing system according to claim 1, wherein The data processing service communicates with the data acquisition service and the edge gateway service based on the MQTT protocol.

3. The data processing system according to claim 1, wherein The cloud includes an EMQX message middleware; The cloud and the edge computing end communicate through the EMQX message middleware; And / or, a device management service and a data storage service are running on the cloud; And / or, the data processing service and the data processing plugin are developed based on the Python language.

4. The data processing system according to any one of claims 1-3, characterized in that, The data processing plugin includes an attribute reporting interface, an event reporting interface, an attribute writing interface, and a service call interface; The data processing process update operation corresponds to the attribute writing interface and the service call interface; The reported data processing operation corresponds to the attribute reporting interface and the event reporting interface; The device offline debugging operation corresponds to the attribute writing interface and the service call interface; The device operation control operation corresponds to the attribute reporting interface, the event reporting interface, the attribute writing interface, and the service call interface.

5. A data processing method based on plugins, characterized in that, The data processing method is implemented by using a plugin-based data processing system as described in any one of claims 1-4; The data processing method includes: Running a data processing service and a data processing plugin; Implementing a data processing process update operation, a device offline debugging operation, a reported data processing operation, and a device operation control operation based on the data processing service and the data processing plugin.

6. The data processing method according to claim 5, wherein The steps of implementing a data processing process update operation based on the data processing service and the data processing plugin include: Obtaining an upgraded data processing plugin corresponding to the data processing plugin; Updating the data processing process based on the upgraded data processing plugin and generating update feedback data; Obtaining a plugin update result based on the update feedback data; Or, the steps of implementing a device offline debugging operation based on the data processing service and the data processing plugin include: Obtaining device debugging input data; Calling the data processing plugin to process the device debugging input data to obtain device debugging feedback data; Derive the device debugging result based on the device debugging feedback data; Alternatively, the steps of implementing the reported data processing operation based on the data processing service and the data processing plugin include: Obtain the original operation data generated by the device end; Call the data processing plugin to process the original operation data to obtain the target operation data; Alternatively, the steps of implementing the device operation control operation based on the data processing service and the data processing plugin include: Obtain the original operation data generated by the device end; Call the data processing plugin to analyze the original operation data to determine whether to trigger the device control command; when it is determined that the device control command needs to be triggered, send the device control command to the data acquisition service; Receive the device control command and send the device control command to the device end; Generate device control data based on the device control command and send the device control data to the data processing service; Derive the device control result based on the device control data.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and adapted to run on the processor, characterized in that, When the processor executes the computer program, it implements the plugin-based data processing method according to claim 5 or 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the plugin-based data processing method according to claim 5 or 6.

9. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the plugin-based data processing method according to claim 5 or 6.

Images