Data processing system, method and equipment based on plug-in and medium

Abstract

The invention provides a plug-in-based data processing system, method and device and a medium. The data processing system comprises a cloud end, an edge computing end and a device end. The cloud comprises a man-machine interaction module; the edge computing end comprises a data processing plug-in and runs a data acquisition service, a data processing service and an edge gateway service; and the data processing service is used for carrying out data processing based on a hot update technology and a cache cleaning technology of the data processing plug-in, and realizing data processing process update operation, equipment off-line debugging operation, reported data processing operation and equipment operation control operation. According to the invention, the real-time performance and the response speed of the data processing system are improved; through the data processing plug-in, data processing and conversion can be flexibly carried out, data formats and access modes of different devices are supported, diversified customization requirements of customers are met, cost and complexity of customization development are reduced, and development efficiency and debugging efficiency are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of Internet of Things, and in particular to a plug-in-based data processing system, method, device and medium. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, more and more smart devices are widely used in various fields, such as industrial automation, smart cities, and medical health. The massive data generated by these devices needs to be analyzed and processed in real time or near real time to support quick decision-making. The traditional cloud computing model uploads all data to the cloud for processing, which not only increases the delay of network transmission, but also may cause cloud server overload due to excessive data volume. In order to solve these problems, edge computing came into being. By deploying computing resources close to the data source, localized data processing is achieved, effectively reducing data transmission delays and alleviating pressure on the cloud.

[0003] However, existing IoT device data processing solutions still face the following challenges:

[0004] 1. High hardware resource requirements: A large amount of physical resources (including computing resources, storage resources, and network resources) are required to support large-scale IoT device access and data processing, which places high demands on hardware infrastructure.

[0005] 2. High data processing delay: The delay in data transmission and processing affects the real-time and response speed of the system, especially in low-bandwidth or high-latency network environments, which requires optimization of data transmission and processing processes.

[0006] 3. High cost of customized development: Due to the different data formats and access methods of different devices, there are a lot of customized requirements to process the data reported by the devices, which requires support for flexible data processing and conversion mechanisms. Traditional code implementation methods or streaming processing methods based on Flink (an open source distributed stream processing framework) require quite professional developers to support, resulting in an increase in the overall cost of the project.

[0007] 4. Long development and debugging cycle: Traditional data processing development usually requires developers to modify the code and update the program to the site before it can be implemented. When problems arise and need debugging, this time-consuming process will be repeated, resulting in a long overall project development and debugging cycle. Summary of the invention

[0008] The technical problem to be solved by the present disclosure is to overcome the defects of high hardware resource requirements, high data processing delay, high customized development cost and long development and debugging cycle of equipment data processing in the existing industrial Internet of Things, and to provide a plug-in-based data processing system, method, device and medium.

[0009] The present invention solves the above technical problems through the following technical solutions:

[0010] In a first aspect, a plug-in-based data processing system is provided, the data processing system being applied to the industrial Internet of Things, the data processing system comprising a cloud, an edge computing end, and a device end;

[0011] The edge computing end is respectively connected to the cloud end and the device end in communication;

[0012] The cloud includes a human-computer interaction module;

[0013] The edge computing end includes a data processing plug-in, and the edge computing end runs a data collection service, a data processing service, and an edge gateway service;

[0014] The data processing plug-in depends on the data processing service to run;

[0015] The data processing service is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plug-in, and realize data processing process update operation, device offline debugging operation, reporting data processing operation and device operation control operation.

[0016] Optionally, the human-computer interaction module is used to provide a device operation configuration interface;

[0017] The device operation configuration interface is used to provide an upgrade operation entry for the data processing plug-in to obtain an upgraded data processing plug-in;

[0018] The human-computer interaction module is used to send the upgraded data processing plug-in to the edge gateway service;

[0019] The edge gateway service is used to receive the upgraded data processing plug-in and send the upgraded data processing plug-in to the data processing service;

[0020] The data processing service is used to update the data processing process based on the upgraded data processing plug-in, and generate update feedback data to send to the edge gateway service;

[0021] The edge gateway service is used to receive the update feedback data and send the update feedback data to the cloud;

[0022] The cloud end is used to obtain a plug-in update result based on the update feedback data.

[0023] Optionally, the human-computer interaction module is used to provide a device debugging interface;

[0024] The device debugging interface is used to provide a debugging operation entry for the data processing plug-in to obtain device debugging input data;

[0025] The human-computer interaction module is used to send the device debugging input data to the edge gateway service;

[0026] The edge gateway service is used to receive the device debugging input data and send the device debugging input data to the data processing service;

[0027] The data processing service is used to call the data processing plug-in to process the device debugging input data, obtain device debugging feedback data, and send the device debugging feedback data to the edge gateway service;

[0028] The edge gateway service is used to receive the device debugging feedback data and send the device debugging feedback data to the cloud;

[0029] The cloud end is used to obtain a device debugging result based on the device debugging feedback data.

[0030] Optionally, the data acquisition service is used to obtain the original operation data generated by the device end, and send the original operation data to the data processing service;

[0031] The data processing service is used to call the data processing plug-in to process the original operation data, obtain target operation data, and send the target operation data to the edge gateway service;

[0032] The edge gateway service is used to receive the target operation data and send the target operation data to the cloud.

[0033] Optionally, the data processing plug-in includes device control logic;

[0034] The data acquisition service is used to obtain the original operation data generated by the device end and send the original operation data to the data processing service;

[0035] The data processing service is used to call the data processing plug-in to analyze the original operation data to determine whether to trigger a device control command;

[0036] The data processing service is used to send the device control command to the data acquisition service when it is determined that the device control command needs to be triggered;

[0037] The data acquisition service is used to receive the device control command and send the device control command to the device end;

[0038] The device end is used to generate device control data based on the device control command;

[0039] The data acquisition service is further used to obtain the device control data and send the device control data to the data processing service;

[0040] The data processing service is used to obtain a device control result based on the device control data.

[0041] Optionally, the data processing service communicates with the data collection service and the edge gateway service based on the MQTT (Message Queue Telemetry Transport) protocol.

[0042] Optionally, the cloud includes EMQX (an open source IoT message middleware) message middleware;

[0043] The cloud and the edge computing end communicate via the EMQX message middleware;

[0044] And / or, the cloud runs device management services and data storage services;

[0045] And / or, the data processing service and the data processing plug-in are developed based on Python (a machine language).

[0046] Optionally, the data processing plug-in includes an attribute reporting interface, an event reporting interface, an attribute writing interface and a service calling interface;

[0047] The data processing process update operation corresponds to the attribute write interface and service call interface;

[0048] The reported data processing operation corresponds to an attribute reporting interface and an event reporting interface;

[0049] The device offline debugging operation corresponds to a property writing interface and a service calling interface;

[0050] The device operation control operation corresponds to an attribute reporting interface, an event reporting interface, an attribute writing interface and a service calling interface.

[0051] In a second aspect, a plug-in-based data processing method is provided, wherein the data processing method is implemented by the plug-in-based data processing system described above;

[0052] The data processing method comprises:

[0053] Run data processing services and data processing plug-ins;

[0054] Based on the data processing service and the data processing plug-in, data processing process update operation, equipment offline debugging operation, reporting data processing operation and equipment operation control operation are implemented.

[0055] Optionally, the step of implementing a data processing process update operation based on the data processing service and the data processing plug-in includes:

[0056] Obtain an upgraded data processing plug-in corresponding to the data processing plug-in;

[0057] Updating the data processing process based on the upgraded data processing plug-in and generating updated feedback data;

[0058] Deriving a plug-in update result based on the update feedback data;

[0059] Alternatively, the step of implementing the device offline debugging operation based on the data processing service and the data processing plug-in includes:

[0060] Get device debugging input data;

[0061] Calling a data processing plug-in to process the device debugging input data to obtain device debugging feedback data;

[0062] Deriving a device debugging result based on the device debugging feedback data;

[0063] Alternatively, the step of implementing the reporting data processing operation based on the data processing service and the data processing plug-in includes:

[0064] Obtain the original operation data generated by the device;

[0065] Calling the data processing plug-in to process the original operation data to obtain target operation data;

[0066] Alternatively, the step of implementing the device operation control operation based on the data processing service and the data processing plug-in includes:

[0067] Obtain the original operation data generated by the device;

[0068] Calling the data processing plug-in to analyze the original operation data to determine whether to trigger a device control command; when it is determined that the device control command needs to be triggered, sending the device control command to the data acquisition service;

[0069] receiving the device control command, and sending the device control command to the device end;

[0070] generating device control data based on the device control command, and sending the device control data to the data processing service;

[0071] A device control result is obtained based on the device control data.

[0072] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein the processor implements the above-mentioned plug-in-based data processing method when executing the computer program.

[0073] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the plug-in-based data processing method described above is implemented.

[0074] In a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the plug-in-based data processing method described above is implemented.

[0075] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.

[0076] The positive and progressive effects of this disclosure are:

[0077] The plug-in-based data processing system, method, device and medium disclosed in the present invention, the data processing plug-in relies on the data processing service to run, and the data processing service performs data processing based on the hot update technology and cache cleaning technology of the data processing plug-in, realizing data processing process update operation, device offline debugging operation, reporting data processing operation and device operation control operation; the present invention places the data processing service on the edge computing end, which significantly reduces the data transmission delay and improves the real-time and response speed of the data processing system; relying on the hot update technology of the data processing plug-in, developers can update the plug-in code in real time without restarting the system, quickly respond to business needs, and improve development and debugging efficiency; through the data processing plug-in, data processing and conversion can be flexibly performed, supporting data formats and access methods of different devices, meeting customers' diverse customization needs, reducing the cost and complexity of customized development, and providing a device data offline debugging framework, so that developers can debug the data processing plug-in even without on-site equipment, solving the problem of difficult coordination of on-site equipment startup, shortening the development cycle, and improving development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A first structural diagram of a plug-in-based data processing system provided in Embodiment 1 of the present disclosure;

[0079] Figure 2 A second structural diagram of a plug-in-based data processing system provided in Embodiment 1 of the present disclosure;

[0080] Figure 3A schematic diagram of the communication process between the data processing service, the data acquisition service and the edge gateway service in the plug-in-based data processing system provided in Embodiment 1 of the present disclosure;

[0081] Figure 4 A schematic diagram of implementing a data processing process update operation in a plug-in-based data processing system provided in Embodiment 1 of the present disclosure;

[0082] Figure 5 A schematic diagram of implementing device offline debugging operations in a plug-in-based data processing system provided in Embodiment 1 of the present disclosure;

[0083] Figure 6 A schematic diagram of implementing a reporting data processing operation in a plug-in-based data processing system provided in Embodiment 1 of the present disclosure;

[0084] Figure 7 A schematic diagram of implementing device operation control operations in a plug-in-based data processing system provided in Embodiment 1 of the present disclosure;

[0085] Figure 8 A first flow chart of a plug-in-based data processing method provided in Embodiment 2 of the present disclosure;

[0086] Fig. 9 A second flow chart of the plug-in-based data processing method provided in Embodiment 2 of the present disclosure;

[0087] Fig.10 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0088] The present disclosure is further described below by way of examples, but the present disclosure is not limited to the scope of the examples.

[0089] Prefixes such as "first" and "second" are used in the embodiments of the present disclosure only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present disclosure does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

[0090] The following is a further explanation of the professional terms in this disclosure:

[0091] Physical resources: including computing resources (CPU, GPU), storage resources (memory, hard disk) and network resources (bandwidth, latency).

[0092] Data transmission delay: refers to the time interval from the generation of data to the completion of processing, which is particularly important for applications with high real-time requirements.

[0093] Data format: refers to the way data is organized and represented. Different devices may use different data formats, such as JSON, XML, CSV, etc.

[0094] Hot update: A plug-in update technology that allows code changes to take effect without restarting the application service.

[0095] Edge computing: Bringing computing, storage, and network services closer to the data source or user to reduce latency, improve response time, and save bandwidth.

[0096] Node-RED: A flow-based development tool that allows developers to connect hardware devices and process data through drag-and-drop.

[0097] Flink: An open source distributed stream processing framework for real-time processing of large-scale data streams, supporting high-throughput and low-latency data processing.

[0098] Example 1

[0099] This embodiment provides a plug-in-based data processing system, which is applied to industrial Internet of Things, such as Figure 1 and Figure 2 As shown, the data processing system includes a cloud end 1, an edge computing end 2, and a device end 3;

[0100] The edge computing end 2 is connected to the cloud end 1 and the device end 3 for communication;

[0101] The cloud 1 includes a human-computer interaction module 11;

[0102] The edge computing end 2 includes a data processing plug-in 21, and the edge computing end runs a data collection service 22, a data processing service 23 and an edge gateway service 24;

[0103] The data processing plug-in 21 depends on the data processing service 23 to run;

[0104] The data processing service 23 is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plug-in 21, to implement data processing process update operations, device offline debugging operations, reporting data processing operations and device operation control operations.

[0105] The device side refers to the physical equipment in the customer's factory, which can be any device that can communicate with the edge computing side, such as sensors, injection molding machines, die-casting machines, etc.

[0106] The cloud includes a human-computer interaction module, which can also be called a Web console / management desk. It includes functions such as device management, edge gateway management, template management, data query, and report display, making it convenient for developers, implementers, or customers to perform related operations.

[0107] The edge computing end can be an edge gateway (box), industrial computer, edge server, etc. The edge computing end mainly provides data collection services, data processing services and edge gateway services. The data processing plug-in depends on the data processing service to run.

[0108] The data collection service supports the collection of multiple data sources based on various industrial protocols. It can be actively collected from the device side, or the device side can actively provide data to the data collection service. The data processing service mainly processes the collected device data and provides data caching functions to ensure temporary storage and fast access to data. The edge gateway service mainly provides a unified gateway export service as the only channel for data transmission, ensuring the security and consistency of data, and supporting data encryption, data compression, and flow control.

[0109] Since the data processing service is placed on the edge computing end and supports edge computing, the data processing response speed is significantly faster than the solution of processing data on the cloud server, which significantly reduces the data transmission delay and controls the data processing response time within 10ms, improving the real-time performance and response speed of the data processing system. The entire data processing service only occupies about 10MB of memory and is suitable for deployment on a small edge gateway (i.e., edge computing end).

[0110] The plug-in-based data processing system disclosed in the present invention places the data processing service on the edge computing end, significantly reduces the data transmission delay, improves the real-time performance and response speed of the data processing system, and reduces the hardware resource requirements due to the low resource occupancy of the data processing service. Relying on the hot update technology of the data processing plug-in, developers can update the plug-in code in real time without restarting the system, quickly respond to business needs, and improve development and debugging efficiency. Through the data processing plug-in, data processing and conversion can be flexibly performed, supporting data formats and access methods of different devices, meeting customers' diverse customized needs, reducing the cost and complexity of customized development, and providing an offline debugging framework for device data, so that developers can debug the data processing plug-in even without on-site equipment, solving the problem of difficult coordination of on-site equipment startup, shortening the development cycle, and improving development efficiency.

[0111] In an optional embodiment, if Figure 3 As shown, the data processing service communicates with the data collection service and the edge gateway service based on the MQTT protocol.

[0112] In an optional embodiment, if Figure 3 As shown, the data processing plug-in includes an attribute reporting interface, an event reporting interface, an attribute writing interface, and a service calling interface;

[0113] The data processing process updates the corresponding attribute writing interface and service calling interface;

[0114] Reporting data processing operations corresponds to the attribute reporting interface and event reporting interface;

[0115] Device offline debugging operations correspond to the attribute writing interface and service calling interface;

[0116] The device operation control operations correspond to the attribute reporting interface, event reporting interface, attribute writing interface and service calling interface.

[0117] The data processing service is an independent microservice that includes the overall data processing framework and is deployed on the edge computing side, such as the edge gateway. The data processing service communicates with the data collection service and the edge gateway service respectively through the MQTT protocol.

[0118] The data processing plug-in is a code segment that requires the implementation of some fixed interfaces. The data processing plug-in contains four standard interfaces: attribute reporting interface, event reporting interface, attribute writing interface, and service call interface. It is used to process device data reported by the data collection service and sent from the cloud, and implement data processing process update operations, reported data processing operations, device offline debugging operations, and device operation control operations to meet customers' customized data processing needs and edge device reverse control needs.

[0119] The attribute reporting interface processes the collected attribute key-value pairs through the plug-in script and then outputs them to the cloud.

[0120] The event reporting interface processes the collected real-time events through plug-in scripts and then outputs them to the cloud.

[0121] The attribute writing interface processes the attribute writing commands sent from the Web console (i.e. the aforementioned human-computer interaction module) through the plug-in script and sends them to the device, and returns the execution results to the cloud.

[0122] Service call processing is to process the service call commands issued by the Web console through the plug-in script and then send them to the device, and return the execution results to the cloud.

[0123] Specifically, the data processing process update operation corresponds to the attribute writing interface and the service calling interface; the reporting data processing operation corresponds to the attribute reporting interface and the event reporting interface; the device offline debugging operation corresponds to the attribute writing interface and the service calling interface; the device operation control operation corresponds to the attribute reporting interface, the event reporting interface, the attribute writing interface and the service calling interface.

[0124] In an optional implementation, the human-computer interaction module is used to provide a device operation configuration interface;

[0125] The device operation configuration interface is used to provide an upgrade operation entry for the data processing plug-in to obtain the upgraded data processing plug-in;

[0126] The human-computer interaction module is used to send the upgraded data processing plug-in to the edge gateway service;

[0127] The edge gateway service is used to receive the upgraded data processing plug-in and send the upgraded data processing plug-in to the data processing service;

[0128] The data processing service is used to update the data processing process based on the upgraded data processing plug-in and generate updated feedback data to send to the edge gateway service;

[0129] The edge gateway service is used to receive update feedback data and send the update feedback data to the cloud;

[0130] The cloud is used to derive plug-in update results based on update feedback data.

[0131] Specifically, Figure 4 As shown in the figure, the implementation personnel modify the plug-in content of the data processing plug-in of a certain device in the device operation configuration interface of the cloud, that is, update the plug-in content, and then initiate the synchronization plug-in command, send / push the data processing plug-in (that is, the upgraded data processing plug-in) to the edge gateway service via the industrial protocol, and then forward / push it to the data processing service by the edge gateway service. The data processing service applies the plug-in hot update technology (Hot Reloading) and the old module cache cleaning technology (Invalidate Caches) to perform plug-in hot update. The data processing service will immediately reload all modules using the plug-in, clean up the old modules and reload the modules, reload the processing logic, and use the new plug-in logic to calculate the subsequent reported data or issued commands without re-publishing the corresponding program of the data processing service or restarting the program, so as to realize the data processing process update operation. At the same time, the feedback data is sent to the cloud via the edge gateway service in turn. The cloud obtains the plug-in update result based on the updated feedback data, determines whether the plug-in is successfully updated, realizes the data processing process update operation, and meets the customized data processing requirements of customers.

[0132] In an optional implementation, the human-computer interaction module is used to provide a device debugging interface;

[0133] The device debugging interface is used to provide a debugging operation entry for the data processing plug-in to obtain device debugging input data;

[0134] The human-computer interaction module is used to send device debugging input data to the edge gateway service;

[0135] The edge gateway service is used to receive device debugging input data and send the device debugging input data to the data processing service;

[0136] The data processing service is used to call the data processing plug-in to process the device debugging input data, obtain the device debugging feedback data, and send the device debugging feedback data to the edge gateway service;

[0137] The edge gateway service is used to receive device debugging feedback data and send the device debugging feedback data to the cloud;

[0138] The cloud is used to obtain device debugging results based on device debugging feedback data.

[0139] Specifically, Figure 5 As shown, the implementer writes a piece of simulation data as the device debugging input data in the device debugging interface of the cloud, uses the device debugging function, initiates the debugging command, and sends the device debugging input data to the data processing service via the edge gateway service. The data processing service processes the device debugging input data according to the established plug-in processing logic to obtain the device debugging feedback data, and sends the device debugging feedback data to the cloud via the edge gateway service. The cloud obtains the device debugging result based on the device debugging feedback data, determines whether the plug-in offline debugging is successful, and implements the device offline debugging operation.

[0140] In an optional implementation, the data acquisition service is used to obtain the original operation data generated by the device end and send the original operation data to the data processing service;

[0141] The data processing service is used to call the data processing plug-in to process the original operation data, obtain the target operation data, and send the target operation data to the edge gateway service;

[0142] The edge gateway service is used to receive target operation data and send the target operation data to the cloud.

[0143] The raw operation data may be raw operation attribute data and / or raw operation data.

[0144] Properties are used to describe specific information and status of a device during operation. For example, the current ambient temperature, smart light switch status, fan wind speed level, etc. read by an environmental monitoring device. Properties can be divided into two types: read-write and read-only. The read-write type supports reading and setting property values, while the read-only type only supports reading property values.

[0145] Service refers to the instructions or methods that a device can call externally. Input and output parameters can be set in service calls. Input parameters are the parameters when the service is executed, and output parameters are the results after the service is executed. Compared with attributes, services can implement more complex business logic through a single instruction, such as executing a specific task. Services are divided into two calling modes: asynchronous and synchronous.

[0146] Event refers to the information that the device actively reports to the cloud when it is running. It generally includes information, alarms, and failures that need to be sensed and processed externally. An event can contain multiple output parameters. For example, notification information after a task is completed; temperature and time information when a device fails; operating status when a device alarms, etc. Events can be subscribed and pushed.

[0147] Specifically, Figure 6 As shown in the figure, the data processing service starts the asynchronous processing task and monitors the original operation data reported by the data collection service in real time through the MQTT protocol. It can be actively collected from the device side, or the device side can actively provide data to the data collection service. When the original operation data is received, the corresponding data processing plug-in will be called to process the data before the original operation data flows to the edge gateway service to obtain the target operation data and realize customized processing of the reported attributes and events. The processed target operation data will be sent to the cloud via the edge gateway service for user query or to meet the needs of subsequent report generation, so as to realize the reported data processing operation.

[0148] In an optional embodiment, the data processing plug-in includes device control logic;

[0149] The data collection service is used to obtain the original operation data generated by the device and send the original operation data to the data processing service;

[0150] The data processing service is used to call the data processing plug-in to analyze the original operation data to determine whether to trigger the device control command;

[0151] The data processing service is used to send a device control command to the data collection service when it is determined that a device control command needs to be triggered;

[0152] The data collection service is used to receive device control commands and send them to the device end;

[0153] The device side is used to generate device control data based on the device control command;

[0154] The data collection service is also used to obtain device control data and send the device control data to the data processing service;

[0155] The data processing service is used to derive device control results based on device control data.

[0156] Specifically, Figure 7 As shown, the implementer needs to write the device control logic (also called reverse control logic) in the plug-in, which generally uses a change in a certain attribute value as a signal to trigger a device control command. When the data processing service receives the original operation data reported by the data acquisition service, it immediately calls the corresponding plug-in to make a trigger judgment to determine whether the trigger condition is met. If the trigger logic corresponding to the trigger condition is met, it directly initiates the attribute write or service call, and initiates the device control command at the edge computing end through the device control logic in the plug-in, reducing the device control delay to less than 10ms. At the same time, the device end generates device control data based on the device control command, and the data acquisition service obtains the device control data and sends the device control data to the data processing service. The data processing service obtains the device control result based on the device control data. The data does not need to be transferred to the cloud, realizing the device operation control operation at the edge computing end.

[0157] In an optional embodiment, if Figure 2 As shown, the cloud includes EMQX message middleware;

[0158] The cloud and edge computing end communicate through the EMQX message middleware.

[0159] EMQX is used as the middleware for communication between the cloud and edge computing, so that different devices on the device side report data using independent topics, and report data with high concurrency without interfering with each other.

[0160] In an optional embodiment, if Figure 2 As shown, the cloud runs device management services and data storage services.

[0161] The data storage service uses InfluxDB (an open source time series database) to store time series data reported by the edge computing end, stores the data content as the field data type, stores the device ID as a tag, and stores data points (Point) in different formats in different measurement tables (Measurements) to speed up data query.

[0162] Device management services support full life cycle management including device authentication, registration, configuration, monitoring, alarm, and data management.

[0163] In an optional implementation, the data processing service and the data processing plug-in are developed based on the Python language.

[0164] The data processing service is an independent microservice developed in Python. It contains the overall data processing framework and is deployed on the edge computing side, such as the edge gateway. The data processing service communicates with the data collection service and the edge gateway service respectively through the MQTT protocol.

[0165] The data processing plug-in is a Python code segment that requires the implementation of some fixed interfaces. The data processing plug-in contains four standard interfaces: attribute reporting interface, event reporting interface, attribute writing interface, and service call interface. It is used to process device data reported by the data collection service and sent from the cloud, and implement data processing process update operations, reported data processing operations, device offline debugging operations, and device operation control operations to meet customers' customized data processing needs and edge device reverse control needs.

[0166] The data processing plug-in code is implemented using the Python scripting language, which has a lower professional threshold and can reduce development costs.

[0167] The data processing framework implemented in Python is much smaller than streaming processing frameworks such as Flink, reducing resource overhead.

[0168] By using Python's importlib (a function) module to implement hot updates of data processing plug-ins, developers can update plug-in codes in real time without restarting the system, quickly respond to business needs, and improve development and debugging efficiency.

[0169] The existing common industrial IoT device data processing methods include code customization development implementation, Flink streaming framework implementation, and Node-Red implementation.

[0170] Compared with the code customization development implementation, this disclosure has the following advantages:

[0171] a. Lower development costs: Back-end developers usually use compiled languages ​​such as C++ or Java to develop data processing services. The data processing plug-in code disclosed in this disclosure is implemented using the Python scripting language, which has a lower professional threshold and can reduce development costs.

[0172] b. Shorter development and debugging cycle: The traditional code development mechanism requires developers to modify the code, package and deploy it to the site before further debugging can be done. This process will be repeated when problems arise. Thanks to the plug-in hot update mechanism, this system supports users to modify the plug-in code in the platform, and send it to the data processing service in real time and take effect immediately, thereby reducing the development cycle.

[0173] Compared with the implementation based on Flink streaming framework, this disclosure has the following advantages:

[0174] a. Lower resource usage: Flink is usually deployed on cloud servers and is suitable for IoT projects with large-scale server collection and processing. Its resource usage is very high, with tens or hundreds of GB of memory usage, which is a complete waste of resources for small and medium-sized industrial IoT projects. The data processing service disclosed in this disclosure only occupies about 10MB of memory, with a low resource usage rate.

[0175] b. Lower data processing delay: Since Flink is deployed on cloud servers, there is usually a physical delay of more than 100ms from data collection to sending to the server, which cannot meet the real-time requirements for some device reverse control scenarios. The data processing service disclosed in this disclosure is deployed on the edge computing end close to the device (such as an edge gateway), and the data collection and processing of device data can be completed within 10ms, which reduces the data processing delay.

[0176] Compared with the implementation using the Node-RED processing framework, the present disclosure has the following advantages:

[0177] a. Lower development cost: Node-RED requires developers to have JavaScript language development capabilities, which is usually only familiar to front-end Web developers, which places more restrictions on users. The data processing plug-in disclosed in this disclosure uses Python language, which is more universal and simple, and can be directly written and implemented by implementers, thereby reducing development costs.

[0178] b. Higher degree of customizability: Due to the limitation of pure drag-and-drop mode, Node-RED has reduced its degree of customizability and cannot meet the needs of some more complex data processing scenarios. The Python plug-in of this system supports any form of extension code and synchronous and asynchronous device reverse control logic, which is more flexible and more customizable.

[0179] Example 2

[0180] This embodiment provides a plug-in-based data processing method, which is implemented using the plug-in-based data processing system in Embodiment 1; Figure 8 As shown, the data processing method includes:

[0181] S1. Run data processing services and data processing plug-ins.

[0182] S2. Based on the data processing service and data processing plug-in, data processing process update operation, equipment offline debugging operation, reporting data processing operation and equipment operation control operation are implemented.

[0183] Specifically, the data processing system includes a cloud side, an edge computing side and a device side; the edge computing side is communicated with the cloud side and the device side respectively; the cloud side includes a human-computer interaction module; the edge computing side includes a data processing plug-in, and the edge computing side runs data collection services, data processing services and edge gateway services; the data processing plug-in depends on the data processing service to run; the data processing service is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plug-in, to realize data processing process update operations, device offline debugging operations, reporting data processing operations and device operation control operations.

[0184] The plug-in-based data processing method of this embodiment is implemented using the plug-in-based data processing system in Example 1. The data processing method has the same working principle as the data processing system. For details, please refer to the contents disclosed in Example 1, which will not be repeated here.

[0185] In the plug-in-based data processing method of this embodiment, the data processing plug-in relies on the operation of the data processing service, and the data processing service performs data processing based on the hot update technology and cache cleaning technology of the data processing plug-in to implement data processing process update operations, device offline debugging operations, reporting data processing operations and device operation control operations; placing the data processing service on the edge computing end significantly reduces data transmission delays, improves the real-time and response speed of the data processing system, and has low resource occupancy for the data processing service, reducing hardware resource requirements; relying on the hot update technology of the data processing plug-in, developers can update the plug-in code in real time without restarting the system, quickly respond to business needs, and improve development and debugging efficiency; through the data processing plug-in, data processing and conversion can be flexibly performed, supporting data formats and access methods of different devices, meeting customers' diverse customization needs, reducing the cost and complexity of customized development, and providing a device data offline debugging framework, so that developers can debug the data processing plug-in even without on-site equipment, solving the problem of difficult coordination of on-site equipment startup, shortening the development cycle, and improving development efficiency.

[0186] like Fig. 9 As shown, for the data processing process update operation, the above step S2 specifically includes:

[0187] S201. Obtain an upgraded data processing plug-in corresponding to the data processing plug-in.

[0188] S202: Update the data processing process based on the upgraded data processing plug-in, and generate update feedback data.

[0189] S203: Obtaining a plug-in update result based on the update feedback data.

[0190] Specifically, Figure 4As shown in the figure, the implementation personnel modify the plug-in content of the data processing plug-in of a certain device in the device operation configuration interface of the cloud, that is, update the plug-in content, and then initiate the synchronization plug-in command, send / push the data processing plug-in (that is, the upgraded data processing plug-in) to the edge gateway service via the industrial protocol, and then forward / push it to the data processing service by the edge gateway service. The data processing service applies the plug-in hot update technology (Hot Reloading) and the old module cache cleaning technology (Invalidate Caches) to perform plug-in hot update. The data processing service will immediately reload all modules using the plug-in, clean up the old modules and reload the modules, reload the processing logic, and use the new plug-in logic to calculate the subsequent reported data or issued commands without re-publishing the corresponding program of the data processing service or restarting the program, so as to realize the data processing process update operation. At the same time, the feedback data is sent to the cloud via the edge gateway service in turn. The cloud obtains the plug-in update result based on the updated feedback data, determines whether the plug-in is successfully updated, realizes the data processing process update operation, and meets the customized data processing requirements of customers.

[0191] like Fig. 9 As shown, for the device offline debugging operation, the above step S2 specifically includes:

[0192] S204: Obtain device debugging input data.

[0193] S205: Calling a data processing plug-in to process the device debugging input data to obtain device debugging feedback data.

[0194] S206: Obtaining a device debugging result based on the device debugging feedback data.

[0195] Specifically, Figure 5 As shown, the implementer writes a piece of simulation data as the device debugging input data in the device debugging interface of the cloud, uses the device debugging function, initiates the debugging command, and sends the device debugging input data to the data processing service via the edge gateway service. The data processing service processes the device debugging input data according to the established plug-in processing logic to obtain the device debugging feedback data, and sends the device debugging feedback data to the cloud via the edge gateway service. The cloud obtains the device debugging result based on the device debugging feedback data, determines whether the plug-in offline debugging is successful, and implements the device offline debugging operation.

[0196] like Fig. 9 As shown, for the reporting data processing operation, the above step S2 specifically includes:

[0197] S207: Obtain original operation data generated by the device.

[0198] S208. Calling a data processing plug-in to process the original operation data to obtain target operation data.

[0199] Specifically, Figure 6 As shown in the figure, the data processing service starts the asynchronous processing task and monitors the original operation data reported by the data collection service in real time through the MQTT protocol. It can be actively collected from the device side, or the device side can actively provide data to the data collection service. When the original operation data is received, the corresponding data processing plug-in will be called to process the data before the original operation data flows to the edge gateway service to obtain the target operation data and realize customized processing of the reported attributes and events. The processed target operation data will be sent to the cloud via the edge gateway service for user query or to meet the needs of subsequent report generation, so as to realize the reported data processing operation.

[0200] like Fig. 9 As shown, for the equipment operation control operation, the above step S2 specifically includes:

[0201] S209: Obtain original operation data generated by the device.

[0202] S210, calling the data processing plug-in to analyze the original operation data to determine whether to trigger a device control command; when it is determined that the device control command needs to be triggered, sending the device control command to the data acquisition service.

[0203] S211. Receive a device control command, and send the device control command to the device end.

[0204] S212: Generate device control data based on the device control command, and send the device control data to the data processing service.

[0205] S213. Obtaining a device control result based on the device control data.

[0206] Specifically, Figure 7As shown, the implementer needs to write the device control logic (also called reverse control logic) in the plug-in, which generally uses a change in a certain attribute value as a signal to trigger a device control command. When the data processing service receives the original operation data reported by the data acquisition service, it immediately calls the corresponding plug-in to make a trigger judgment to determine whether the trigger condition is met. If the trigger logic corresponding to the trigger condition is met, it directly initiates the attribute write or service call, and initiates the device control command at the edge computing end through the device control logic in the plug-in, reducing the device control delay to less than 10ms. At the same time, the device end generates device control data based on the device control command, and the data acquisition service obtains the device control data and sends the device control data to the data processing service. The data processing service obtains the device control result based on the device control data. The data does not need to be transferred to the cloud, realizing the device operation control operation at the edge computing end.

[0207] As for the method embodiment, since it basically corresponds to the system embodiment, the relevant parts may refer to the partial description of the system embodiment, and the specific contents will not be repeated here.

[0208] Example 3

[0209] Fig.10 This is a structural schematic diagram of an electronic device showing an example embodiment of the present disclosure, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor, and when the processor executes the computer program, the plug-in-based data processing method in the above embodiment is implemented. Fig.10 The electronic device 80 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0210] like Fig.10 As shown, the electronic device 80 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 80 may include, but are not limited to: at least one processor 81, at least one memory 82, and a bus 83 connecting different system components (including the memory 82 and the processor 81).

[0211] The bus 83 includes a data bus, an address bus, and a control bus.

[0212] The memory 82 may include a volatile memory, such as a random access memory (RAM) 821 and / or a cache memory 822 , and may further include a read-only memory (ROM) 823 .

[0213] The memory 82 may also include a program tool 825 (or utility) having a set (at least one) of program modules 824, such program modules 824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0214] The processor 81 executes various functional applications and data processing by running the computer programs stored in the memory 82, such as the plug-in-based data processing method provided in the above embodiment.

[0215] The electronic device 80 can also communicate with one or more external devices 84 (e.g., keyboards, pointing devices, etc.). This communication can be performed through an input / output (I / O) interface 85. In addition, the electronic device 80 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter 86. As shown in the figure, the network adapter 86 communicates with other modules of the electronic device 80 through a bus 83. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 80, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0216] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0217] Example 4

[0218] The embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the plug-in-based data processing method provided in the above embodiments is implemented.

[0219] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0220] Example 5

[0221] The embodiments of the present disclosure also provide a computer program product, including a computer program, which implements the plug-in-based data processing method provided in the above embodiments when executed by a processor.

[0222] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0223] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A plug-in-based data processing system, characterized in that: The data processing system is applied to the industrial Internet of Things, and the data processing system includes a cloud end, an edge computing end, and a device end; The edge computing end is respectively connected to the cloud end and the device end in communication; The cloud includes a human-computer interaction module; The edge computing end includes a data processing plug-in, and the edge computing end runs a data collection service, a data processing service, and an edge gateway service; The data processing plug-in depends on the data processing service to run; The data processing service is used to perform data processing based on the hot update technology and cache cleaning technology of the data processing plug-in, and realize data processing process update operation, device offline debugging operation, reporting data processing operation and device operation control operation.

2. The data processing system according to claim 1, characterized in that The human-computer interaction module is used to provide a device operation configuration interface; The device operation configuration interface is used to provide an upgrade operation entry for the data processing plug-in to obtain an upgraded data processing plug-in; The human-computer interaction module is used to send the upgraded data processing plug-in to the edge gateway service; The edge gateway service is used to receive the upgraded data processing plug-in and send the upgraded data processing plug-in to the data processing service; The data processing service is used to update the data processing process based on the upgraded data processing plug-in, and generate update feedback data to send to the edge gateway service; The edge gateway service is used to receive the update feedback data and send the update feedback data to the cloud; The cloud end is used to obtain a plug-in update result based on the update feedback data.

3. The data processing system according to claim 1, characterized in that: The human-computer interaction module is used to provide a device debugging interface; The device debugging interface is used to provide a debugging operation entry for the data processing plug-in to obtain device debugging input data; The human-computer interaction module is used to send the device debugging input data to the edge gateway service; The edge gateway service is used to receive the device debugging input data and send the device debugging input data to the data processing service; The data processing service is used to call the data processing plug-in to process the device debugging input data, obtain device debugging feedback data, and send the device debugging feedback data to the edge gateway service; The edge gateway service is used to receive the device debugging feedback data and send the device debugging feedback data to the cloud; The cloud end is used to obtain a device debugging result based on the device debugging feedback data.

4. The data processing system according to claim 1, characterized in that: The data acquisition service is used to obtain the original operation data generated by the device end and send the original operation data to the data processing service; The data processing service is used to call the data processing plug-in to process the original operation data, obtain target operation data, and send the target operation data to the edge gateway service; The edge gateway service is used to receive the target operation data and send the target operation data to the cloud.

5. The data processing system according to claim 1, characterized in that: The data processing plug-in includes device control logic; The data acquisition service is used to obtain the original operation data generated by the device end and send the original operation data to the data processing service; The data processing service is used to call the data processing plug-in to analyze the original operation data to determine whether to trigger a device control command; The data processing service is used to send the device control command to the data acquisition service when it is determined that the device control command needs to be triggered; The data acquisition service is used to receive the device control command and send the device control command to the device end; The device end is used to generate device control data based on the device control command; The data acquisition service is further used to obtain the device control data and send the device control data to the data processing service; The data processing service is used to obtain a device control result based on the device control data.

6. The data processing system according to claim 1, characterized in that: The data processing service communicates with the data collection service and the edge gateway service based on the MQTT protocol.

7. The data processing system according to claim 1, characterized in that: The cloud includes EMQX message middleware; The cloud and the edge computing end communicate via the EMQX message middleware; And / or, the cloud runs device management services and data storage services; And / or, the data processing service and the data processing plug-in are developed based on Python language.

8. The data processing system according to any one of claims 1 to 7, characterized in that: The data processing plug-in includes an attribute reporting interface, an event reporting interface, an attribute writing interface and a service calling interface; The data processing process update operation corresponds to the attribute write interface and service call interface; The reported data processing operation corresponds to an attribute reporting interface and an event reporting interface; The device offline debugging operation corresponds to a property writing interface and a service calling interface; The device operation control operation corresponds to an attribute reporting interface, an event reporting interface, an attribute writing interface and a service calling interface.

9. A plug-in-based data processing method, characterized in that: The data processing method is implemented by a plug-in-based data processing system as described in any one of claims 1 to 8; The data processing method comprises: Run data processing services and data processing plug-ins; Based on the data processing service and the data processing plug-in, data processing process update operation, equipment offline debugging operation, reporting data processing operation and equipment operation control operation are implemented.

10. The data processing method according to claim 9, characterized in that: The steps of implementing the data processing process update operation based on the data processing service and the data processing plug-in include: Obtain an upgraded data processing plug-in corresponding to the data processing plug-in; Updating the data processing process based on the upgraded data processing plug-in and generating updated feedback data; Deriving a plug-in update result based on the update feedback data; Alternatively, the step of implementing the device offline debugging operation based on the data processing service and the data processing plug-in includes: Get device debugging input data; Calling a data processing plug-in to process the device debugging input data to obtain device debugging feedback data; Deriving a device debugging result based on the device debugging feedback data; Alternatively, the step of implementing the reporting data processing operation based on the data processing service and the data processing plug-in includes: Obtain the original operation data generated by the device; Calling the data processing plug-in to process the original operation data to obtain target operation data; Alternatively, the step of implementing the device operation control operation based on the data processing service and the data processing plug-in includes: Obtain the original operation data generated by the device; Calling the data processing plug-in to analyze the original operation data to determine whether to trigger a device control command; when it is determined that the device control command needs to be triggered, sending the device control command to the data acquisition service; receiving the device control command, and sending the device control command to the device end; generating device control data based on the device control command, and sending the device control data to the data processing service; A device control result is obtained based on the device control data.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, the plug-in-based data processing method according to claim 9 or 10 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the plug-in-based data processing method according to claim 9 or 10 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the plug-in-based data processing method according to claim 9 or 10 is implemented.

Images