Method, system, and device for distributed device access, control, and data transmission

Abstract

The present invention discloses a method, a system and a device for distributed device access, control and data transmission. The method includes the following steps: separating the management platform from the working nodes and uniformly managing the distributed deployment scheme; constructing a technical scheme framework of core service + components; realizing dynamic publishing, real-time scheduling, load balancing and abnormal switching of different tasks; integrating professional AI skills in various industries in the form of a structured proxy engine and converting original unstructured data into structured data; providing a retransmission mechanism and ensuring the avoidance of data omission and loss in case of structured exceptions. The present invention realizes a core service + component architecture, separates the docking of highly customized components with the business from the framework, decouples the access business from the core service, and at the same time provides a unified and visual component warehouse and task scheduling management, which can flexibly expand various tasks and meet various types of data docking.

Description

Technical Field

[0001] The present invention relates to the fields of edge computing and the Internet of Things. Specifically, it relates to methods, systems, and devices for distributed device access, control, and data transmission. Background Art

[0002] With the wide application of the Internet of Things and AI technologies in cities, the coverage area of edge sensing devices and the amount of collected data have increased exponentially. More and more sensing data, such as face, body, human behavior actions, items, motor vehicles, non-motor vehicles, location information, WiFi, Rfid (Radio Frequency Identification Technology), access control, temperature and humidity, power, and other business information, have been widely applied to the field of public safety. However, existing resources have problems such as a large number of devices, various types, difficult to unified management and control, and the inability to collect, structurally process, clean, and customize and share various business information simultaneously. There is an urgent need to structurally process and converge each sensing device and data to achieve centralized management of devices and data.

[0003] Similar implementation solutions in the prior art are as follows:

[0004] 1. Adopt a solution of access framework + plug-in. The framework takes the data exchange engine as the core, and the plug-in is the data access and sharing middleware. The framework handles some technical problems of high-speed data exchange such as concurrency and caching, provides a simple interface to interact with the plug-in, and the plug-in transports the converted data into heterogeneous systems to achieve high-speed exchange and sharing of enterprise heterogeneous data.

[0005] 2. Through a solution of data source adapter + kettle platform and ETL tool + task scheduling engine + target end adapter, data access and sharing are achieved. The kettle platform and ETL tool are used to make incremental judgments through timestamps, flag bits, and log analysis, and use data filtering, data routing, and data association plug-ins to process data. The task scheduling engine is used to complete the configuration and execution of the entire task.

[0006] 3. Through a solution of customized program versions, data access and sharing are achieved. When accessing and sharing front-end devices or platforms of different manufacturers, a program version that maps one-to-one with their protocols, data fields, and dictionary values is developed. Even in some cases, for devices or platforms of the same manufacturer but different models, different program versions need to be developed.

[0007] Disadvantages of the existing methods:

[0008] Solution 1:

[0009] It has a single function, poor generality and flexibility, and cannot quickly adapt to the access of different data types. It is unable to visually manage the plug-ins (a large number of plug-ins) and their configuration parameters uniformly, and dynamically configure the parameters of each plug-in to take effect in real time.

[0010] Solution 2:

[0011] The access methods are limited, only supporting the access and sharing of data such as databases, web services, and files, and unable to uniformly and integratively connect and control perception devices such as face capture cameras, vehicle capture cameras, alarm hosts, access controls, and video devices.

[0012] Solution 3:

[0013] The development and maintenance costs are high, and the reuse rate is low. Each time a new data source is accessed, it brings work in multiple aspects such as development, debugging, and operation and maintenance.

[0014] At the same time, the common disadvantages of Solution 1, Solution 2, and Solution 3:

[0015] The cost of troubleshooting problems is high, and there is a lack of effective means to timely discover abnormal problems in the system and data. When accessing and sharing data of heterogeneous devices (or systems), it is impossible to prove innocence in a timely manner. Especially on the internal network, it is not remotely accessible, it is difficult to locate problems with each other, and the speed of promoting and solving problems is slow.

[0016] Data statistics is difficult and work efficiency is low. There is a lack of systematic and multi-dimensional display of the current data access and sharing situation, making it difficult for customers to clearly understand the data quality and sharing degree of each current device (including third-party platforms and systems).

[0017] There is a lack of a process for converting real-time unstructured data into structured data and sharing it.

[0018] In response to the problems in the related technologies, no effective solution has been proposed yet. Summary of the Invention

[0019] In response to the problems in the related technologies, the present invention proposes a method, system, and device for distributed device access, control, and data transmission to overcome the above-mentioned technical problems existing in the existing related technologies.

[0020] To this end, the specific technical solutions adopted by the present invention are as follows:

[0021] According to one aspect of the present invention, there is provided a method for distributed device access, control, and data transmission, the method comprising the following steps:

[0022] S1. Separate the management platform from the working nodes, and run a number of working nodes in parallel, while uniformly managing the distributed deployment scheme;

[0023] S2. Build the technical solution framework of the core service + components on the working node, and use the core service as the center;

[0024] S3. On the working node and based on KeepAlived, task scheduling engine, ETCD and component management service, implement dynamic publishing, real-time scheduling, load balancing and exception switching of different tasks;

[0025] S4. Integrate professional AI skills in each industry by means of the structured proxy engine in the core service, convert the original unstructured data into structured data, and realize parallel transmission of the original unstructured data and secondary structured data;

[0026] S5. Use the structured proxy engine to provide a retransmission mechanism and ensure that data omission and loss are avoided in case of structured exceptions.

[0027] Furthermore, the components include access components and shared components, and each component corresponds to a task.

[0028] Furthermore, the management platform communicates with the working node using the Message Queuing Telemetry Transport Protocol to achieve reliable data transmission in low-bandwidth and unstable network environments;

[0029] The working node actively registers with the management platform, and the management platform provides device control instructions and management distribution of task scheduling to the working node, and performs visualization processing according to the information reported by the working node;

[0030] The working node completes the reallocation and service scheduling of each task according to the tasks issued by the management platform, and loads, starts, shuts down and unloads the access components or shared components corresponding to the tasks through the component management service.

[0031] Furthermore, the access component actively attempts to connect to the device according to the assigned task message. If the connection is successful within the specified number of connection attempts and time, it actively requests data or subscribes to data according to the protocol process of the device, decodes the returned data, and uploads the qualified data to the topic of Data-Hub at the same time;

[0032] If the connection is not successful, it will no longer interact with the corresponding device;

[0033] The access component uploads the data generated during the connection process to Data-Hub according to the agreed data logging format and data logging matters, records the abnormal situations generated during the upload process in the log file, and reports them to the management platform at the same time.

[0034] Furthermore, the core service cleans the data subscribed to the Data-Hub according to predefined rules and republishes the cleansed data to the topic of the Data-Hub;

[0035] The predefined rules include the following steps:

[0036] The cleaning rule chains for the data to be cleaned are configured in advance on the management platform. Each cleaning rule chain contains several rule nodes, and a complete data transmission process is formed by selecting the rule nodes.

[0037] After the configuration is completed, the message will be automatically persisted in a JSON format string and sent to the working node under the management of the task scheduler;

[0038] The core service encapsulates each rule node and entity business object into several Actors. When the management platform sends a rule engine message, the working node parses the sent cleaning rule chain information in real time and dynamically loads and starts the corresponding Actors.

[0039] The core service subscribes to each piece of data received in the Data Hub, and delivers it to each actor for data cleansing according to the order of the cleansing rule chain. The cleansed data is then sent back to the business topic of the Data Hub.

[0040] The core service reports the metadata and cleaned data to the storage platform according to the storage platform address issued by the management platform, and reports the task status of the working node cleaning data to the management platform.

[0041] Furthermore, the sharing component shares the data accessed or cleaned in the Data-Hub to different platforms or systems according to the data sharing engine rules. At the same time, the data generated during sharing is recorded in log files and reported according to the agreed tracking format and tracking items.

[0042] The data sharing engine rules include the following steps:

[0043] Configure the sharing rule chain for data sharing on the management platform. After the configuration is completed, the message will be automatically persisted in a JSON format string and sent to the working node through the task scheduling management.

[0044] The core service parses the data sharing rule engine message sent by the management platform, and the corresponding sharing component actively attempts to connect to the platform to be shared based on the task information sent, and determines whether the data channel to be shared is normal;

[0045] If the connection is successful within the specified number of connection attempts and time, the data transfer channel is normal and the next step is carried out. If not, the next step is not carried out and the exception log information is reported to the management platform;

[0046] The core service dynamically loads and starts the corresponding Actor according to the data sharing rule engine message, and delivers the data subscribed on the Data-Hub to each Actor for cleaning in the order of the sharing rule chain, and delivers the cleaned data to the sharing component;

[0047] The sharing component encapsulates and reports the cleaned data to the protocol platform, and reports the task status of sharing data by the working node to the management platform.

[0048] Further, the dynamic publishing, real-time scheduling, load balancing and exception switching of different tasks based on KeepAlived, task scheduling engine, ETCD and component management service in the working node include the following steps:

[0049] Map the cluster virtual IP and local server IP based on KeepAlived, detect the local task scheduling engine service, and at the same time, based on ETCD, provide mutual registration, mutual discovery, health monitoring and task sharing among nodes in the cluster;

[0050] Use the task scheduling engine service to parse, split and sort the message tasks sent by the management platform, and re-publish the message tasks to ETCD. At the same time, report the health status of each microservice of the local node obtained to the management platform;

[0051] The component management service subscribes to the task messages on the ETCD service and detects whether it is necessary to pull and load components from the remote component repository;

[0052] If not, directly start the corresponding components locally, and each component automatically controls the device and processes data according to the issued configuration messages;

[0053] View the data content accessed or shared by each component and the running status of each component in real time, quickly judge whether the components meet the production environment requirements, and perform visual verification and debugging on the components.

[0054] Further, the integration of professional AI skills in each industry by using the structured proxy engine in the core service, converting the original unstructured data into structured data, and realizing the parallel transmission of the original unstructured data and the secondary structured data; using the structured proxy engine to provide a retransmission mechanism and ensuring the avoidance of data omission and loss in case of structured exceptions include the following steps:

[0055] The structured proxy engine automatically adapts the assigned task messages to the structured engines of AI algorithm vendors, and issues and analyzes the original unstructured data obtained from the Data-Hub through the structured engine interfaces of the algorithm vendors;

[0056] If the recognition is successful during the analysis, the result set is parsed according to the predefined rules, and if it meets the business requirements, it is published to the structured data topic in the Data-Hub;

[0057] If the recognition is not successful, this original unstructured data is discarded and logged. And if there is an abnormal issue or recognition failure, this original unstructured data is cached to the secondary recognition and retransmission topic in the Data-Hub, and re-subscribed. At the same time, the recognition task is re-issued within the predefined time until the recognition is successful within the specified time or number of times.

[0058] According to another aspect of the present invention, there is provided a distributed device access, control, and data transmission system, which includes a node deployment module, a framework construction module, a task processing module, and a data structure conversion module;

[0059] Among them, the node deployment module is used to separate the management platform from the working nodes, run several working nodes in parallel, and uniformly manage the distributed deployment solution;

[0060] The framework construction module is used to construct a technical solution framework of core service + components on the working nodes, with the core service as the center;

[0061] The task processing module is used to implement dynamic publishing, real-time scheduling, load balancing, and exception switching of different tasks on the working nodes based on KeepAlived, a task scheduling engine, ETCD, and a component management service;

[0062] The data structure conversion module is used to integrate professional AI skills in various industries in the way of the structured proxy engine in the core service, convert the original unstructured data into structured data, and realize parallel transmission of the original unstructured data and the secondary structured data; use the structured proxy engine to provide a retransmission mechanism, and ensure that data omission and loss are avoided in case of structured exceptions.

[0063] According to yet another aspect of the present invention, there is provided a distributed device access, control, and data transmission device for implementing a distributed device access, control, and data transmission method.

[0064] The beneficial effects of the present invention are:

[0065] (1) The present invention realizes a core service + component architecture, separates the high-customization component docking service from the framework, decouples the access service from the core service, and at the same time provides a unified and visual component repository and task scheduling management, which can flexibly expand various tasks and meet various types of data docking. It solves the problem that device data of video, picture, and text types cannot be uniformly integrated and collected, and the connection and control problems of a large number of devices that support standard protocols such as Modbus, MQTT, BLE, OPC, Onvif, Rtsp, GB28181, GB1399 / 1400, JT 808, etc. or other custom protocols.

[0066] (2) Through the component debugging function, component developers only need to focus on the customized development of their docking services and can perform data interaction verification with the platform locally. After passing the verification, they can be seamlessly integrated into the production environment. Especially in the intranet or remote environment, it greatly improves the component development efficiency.

[0067] (3) By reporting access details and sharing details, as well as burying points, analyzing, and reporting exception logs, visual data operation and maintenance are realized. Statistical data information is obtained from multiple dimensions such as device data tracking, data reconciliation, real-time alarm, task link monitoring, and structuring, enabling users to clearly understand the device data access quality and data sharing degree of the current heterogeneous system at a glance, and improving the work efficiency of users. It solves the problems that users generally have headaches about, such as abnormal edge perception devices, inaccurate data collection, inconsistent sharing, and the problem that they cannot be discovered in time.

[0068] (4) Through the structured proxy engine, it quickly adapts to the structured engines of various algorithm manufacturers, converts the accessed unstructured data into the structured data required by users, and can share it with the original unstructured data to third parties as needed. It solves the problems of users' automatic adaptation of algorithm models for video and picture data, as well as data retry and retransmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0070] Figure 1 is a flowchart of a method for distributed device access, control, and data transmission according to an embodiment of the present invention;

[0071] Figure 2 is a schematic diagram of the relationship between a management platform and a working node according to an embodiment of the present invention;

[0072] Figure 3 is the principle block diagram of the management platform and the working nodes in the embodiments of the present invention;

[0073] Figure 4 is the schematic diagram when realizing dynamic release, real-time scheduling, load balancing and abnormal switching of different tasks according to the embodiments of the present invention;

[0074] Figure 5 is the timing diagram of task distribution according to the embodiments of the present invention;

[0075] Figure 6 is the timing diagram of data preview according to the embodiments of the present invention;

[0076] Figure 7 is the flowchart when converting unstructured data into structured data according to the embodiments of the present invention. Specific Embodiments

[0077] To further illustrate the embodiments, the present invention provides drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be combined with the relevant descriptions in the specification to explain the operation principle of the embodiments. With reference to these contents, those of ordinary skill in the art should be able to understand other possible implementation manners and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are usually used to represent similar components.

[0078] According to the embodiments of the present invention, a method, a system and a device for distributed device access, control and data transmission are provided. The management platform provides visual device management, rule engine, real-time alarm, link tracking, multi-dimensional data statistics, and component warehouse management, etc. The working nodes automatically perform task load balancing according to the configuration tasks issued by the management platform, pull and start the corresponding components, and realize automatic extraction, verification, conversion and data sharing, abnormal alarm, and abnormal switching of the data of the access devices. At the same time, it also supports visual configuration of node information on the working nodes using an App or a Web page, as well as configuration and upload of component information on the nodes.

[0079] Now, the present invention will be further described in combination with the drawings and specific embodiments. As Figure 1 shown, according to an embodiment of the present invention, a method for distributed device access, control and data transmission is provided. The method includes the following steps:

[0080] As Figure 2 shown, S1. Separate the management platform from the working nodes, and run several working nodes in parallel, and at the same time, uniformly manage the distributed deployment solution; that is, it supports single-node mode deployment and also supports cluster mode deployment;

[0081] AsFigure 3 As shown in Figure 3 , S2. Build a technical solution framework for core services + components on the working node, with the core service as the center, including DATA-HUB (a processing platform for streaming data), a task scheduling engine, data preview, data cleaning, log analysis, and a structured proxy engine;

[0082] In one embodiment, the components include access components and shared components, and each component corresponds to a task. Each task is associated with a target data source and a series of filtering, verification, and conversion, and is a complete operating unit.

[0083] In one embodiment, the management platform and the working node communicate using the Message Queuing Telemetry Transport (MQTT) protocol to achieve reliable data transmission in low-bandwidth and unstable network environments. The management platform corresponds to the MQTT server, and the working node is the MQTT client;

[0084] The working node actively registers with the management platform, reports details of device access and data sharing, abnormal log information, and requests to preview the data stream of the device on the current working node; and the management platform provides device control instructions and management distribution of task scheduling to the working node, and performs visual processing based on the information reported by the working node, including data report statistics, alarm display, link tracing, real-time preview, etc.;

[0085] The working node completes the reallocation of each task and service scheduling according to the task messages sent by the management platform, and loads, starts, shuts down, and unloads the access components or shared components corresponding to the relevant tasks through the ComponentManager.

[0086] In one embodiment, the access component actively attempts to connect to the device (or system) according to the assigned task message. If the connection is successful within the specified number of connection attempts and time, it actively requests data or subscribes to data according to the protocol process of the device, decodes the returned data, and uploads the qualified data to the topic of Data-Hub at the same time;

[0087] If the connection is not successful, it will no longer interact with the corresponding device;

[0088] The access component uploads the data generated during the connection process to Data-Hub according to the agreed-upon data logging format and data logging matters, including connecting to the device, requesting data, subscribing to data, and decoding data, and records the abnormal situations generated during the upload process in the log file and reports them to the management platform at the same time.

[0089] In one embodiment, the core service cleans the data subscribed and accessed in the Data-Hub according to predefined rules, and republishes the cleaned data to the topics in the Data-Hub;

[0090] Among them, the predefined rules include the following steps:

[0091] Pre-configure the cleaning rule chain for the data to be cleaned on the management platform in advance. Each cleaning rule chain contains several rule nodes, and each rule node corresponds to a corresponding service (such as transformation, filtering, alarm, alarm cancellation, delay, statistics, forwarding, etc.). At the same time, a complete data transmission process is formed by selecting rule nodes;

[0092] After the configuration is completed, the message is automatically persisted as a JSON-formatted string and sent to the worker node under the management of task scheduling;

[0093] The core service adopts the Actor model design concept. The core service encapsulates each rule node and entity business object into several Actors. Each Actor has its own dedicated processing logic. When the management platform issues a rule engine message, the worker node parses the issued cleaning rule chain information in real time, and dynamically loads and starts the corresponding Actor. Among them, the Actor model is a conceptual model used to handle concurrent computing. It defines a set of general rules on how a series of system components should act and interact. An Actor refers to a basic computing unit that can receive a message and perform calculations based on it. Actors communicate by sending messages. The transmission of messages is asynchronous and is processed through a mail queue. Each Actor is completely independent and can execute their operations simultaneously.

[0094] The core service subscribes to each piece of data that has been accessed in the Data-Hub, and hands it over to each Actor for data cleaning in the order of the cleaning rule chain, and returns the cleaned data to the business topic in the Data-Hub;

[0095] The core service reports the metadata and the cleaned data to the storage platform according to the storage platform address issued by the management platform, and at the same time reports the task status of the worker node for cleaning data to the management platform.

[0096] In one embodiment, the shared component shares the data accessed or cleaned in the Data-Hub to different platforms or systems according to the data sharing engine rules, and at the same time, according to the agreed data logging format and data logging matters, records the data generated during sharing into a log file and reports it, including the abnormal situations and statistical details generated during sharing;

[0097] Among them, the data sharing engine rules include the following steps:

[0098] Configure a shared rule chain for the data to be shared on the management platform, select the data types, data fields, and conditions to be met for sharing. Multiple shared rule nodes can be selected simultaneously in a rule chain, such as Redis nodes, Kafka nodes, MQTT nodes, RabbitMQ nodes, Rest API protocol platforms, GB / T 28181 protocol platforms, GB 1399 / 1400 protocol platforms, JT808 protocol platforms, etc. After configuration, the message is automatically persisted as a JSON-formatted string and sent to the worker nodes through task scheduling management;

[0099] The core service parses the data sharing rule engine messages sent by the management platform, and the scheduling component management service (ComponentManager) loads and starts the corresponding shared components. And the corresponding shared components actively attempt to connect to the platform to be shared according to the sent task information, and determine whether the data channels to be shared are normal;

[0100] If the connection is successful within the specified number of connection attempts and time, the data transfer channel is normal, and the next step is performed. If not, the next step is not performed, and the exception log information is reported to the management platform;

[0101] The core service dynamically loads and starts the corresponding Actors according to the data sharing rule engine messages, and delivers the data subscribed on the Data-Hub to each Actor for cleaning in the order of the shared rule chain, and delivers the cleaned data to the shared components;

[0102] The shared components encapsulate the cleaned data according to the specific requirements of the component protocol and report it to the protocol platform. If an exception occurs during reporting, such as reporting an exception, its exception information is captured and written to the log, and an attempt is made to re-upload. If it still cannot be uploaded within the specified number of attempts and time, it is abandoned, and the task status of the worker node sharing the data is reported to the management platform.

[0103] As Figures 4 - 6 shown, S3. On the worker nodes and based on KeepAlived (a service software that ensures high availability of the cluster in cluster management), task scheduling engine, ETCD (an open-source, distributed key-value data storage system), and component management service, dynamic publishing, real-time scheduling, load balancing, and exception switching of different tasks are implemented;

[0104] In one embodiment, the implementation of dynamic publishing, real-time scheduling, load balancing, and exception switching of different tasks on the worker nodes and based on KeepAlived, task scheduling engine, ETCD, and component management service includes the following steps:

[0105] Based on the open-source service KeepAlived, map the cluster virtual IP and the local server IP, and detect the local task scheduling engine service. At the same time, based on the open-source service ETCD, provide mutual registration, mutual discovery, health monitoring, and task sharing among nodes in the cluster;

[0106] Use the task scheduling engine service (ClusterScheduling) to parse the message tasks sent by the management platform, split and organize the tasks according to the resource utilization on the existing nodes, and republish the message tasks to ETCD to achieve task load balancing on each node. At the same time, report the health status of each microservice on the local node to the management platform;

[0107] The component management service subscribes to the task messages on the ETCD service and detects whether it is necessary to pull and load components from the remote component repository;

[0108] If not, directly start the corresponding components locally, and each component automatically performs device control and data processing according to the sent configuration messages, including device connection, data access, data decoding, verification, data conversion, data sharing, exception logging, and sending device control instructions (such as switch, clock synchronization, volume size, temperature, pan-tilt direction, etc.);

[0109] Users can view the data content accessed or shared by each component and the operation status of each component in real time on the data preview interface of the mobile terminal (App) or the PC terminal, quickly judge whether the components meet the requirements of the production environment, and perform visual verification and debugging on the components.

[0110] S4. Integrate professional AI skills in each industry by using the structured proxy engine in the core service, convert the original unstructured data into structured data, and achieve parallel transmission of the original unstructured data and the secondary structured data; prevent the original data from being polluted due to abnormal algorithms of third-party AI manufacturers or data interruption due to abnormal structured engines;

[0111] As Figure 7 shown, S5. Use the structured proxy engine to provide a retransmission mechanism and ensure that data omission and loss are avoided in case of structured exceptions;

[0112] In one embodiment, integrating professional AI skills in each industry by using the structured proxy engine in the core service, converting the original unstructured data into structured data, and achieving parallel transmission of the original unstructured data and the secondary structured data; using the structured proxy engine to provide a retransmission mechanism and ensuring that data omission and loss are avoided in case of structured exceptions includes the following steps:

[0113] The structured proxy engine automatically adapts the assigned task messages to the structured engines of AI algorithm vendors, and issues and analyzes the original unstructured data obtained from the Data-Hub through the structured engine interfaces of the algorithm vendors;

[0114] If the recognition is successful during the analysis, the result set is parsed according to the predefined rules, and if it meets the business requirements, it is published to the structured data topic in the Data-Hub;

[0115] If the recognition is unsuccessful, this piece of original unstructured data is discarded and logged. And if there is an exception during the issuance or the recognition fails, this piece of original unstructured data is cached to the secondary recognition and retransmission topic in the Data-Hub, and re-subscribed. At the same time, the recognition task is re-issued within the predefined time until the recognition is successful within the specified time or number of times, otherwise it is discarded.

[0116] According to another embodiment of the present invention, a system for distributed device access, control, and data transmission is provided. The system includes a node deployment module, a framework construction module, a task processing module, and a data structure conversion module;

[0117] Among them, the node deployment module is used to separate the management platform from the working nodes, run several working nodes in parallel, and uniformly manage the distributed deployment solution;

[0118] The framework construction module is used to construct a technical solution framework of core service + components, with the core service as the center;

[0119] The task processing module is used to realize the dynamic issuance, real-time scheduling, load balancing, and exception switching of different tasks based on KeepAlived, task scheduling engine, ETCD, and component management service;

[0120] The data structure conversion module is used to integrate professional AI skills in various industries in the way of a structured proxy engine, convert the original unstructured data into structured data, and realize the parallel transmission of the original unstructured data and the secondary structured data; provide a retransmission mechanism and ensure that data omission and loss are avoided in case of structured exceptions.

[0121] According to yet another embodiment of the present invention, a device for distributed device access, control, and data transmission is provided, implementing a method for distributed device access, control, and data transmission.

[0122] In summary, the present invention realizes a core service + component architecture, separates the highly customized component docking service from the framework, decouples the access service from the core service, and at the same time provides a unified and visual component repository and task scheduling management, which can flexibly expand various tasks and meet various types of data docking. It solves the problem that device data of video, picture, and text types cannot be uniformly integrated and collected, as well as the connection and control problems of a large number of devices that support standard protocols such as Modbus, MQTT, BLE, OPC, Onvif, Rtsp, GB28181, GB1399 / 1400, JT 808 or other custom protocols. Through the component debugging function, component developers only need to focus on the customized development of their docking services and can perform data interaction verification between the local and the platform. After passing the verification, they can be seamlessly integrated into the production environment. Especially in the intranet or remote environment, it greatly improves the component development efficiency. By reporting access details and sharing details, as well as burying points, analyzing, and reporting exception logs, visual data operation and maintenance are realized, and data information is statistically analyzed from multiple dimensions such as device data tracking, data reconciliation, real-time alarm, task link monitoring, and structuring, enabling users to clearly understand the device data access quality and data sharing degree of the current heterogeneous system at a glance, and improving the work efficiency of users. It solves the problems that users generally have headaches about, such as abnormal edge perception devices, inaccurate data collection, inconsistent sharing, and the problem that they cannot be discovered in time. Through the structured proxy engine, it quickly adapts to the structured engines of various algorithm manufacturers, converts the accessed unstructured data into the structured data required by users, and can share it with the original unstructured data with the third party as needed. It solves the problems of users' automatic adaptation of algorithm models for video and picture data, data retry, and retransmission.

[0123] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for distributed device access, control, and data transmission, characterized in that, The method includes the following steps: S1. Separate the management platform from the working nodes, run several working nodes in parallel, and uniformly manage the distributed deployment solution; S2. Build a technical solution framework of core service + components on the working nodes, with the core service as the center; S3. On the working nodes, implement dynamic publishing, real-time scheduling, load balancing, and exception switching of different tasks based on KeepAlived, task scheduling engine, ETCD, and component management service; S4. Integrate professional AI skills in each industry in the way of the structured proxy engine in the core service, convert the original unstructured data into structured data, and realize the parallel transmission of the original unstructured data and the secondary structured data; S5. Use the structured proxy engine to provide a retransmission mechanism and ensure that data omission and loss are avoided in case of structured exceptions.

2. The method for distributed device access, control and data transmission according to claim 1, wherein The components include access components and shared components, and each component corresponds to a task.

3. The method for distributed device access, control and data transmission according to claim 1, characterized in that The management platform and the working nodes communicate using the Message Queuing Telemetry Transport protocol to achieve reliable data transmission in low-bandwidth and unstable network environments; The working nodes actively register with the management platform, and the management platform provides device control instructions and management distribution of task scheduling to the working nodes, and performs visual processing based on the information reported by the working nodes; The working nodes complete the reallocation and service scheduling of each task according to the tasks issued by the management platform, and load, start, close, and unload the access components or shared components corresponding to the tasks through the component management service.

4. The method for distributed device access, control and data transmission according to claim 3, wherein The access component actively attempts to connect to the device according to the assigned task message. If the connection is successful within the specified number of connection attempts and time, it actively requests data or subscribes to data according to the device's protocol process, decodes the returned data, and uploads the qualified data to the topic of Data-Hub at the same time; If the connection is not successful, it will no longer interact with the corresponding device; The access component uploads the data generated during the connection process to Data-Hub according to the agreed-upon data point format and data point matters, records the abnormal situations generated during the upload process in the log file, and reports them to the management platform at the same time.

5. The method for distributed device access, control, and data transmission according to claim 4, characterized in that, The core service cleans the data subscribed and accessed in Data-Hub according to the predefined rules, and republishes the cleaned data to the topic of Data-Hub; Among them, the predefined rules include the following steps: Pre-configure the cleaning rule chain for the data to be cleaned on the management platform in advance. Each cleaning rule chain contains several rule nodes, and a complete data transmission process is formed by selecting the rule nodes; After the configuration is completed, the message is automatically persisted in the form of a json-formatted string and sent to the working nodes through the management of task scheduling; The core service encapsulates each rule node and entity business object into several Actors. When the management platform issues a rule engine message, the working nodes parse the information of the issued cleaning rule chain in real time, and dynamically load and start the corresponding Actors; The core service subscribes to each piece of data that has been connected in the Data-Hub, and delivers it to each Actor for data cleaning in the order of the cleaning rule chain, and returns the cleaned data to the business theme of the Data-Hub; The core service reports the metadata and the cleaned data to the storage platform according to the storage platform address sent by the management platform, and at the same time reports the task status of the work node for cleaning data to the management platform.

6. The method for distributed device access, control, and data transmission according to claim 5, characterized in that The shared component shares the data connected or cleaned in the Data-Hub to different platforms or systems according to the data sharing engine rules, and at the same time records the data generated during sharing into the log file and reports it according to the agreed data logging format and data logging matters; Among them, the data sharing engine rules include the following steps: Configure the data sharing rule chain for the data to be shared on the management platform, and automatically persist the message as a JSON-formatted string after configuration, and issue it to the work node through the management of task scheduling; The core service parses the data sharing rule engine message sent by the management platform, and the corresponding shared component actively tries to connect to the platform to be shared according to the sent task information, and judges whether the data channel to be shared is normal; If the connection is successful within the specified number of connection attempts and time, the data transfer channel is normal and the next step is carried out. If not, the next step is not carried out, and the abnormal log information is reported to the management platform; The core service dynamically loads and starts the corresponding Actor according to the data sharing rule engine message, and delivers the data subscribed on the Data-Hub to each Actor for cleaning in the order of the sharing rule chain, and delivers the cleaned data to the shared component; The shared component encapsulates and reports the cleaned data to the protocol platform, and reports the task status of the work node for sharing data to the management platform.

7. The method for distributed device access, control, and data transmission according to claim 6, characterized in that, The implementation of dynamic publishing, real-time scheduling, load balancing and exception switching of different tasks based on KeepAlived, task scheduling engine, ETCD and component management service in the work node includes the following steps: Map the cluster virtual IP and the local server IP based on KeepAlived, detect the local task scheduling engine service, and at the same time, based on ETCD, provide mutual registration, mutual discovery, health monitoring and task sharing among the nodes in the cluster; Use the task scheduling engine service to parse, split and sort the message tasks sent by the management platform, and re-publish the message tasks to ETCD, and at the same time report the health status of each microservice of the local node obtained to the management platform; The component management service subscribes to the task messages on the ETCD service and detects whether it is necessary to pull and load components from the remote component repository; If not, directly start the corresponding component locally, and each component automatically performs device control and data processing according to the sent configuration message; View the data content accessed or shared by each component and the running status of each component in real time, quickly judge whether the component meets the requirements of the production environment, and perform visual verification and debugging on the component.

8. The method for distributed device access, control and data transmission according to claim 7, characterized in that, Integrate professional AI skills in various industries by using the structured proxy engine in the core service, convert the original unstructured data into structured data, and achieve parallel transmission of the original unstructured data and the secondary structured data; use the structured proxy engine to provide a retransmission mechanism, and ensure that data omission and loss are avoided in case of structured exceptions, including the following steps: The structured proxy engine automatically adapts the assigned task message to the structured engine of the AI algorithm manufacturer, and issues and analyzes the original unstructured data obtained from the Data-Hub through the structured engine interface of the algorithm manufacturer; If the recognition is successful during the analysis, parse the result set according to the predefined rules, and if it meets the business requirements, publish it to the structured data topic in the Data-Hub; If the recognition is unsuccessful, discard this piece of original unstructured data and record the log. If the issuance is abnormal or the recognition fails, cache this piece of original unstructured data to the secondary recognition retransmission topic in the Data-Hub, resubscribe, and reissue the recognition task within the predefined time until the recognition is successful within the specified time or number of times.

9. A system for distributed device access, control, and data transmission, characterized in that, A method for implementing distributed device access, control, and data transmission according to any one of claims 1-8, the system includes a node deployment module, a framework construction module, a task processing module, and a data structure conversion module; Among them, the node deployment module is used to separate the management platform from the working nodes, run several working nodes in parallel, and uniformly manage the distributed deployment solution; The framework construction module is used to construct a technical solution framework of core service + components on the working nodes, with the core service as the center; The task processing module is used to implement dynamic publishing, real-time scheduling, load balancing, and exception switching of different tasks on the working nodes based on KeepAlived, task scheduling engine, ETCD, and component management service; The data structure conversion module is used to integrate professional AI skills in various industries by using the structured proxy engine in the core service, convert the original unstructured data into structured data, and achieve parallel transmission of the original unstructured data and the secondary structured data; use the structured proxy engine to provide a retransmission mechanism, and ensure that data omission and loss are avoided in case of structured exceptions.

10. A device for distributed device access, control, and data transmission, characterized in that, A method for implementing distributed device access, control, and data transmission according to any one of claims 1-8.

Images