Data processing method, driving center node and internet of things driving center system

Abstract

The application discloses a data processing method, a driving center node and an Internet of Things driving center system. The method comprises the following steps: consuming current to-be-processed messages of a target message topic from a message queue cluster, wherein the to-be-processed messages comprise to-be-processed data; matching a corresponding target processing unit from a current processor routing mapping relationship according to a function type of the target message topic and the to-be-processed messages, wherein the processor routing mapping relationship is used for recording a combination of each message topic and a message function type and a mapping relationship between each processing unit in a current driving JAR; and distributing a data processing task of the to-be-processed messages to a target thread pool bound with the target message topic for execution according to the target message topic, wherein the target thread pool is used for executing data processing logic of the target processing unit according to the data processing task. The application can improve system expansibility thread blocking and resource waste of an Internet of Things data processing platform.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, specifically to a data processing method, a driving center node, and an IoT driving center system. Background Technology

[0002] In the field of cloud-to-cloud data acquisition for the Internet of Things (IoT), IoT data processing platforms typically adopt a statically compiled driver architecture. The interfaces for processing data are tightly coupled with the core framework. The driver JARs for each platform interface need to be statically compiled and deployed along with the main program. Adding or modifying drivers requires restarting the service. Furthermore, message data processing uses a globally fixed single thread pool to process all message data, which can easily lead to thread blocking, resource waste, and poor system scalability. Summary of the Invention

[0003] This invention provides a data processing method, a driving center node, and an IoT driving center system, aiming to solve the technical problems of thread blocking, resource waste, and poor system scalability in related IoT data processing platforms.

[0004] In a first aspect, the present invention provides a data processing method based on an Internet of Things (IoT) driver center system, applied to a driver center node, wherein the IoT driver center system includes the driver center node and a message queue cluster, and the data processing method includes: Consume the currently pending messages for the target message topic from the message queue cluster, wherein the pending messages include pending data; Based on the target message topic and the function type of the message to be processed, the corresponding target processing unit is matched from the current processor routing mapping relationship. The processor routing mapping relationship is used to record the mapping relationship between each combination of message topic and message function type and each processing unit in the current driver JAR. Based on the target message topic, the data processing task of the message to be processed is distributed to the target thread pool bound to the target message topic for execution; The target thread pool is used to execute the data processing logic of the target processing unit according to the data processing task.

[0005] In some embodiments, the current driver JAR is updated by loading the driver in the following ways: In response to receiving a message about a driver configuration change event, the current driver metadata is read from the database based on the driver identifier and version information in the message. The current driver metadata includes the current driver version, remote download address, and checksum. Download the current driver JAR from the remote download address and perform an integrity check based on the checksum. In response to the current driver JAR passing validation, the corresponding current class loader is created. Load the current driver JAR using the current class loader, and instantiate the processing units in the current driver JAR package; Update the current processor routing mapping based on each processing unit in the current driver JAR package.

[0006] In some embodiments, after updating the current processor routing mapping based on the processing units in the current driver JAR package, the method further includes: The historical class loader corresponding to the historical driver JAR is marked as expired, and the historical class loader is reclaimed with a delay after the pending task corresponding to the historical driver JAR is completed.

[0007] In some embodiments, after distributing the data processing task of the message to be processed to a target thread pool bound to the target message topic for execution according to the target message topic, the method further includes: The execution status of each data processing task is checked periodically by a daemon thread; Interrupt timed-out tasks and release the corresponding processing thread resources; Retry the execution of the task that has failed, or transfer the pending message corresponding to the task to a pre-defined exception message topic in the message queue cluster.

[0008] In some embodiments, before periodically checking the execution status of each data processing task through a daemon thread, the method further includes: writing the task execution context of each data processing task into a timeout task mapping table, and recording the corresponding message topic and the cache location of the corresponding message to be processed in the message queue cluster. The method of periodically checking the execution status of each data processing task through a daemon thread includes: The daemon thread iterates through the task execution context of each data processing task in the timeout task mapping table to determine the execution status of each data processing task and whether each data processing task has timed out. The execution status of each data processing task is recorded in the database.

[0009] In some embodiments, after matching the corresponding target processing unit from the current processor routing mapping, the method further includes: In response to the failure to find a corresponding target processing unit in the processor routing mapping, the message to be processed is marked as an abnormal message and recorded in the database.

[0010] In some embodiments, the IoT driver center system further includes a driver gateway, and before consuming the currently pending messages for the target message topic from the message queue cluster, the method further includes: The driver gateway acquires the data to be processed and converts it into a message, which is then delivered to the message queue cluster.

[0011] Secondly, the present invention also provides a driving center node, the driving center node comprising: The message consumption module is used to consume the currently pending messages of the target message topic from the message queue cluster, wherein the pending messages include pending data. The processor routing module is used to match the corresponding target processing unit from the current processor routing mapping relationship according to the target message topic and the function type of the message to be processed. The processor routing mapping relationship is used to record the mapping relationship between each combination of message topic and message function type and each processing unit in the current driver JAR. The message processing module is used to distribute the data processing task of the message to be processed to the target thread pool bound to the target message topic for execution, based on the target message topic. The target thread pool is used to execute the data processing logic of the target processing unit according to the data processing task. In some embodiments, the driver center node further includes a dynamic class loading module; The dynamic class loading module is used to execute the driver loading and update process of the current driver JAR, which includes: In response to receiving a message about a driver configuration change event, the current driver metadata is read from the database based on the driver identifier and version information in the message. The current driver metadata includes the current driver version, remote download address, and checksum. Download the current driver JAR from the remote download address and perform an integrity check based on the checksum. In response to the current driver JAR passing validation, the corresponding current class loader is created. Load the current driver JAR using the current class loader, and instantiate the processing units in the current driver JAR package; Update the current processor routing mapping based on each processing unit in the current driver JAR package.

[0012] Thirdly, the present invention also provides an Internet of Things (IoT) driving center system, the IoT driving center system comprising: The driver gateway is used to maintain and manage the system driver configuration and collect data to be processed, and convert driver configuration change events and data to be processed into messages and deliver them to the message queue cluster. A drive center node is used to execute the data processing method provided in any embodiment of the first aspect; The message queue cluster is used to manage messages under each message topic, including pending messages corresponding to pending data and messages related to driver configuration change events. The database is used to store the latest driver configuration, the data to be processed collected by the driver gateway, and the data processing results of the driver center node.

[0013] According to the data processing method of the IoT-driven center system provided in this embodiment of the invention, after consuming messages to be processed in the message queue cluster, the messages to be processed are routed to the corresponding target processing units according to the message topic and function type of the messages to be processed through a pre-configured processor routing mapping relationship. Messages to be processed with different topics and function types can be directly routed to their respective matching target processing units for processing. This accurately locates and matches the corresponding processing units for heterogeneous data messages from different services and devices, unifies the routing entry point for heterogeneous data messages, adapts to multiple protocols and multiple types of IoT devices, shields the underlying protocol differences of different data messages, and enables the system to have good versatility. It offers both security and compatibility with scalability. Furthermore, by pre-binding different message topics to corresponding independent thread pools, once a message to be processed is matched with a target processing unit, the data processing task for that message is distributed to the target thread pool bound to the target message topic for execution. The target thread pool executes the data processing logic of the target processing unit according to the data processing task. This achieves topic-based isolation of thread resources, ensuring that different topics do not interfere with each other. This facilitates the on-demand adjustment of thread resources for each topic, supports topic traffic fluctuations, reduces the risk of message backlog, effectively improves the problem of global thread blocking and thread contention caused by a sudden increase in traffic to a single topic, and helps to reduce system resource waste, thereby improving system resource utilization and system stability. Attached Figure Description

[0014] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the system architecture of an Internet of Things (IoT) drive center system provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a data processing method based on an Internet of Things (IoT) driven center system provided in an embodiment of the present invention; Figure 3This is a schematic diagram of the driver JAR loading and updating process of the driver center node in an embodiment of the present invention; Figure 4 A block diagram illustrating the composition of a driving center node provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0018] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0019] In the field of cloud-to-cloud data acquisition for the Internet of Things (IoT), IoT data processing platforms typically adopt a statically compiled driver architecture. The interfaces for processing data are tightly coupled with the core framework. The driver JARs for each platform interface need to be statically compiled and deployed along with the main program. Adding or modifying drivers requires restarting the service. Furthermore, message data processing uses a globally fixed single thread pool to process all message data, which can easily lead to thread blocking, resource waste, and poor system scalability.

[0020] To address these issues, this invention provides a data processing method, a driving center node, and an IoT driving center system based on an IoT driving center system. The aim is to effectively solve technical problems such as thread blocking, resource waste, and poor system scalability in related IoT data processing platforms.

[0021] Figure 1 This is a schematic diagram of the system architecture of an Internet of Things (IoT) drive center system provided in an embodiment of the present invention, such as... Figure 1As shown, the IoT driver center system includes a driver gateway 101, a driver center node 102, a message queue cluster 103, and a database 104.

[0022] Among them, the driver gateway 101, as the upstream entry point of the system, is responsible for maintaining and managing the driver configuration of the system and collecting data to be processed (such as device data, alarm data and historical data uploaded by third-party devices). It persists the latest driver configuration and the collected data to be processed to the database 104, and through the Outbox mechanism, it transforms driver configuration change events and data to be processed into messages and delivers them to the message queue cluster 103. The Outbox mechanism is a message publishing mechanism and an architectural pattern that ensures reliable message publishing.

[0023] Message queue cluster 103, acting as the system's message relay layer, is responsible for managing messages under each message topic, buffering and distributing message data under different topics, including pending messages corresponding to pending data and messages indicating driver configuration change events. For example, message queue cluster 103 can use a Kafka cluster, which is a distributed message queue (MQ) based on a publish / subscribe model. A message topic is a logical classification unit in the message queue according to business type. Each message topic may include, for example, a device real-time data (pending data) topic, an alarm topic, a driver configuration update topic, etc. Each message topic can be divided into multiple message partitions. A partition is the smallest physical unit within a topic that enables parallel message storage and consumption. Partitions under the same topic are independent of each other, and messages within a partition are cached in an ordered manner according to their corresponding offset information. Each message has unique offset information.

[0024] Database 104 is responsible for synchronously storing the latest driver configuration, the data to be processed collected by the driver gateway, and the data processing results of the driver center node, as well as synchronously sharing the execution status of the data processing tasks of the driver center node; Database 104 in this embodiment of the invention can be implemented using any persistent database; As a downstream processing unit of the system, the driver center node 102 is responsible for subscribing to messages of the required message topics, consuming messages from the message queue cluster 103, and loading and processing the driver based on the driver configuration in the database. During the data processing, the driver center node 102 locates and matches the target processing unit according to the message topic and function type, executes the data processing logic of the target processing unit through an independent thread pool built according to the message topic dimension, processes the data to be processed, and performs timeout checks on the data processing tasks through a timeout check mechanism. The processing results of the data processing tasks are finally written to the database or output to the downstream system again through the message queue cluster, forming a complete data processing closed loop.

[0025] The driver gateway 101 and the driver center node 102 are asynchronously decoupled through the message queue cluster 103, and the driver configuration synchronization and state sharing are completed through the database 104, thereby forming a clear division of upstream and downstream responsibilities and message sending and receiving links.

[0026] According to an embodiment of the present invention, a data processing method based on an Internet of Things (IoT) driving center system is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] This invention provides a data processing method based on an IoT driver center system, applied to the aforementioned driver center node. The IoT driver center system includes a driver center node and a message queue cluster. Figure 2 This is a flowchart illustrating a data processing method based on an IoT-driven central system, as provided in an embodiment of the present invention. Figure 2 As shown, the data processing method includes the following steps: Step S201: Consume the current pending messages for the target message topic from the message queue cluster. The pending messages include pending data.

[0028] When the system starts, the driver center node loads the current driver JAR. During the data processing phase, the driver center node pulls the messages to be processed corresponding to the data to be processed under the specified target message topic from the message queue cluster.

[0029] Step S202: Based on the target message topic and the function type of the message to be processed, match the corresponding target processing unit from the current processor routing mapping relationship. The processor routing mapping relationship is used to record the mapping relationship between each combination of message topic and message function type and each processing unit in the current driver JAR.

[0030] In this embodiment of the invention, the driver JAR is an executable driver component that contains IoT device protocol adaptation and data processing logic. One driver JAR corresponds to one device protocol, such as the device protocol of photovoltaic inverter, charging pile, energy storage device, etc. The driver center node is a core system service that can be used to load, manage, schedule and execute the logic of the driver JAR. The driver center node can load the current driver JAR through a dynamic class loading mechanism, obtain and instantiate processing units from it to complete the processing of heterogeneous IoT data.

[0031] A processing unit, also known as a device protocol adapter or handler, is an executable logical unit within a driver JAR used to perform data protocol parsing, format conversion, and business processing on the data to be processed. A driver JAR can contain one or more processing units. In a driver JAR, a processing unit is a class file. By loading the driver JAR, the processing unit class file is instantiated into a processing unit instance or object that can be called and executed.

[0032] In this embodiment of the invention, the driver center node is pre-configured with the current processor routing mapping relationship, records the combination of each message topic and message function type and the mapping relationship between each processing unit in the current driver JAR, and synchronously records the current processor routing mapping relationship in the database to ensure consistency across multiple nodes. The message function type (functionType) is an identifier used to distinguish message service types. In the processor routing mapping relationship, the function type and message topic (Topic) together constitute the routing key in the processor routing mapping relationship, and different routing keys map to different processing units.

[0033] After consuming messages to be processed in the message queue cluster, messages of different topics and functional types can be directly routed to their respective matching target processing units for processing through pre-configured processor routing mapping relationships. This enables precise location and matching of corresponding processing units for heterogeneous data messages from different businesses and devices, ensuring that each message corresponds to its own dedicated processing logic and avoiding confusion and mismatch in business logic. The message consumption layer of the message queue cluster is completely decoupled from the specific protocol processing and business processing layer. When adding new device types or new business types, only the driver JAR and the corresponding processor routing mapping relationship configuration need to be added, enabling flexible matching of messages and processing units. The routing rules are decoupled from the code, requiring no modification to the core scheduling code, resulting in strong scalability. The routing entry point for different data messages is unified, adapting to multiple protocols and multiple types of IoT devices, shielding the underlying protocol differences of different message data. The upper-layer framework does not need to be aware of specific device protocols, giving the system good versatility and compatibility.

[0034] Step S203: Based on the target message topic, distribute the data processing tasks of the message to be processed to the target thread pool bound to the target message topic for execution; wherein, the target thread pool is used to execute the data processing logic of the target processing unit according to the data processing tasks.

[0035] In this embodiment of the invention, thread pools are pre-created independently, each bound to a specific message topic. Each thread pool is a collection of thread execution resources, including at least one processing thread. By isolating data processing tasks under different topics, interference between thread execution resources for data processing tasks under different topics is avoided. After consuming the messages to be processed and matching them to the corresponding target processing unit, the data processing tasks for the messages to be processed are distributed to the target thread pools bound to the target message topics for execution, based on the target message topic.

[0036] Understandably, when the target thread pool receives one or more data processing tasks for a corresponding bound target message topic, it allocates one or more idle processing threads to execute these tasks. Each thread processes one task at a time. After a task is completed, the thread is released back to the thread pool to continue processing the next task, thus achieving concurrent processing and efficient utilization of thread resources. Each thread executes the data processing logic of the target processing unit according to its assigned task, thereby processing the data to be processed corresponding to the task. When the number of data processing tasks for the target message topic exceeds the number of currently idle processing threads in the target thread pool, some tasks will be queued and not executed immediately.

[0037] According to the data processing method provided in this embodiment of the invention, after consuming messages to be processed in the message queue cluster, the messages to be processed are routed to the corresponding target processing units according to the message topic and function type of the messages to be processed through a pre-configured processor routing mapping relationship. Messages to be processed with different topics and function types can be directly routed to their respective matching target processing units for processing. This accurately locates and matches the corresponding processing units for heterogeneous data messages from different services and devices, unifies the routing entry point for heterogeneous data messages, adapts to multiple protocols and multiple types of IoT devices, shields the underlying protocol differences of different data messages, and enables the system to have good versatility and compatibility for expansion. The system possesses the capability to handle data processing tasks. Furthermore, by pre-binding different message topics to corresponding independent thread pools, once a message is matched to a target processing unit, its data processing task is distributed to the target thread pool bound to that message topic. The target thread pool executes the data processing logic of the target processing unit based on the data processing task. This achieves topic-based isolation of thread resources, ensuring that different topics do not interfere with each other. This facilitates the on-demand adjustment of thread resources for each topic, supports fluctuations in topic traffic, reduces the risk of message backlog, effectively improves the problem of global thread blocking and thread contention caused by sudden increases in traffic to a single topic, and helps to reduce system resource waste, thereby improving system resource utilization and stability.

[0038] In some embodiments, the IoT driver center system further includes a driver gateway. Before consuming the current pending messages of the target message topic from the message queue cluster, the data processing method further includes: the driver gateway acquiring the pending data and converting the pending data into messages for delivery to the message queue cluster.

[0039] Specifically, the driver gateway acquires data to be processed from IoT devices or platforms. This data includes multi-source heterogeneous data such as real-time device data, alarm data, and historical data. After collecting and preprocessing the data, the driver gateway writes the preprocessed data into the database for storage. It then uses the Outbox message publishing mechanism to convert the data into messages that the message queue can recognize and delivers them to the corresponding message topics in the message queue cluster according to the data type and business scenario.

[0040] In some embodiments, after matching a corresponding target processing unit from the current processor routing mapping, the data processing method further includes: in response to not matching a corresponding target processing unit in the processor routing mapping, marking the message to be processed as an abnormal message and recording it in the database.

[0041] Specifically, when matching the corresponding target processing unit from the current processor routing mapping relationship, if no corresponding target processing unit is found in the processor routing mapping relationship or the target processing unit conversion fails, the fallback logic is triggered, the message to be processed is marked as an abnormal message and recorded in the database or log system, and the default processing strategy or the message can be rolled back to or discarded.

[0042] If a corresponding target processing unit is matched in the processor routing mapping relationship, the routing hit result and processing status can be logged or persisted as needed for subsequent problem tracing and link analysis, thereby ensuring the observability and fault tolerance of the message routing process.

[0043] Figure 3 This is a schematic diagram of the driver JAR loading and update process of the driver center node in an embodiment of the present invention. In some embodiments, such as... Figure 3 As shown, the current driver JAR of the driver center node is loaded and updated in the following way: Step S301: In response to receiving a message about a driver configuration change event, read the current driver metadata from the database based on the driver identifier and version information in the message. The current driver metadata includes the current driver version, remote download address, and checksum.

[0044] Specifically, in addition to subscribing to message topics for data to be processed, the driver center node also needs to subscribe to message topics for driver configuration changes in order to synchronously obtain driver configuration change information through the driver configuration management of the driver gateway.

[0045] When the driver gateway receives a business request for a driver configuration change, it obtains the current driver metadata of the currently updated driver JAR, including the current driver version, remote download address, and checksum. It then writes the current driver metadata into the database for storage and generates a driver configuration change event. Through the Outbox message publishing mechanism, it transforms the driver configuration change event into a driver configuration change event message and delivers it to the corresponding driver configuration change message topic in the message queue cluster. This driver configuration change event message can carry the driver identifier and version information of the current driver metadata stored in the database.

[0046] After the driver center node receives a message about a driver configuration change event, it reads the current driver metadata corresponding to the driver identifier and version information in the message from the database.

[0047] Step S302: Download the current driver JAR from the remote download address and perform an integrity check based on the checksum.

[0048] The driver center node downloads the current driver JAR to be updated from the corresponding remote download address (such as the URL link of the current driver JAR) based on the remote download address in the current driver metadata. It then uses the checksum carried in the current driver metadata to perform SHA-256 integrity verification on the downloaded current driver JAR to verify that the current driver JAR has not been tampered with, damaged or incomplete during transmission, ensuring that the loaded driver is legal, valid and secure.

[0049] If the verification fails or the download fails, cancel the current driver loading and update, and revert to the previous effective driver version. At the same time, record the verification failure or download failure status to the database or log system. If the verification passes, proceed to step S303.

[0050] Step S303: In response to the current driver JAR passing verification, create the corresponding current class loader.

[0051] The class loader is responsible for dynamically loading driver JARs. It loads the class files of the processing units in the driver JAR into memory and instantiates them into executable Class objects, thus obtaining the processing unit instance. Different driver JARs are loaded by independent class loaders to achieve driver isolation and hot updates.

[0052] Step S304: Load the current driver JAR using the current class loader and instantiate each processing unit in the current driver JAR package.

[0053] In response to the current driver JAR passing the verification, a current class loader corresponding to the current driver JAR is created to achieve isolation from already loaded classes. The current class loader is used to load and parse the current driver JAR, and instantiate each processing unit (processor / adapter) contained in the current driver JAR.

[0054] Step S305: Update the current processor routing mapping relationship according to each processing unit in the current driver JAR package.

[0055] Each processing unit in the current driver JAR that has been instantiated will be updated and maintained according to the matching identifiers such as the message topic and function type corresponding to each processing unit. The existing processor routing mapping relationship of the system will be updated and maintained. The newly instantiated processing unit will be registered in the processor routing mapping relationship table, replacing the routing relationship corresponding to the old version processing unit, and establishing a one-to-one mapping binding between the message topic, function type and the newly instantiated processing unit.

[0056] By updating the processor routing mapping, when subsequent messages of the corresponding topic and function type are consumed, the processor routing mechanism can accurately locate the latest effective processing unit, realizing hot driver updates. The processing unit or device protocol adapter of the driver can be updated without restarting the driver central node service, and the routing rules can be seamlessly switched after the driver version is updated, ensuring the normal connection of subsequent data processing processes.

[0057] In some embodiments, the switching and updating of the old and new driver versions is carried out with the messages of the message queue cluster as the boundary, ensuring that the same message flow is processed under a single driver version. After updating the current processor routing mapping relationship according to each processing unit in the current driver JAR package, the data processing method further includes: marking the historical class loader corresponding to the historical driver JAR as expired, and delaying the recycling of the historical class loader after the pending task corresponding to the historical driver JAR is completed.

[0058] After the new driver JAR is loaded and updated, the historical class loader corresponding to the historical driver JAR is marked as stale. The lifecycle of the historical class loader is bound to its associated pending task. After the associated pending task is completed, the historical class loader is reclaimed with a delay, and the loader state change and reclamation result are recorded.

[0059] By marking historical class loaders of historical driver JARs as overdue for garbage collection instead of immediate destruction, the associated pending tasks that are still running are allowed to continue to complete, avoiding abnormal interruption of tasks being processed and data processing failures caused by the forced unloading of old drivers. This achieves a smooth, uninterrupted switchover of business processes during hot driver version updates. If historical class loaders are destroyed immediately, the executing threads will still reference the processing unit classes in the old drivers, which can easily lead to class not found, class unloading conflicts, memory leaks, and JVM exceptions. By marking overdue class loaders for garbage collection and delaying garbage collection after tasks are completed, it is ensured that threads are released only after they are no longer referenced by historical class loaders, avoiding runtime exceptions. The delayed garbage collection mechanism reserves a complete execution cycle for existing tasks, preventing batch task errors and message backlogs during hot driver updates, and enhancing the operational stability of the driver center in version iteration and dynamic update scenarios. Historical class loaders and associated resources are uniformly collected only after all pending tasks have been executed, avoiding the long-term occupation of memory by invalid resources, achieving orderly release of memory resources after driver version replacement, and improving the problem of memory resource waste in frequent update scenarios.

[0060] In some application scenarios, for deployments with multiple driver center nodes, messages about driver configuration change events are broadcast through a message queue cluster. Each driver center node can combine lease or version checking mechanisms to avoid repeatedly loading the latest driver JAR, thus achieving consistent synchronization of driver updates across nodes.

[0061] In some embodiments, after distributing the data processing tasks of the messages to be processed to the target thread pool bound to the target message topic according to the target message topic, the data processing method further includes: periodically checking the execution status of each data processing task through a daemon thread; interrupting timed-out tasks and releasing the corresponding processing thread resources; retrying execution of tasks with execution errors, or transferring the messages to be processed corresponding to the tasks with execution errors to a preset error message topic in the message queue cluster.

[0062] In some embodiments, independent thread pools are pre-created according to each message topic dimension, and a guardian thread is also created. Each message topic corresponds to an independent thread pool. The thread pool parameters include the minimum number of threads that must always be alive in the thread pool (corePoolSize), the maximum number of resources that the thread pool can create (maxPoolSize), the capacity of the blocking queue used to store tasks waiting to be executed in the thread pool (queueCapacity), the lifespan of idle threads in the thread pool (keepAliveTime), and the task timeout threshold, etc. The thread pool parameters of each thread pool are configured by the driver gateway and stored in the database. They are loaded and refreshed when the system starts or when a configuration change event (issued through the message queue cluster) occurs, thereby achieving resource scheduling consistent with the message flow.

[0063] After consuming messages to be processed and matching them with target processing units, the central node distributes the data processing tasks of the messages to the thread pool management tool (ThreadPoolExecutor) of the target thread pool bound to the target message topic, based on the target message topic. The thread pool management tool of the target thread pool schedules idle processing threads to execute the data processing tasks. When a new data processing task is submitted, if the number of currently running threads in the target thread pool is less than corePoolSize, a new thread is created to handle the request, even if other auxiliary threads are idle. If the number of currently running threads in the target thread pool is more than corePoolSize but less than maxPoolSize, a new thread is created only when the task queue is full. If corePoolSize and maxPoolSize are set to the same value, a fixed-size thread pool is created. If maxPoolSize is set to a basic unbounded value, the target thread pool is allowed to accommodate any number of concurrent tasks.

[0064] By isolating independent thread pools for each message topic according to the Topic dimension, and allocating thread resources for each message topic's thread pool on demand, resource utilization is improved and resource waste is reduced.

[0065] In some embodiments, before periodically checking the execution status of each data processing task through a daemon thread, the data processing method further includes: writing the task execution context of each data processing task into a timeout task mapping table, and recording the corresponding message topic and the cache location of the corresponding message to be processed in the message queue cluster.

[0066] Specifically, when each data processing task is submitted, the task execution context of each data processing task is written into the timeout task mapping table (taskTimeoutMap), and the corresponding message topic and the cache location of the corresponding message to be processed in the message queue cluster are recorded. The cache location of the message to be processed in the message queue cluster includes the partition of the message topic where the message to be processed is located and the offset information in the partition.

[0067] In some embodiments, the execution status of each data processing task is checked periodically by a daemon thread, including: traversing the task execution context of each data processing task in the timeout task mapping table by the daemon thread, determining the execution status of each data processing task, and determining whether each data processing task has timed out; and recording the execution status of each data processing task in the database.

[0068] Specifically, the daemon thread Guardian executes the timeout check task checkTimeoutTasks() at fixed intervals. By traversing the task execution context of each data processing task in the timeout task mapping table taskTimeoutMap, the daemon thread determines the execution status of each data processing task and whether it has timed out. For example, if the execution time of a data processing task exceeds a preset timeout threshold, it is determined that the task has timed out, and the cancel function is called to interrupt the task execution and release the corresponding thread resources. cancel is a tool in the thread pool used to cancel asynchronous tasks. Simultaneously, the execution status of each data processing task (e.g., timed out, failed, or completed normally) is recorded in the database for task auditing and recovery.

[0069] By checking task timeouts in the daemon thread, zombie tasks are automatically cleaned up, abnormally occupied thread resources are released in a timely manner, thread reuse is ensured efficiently, and resource utilization is improved.

[0070] In some embodiments, when a task is canceled or execution fails, the consumption position control policy of the message queue cluster determines whether to retry the task or transfer the pending messages corresponding to the failed task to a preset exception message topic to avoid message loss. In the case of node restart, the processing progress of the data processing task can be restored by combining the execution status of each data processing task in the database with the cache location of the corresponding pending messages in the message queue cluster (such as message partition and offset information).

[0071] In some embodiments, in a multi-driver central node deployment scenario, task distribution is achieved based on the partitioned consumption mechanism of the message queue cluster, and duplicate execution is avoided through partition ownership or lease mechanism.

[0072] In this embodiment of the invention, a drive center node is also provided, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0073] Figure 4 This is a block diagram illustrating the composition of a driving center node according to an embodiment of the present invention. The embodiment of the present invention provides a driving center node, such as... Figure 4 As shown, the drive center node includes: The message consumption module 401 is used to consume the currently pending messages of the target message topic from the message queue cluster. The pending messages include pending data. The processor routing module 402 is used to match the corresponding target processing unit from the current processor routing mapping relationship according to the target message topic and the function type of the message to be processed. The processor routing mapping relationship is used to record the mapping relationship between each combination of message topic and message function type and each processing unit in the current driver JAR. The message processing module 403 is used to distribute the data processing tasks of the message to be processed to the target thread pool bound to the target message topic for execution, according to the target message topic; wherein, the target thread pool is used to execute the data processing logic of the target processing unit according to the data processing tasks.

[0074] In some embodiments, such as Figure 4 As shown, the driver center node also includes a dynamic class loading module 404; the dynamic class loading module 404 is used to execute the driver loading and update process of the current driver JAR. The driver loading and update process includes: in response to receiving a message of a driver configuration change event, reading the current driver metadata from the database according to the driver identifier and version information in the message. The current driver metadata includes the current driver version, remote download address, and checksum; downloading the current driver JAR from the remote download address and performing integrity verification according to the checksum; in response to the current driver JAR passing the verification, creating the corresponding current class loader. Load the current driver JAR using the current class loader, and instantiate the processing units in the current driver JAR package; Update the current processor routing mapping based on each processing unit in the current driver JAR package.

[0075] The driving center node provided in this embodiment of the invention can execute the data processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments above, and will not be repeated here.

[0076] This invention also provides an Internet of Things (IoT) driver center system, such as... Figure 1 As shown, the IoT-driven central system includes: Driver Gateway 101 is used to maintain and manage the driver configuration of the system and collect data to be processed, and convert driver configuration change events and data to be processed into messages and deliver them to the message queue cluster. The central node 102 is used to execute the data processing method of any of the above embodiments; Message queue cluster 103 is used to manage messages under each message topic, including pending messages corresponding to pending data and messages of driver configuration change events; Database 104 is used to store the latest driver configuration, the data to be processed collected by the driver gateway, and the data processing results of the driver center node.

[0077] The IoT drive center system provided in this embodiment of the invention can execute the data processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0078] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention.

[0079] The following is a detailed reference. Figure 5 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0080] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0081] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the data processing method of the embodiments of the present invention.

[0082] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0083] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the data processing methods shown in the above embodiments are implemented.

[0084] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0085] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data processing method based on an Internet of Things (IoT) driven central system, characterized in that, Applied to a driver center node, the IoT driver center system includes the driver center node and a message queue cluster, and the method includes: Consume the currently pending messages for the target message topic from the message queue cluster, wherein the pending messages include pending data; Based on the target message topic and the function type of the message to be processed, the corresponding target processing unit is matched from the current processor routing mapping relationship. The processor routing mapping relationship is used to record the combination of each message topic and message function type and the mapping relationship between each processing unit in the current driver JAR. Based on the target message topic, the data processing task of the message to be processed is distributed to the target thread pool bound to the target message topic for execution; The target thread pool is used to execute the data processing logic of the target processing unit according to the data processing task.

2. The data processing method according to claim 1, characterized in that, The current driver JAR is loaded and updated in the following way: In response to receiving a message about a driver configuration change event, the current driver metadata is read from the database based on the driver identifier and version information in the message. The current driver metadata includes the current driver version, remote download address, and checksum. Download the current driver JAR from the remote download address and perform an integrity check based on the checksum. In response to the current driver JAR passing validation, the corresponding current class loader is created. Load the current driver JAR using the current class loader, and instantiate the processing units in the current driver JAR package; Update the current processor routing mapping based on each processing unit in the current driver JAR package.

3. The data processing method according to claim 2, characterized in that, After updating the current processor routing mapping based on each processing unit in the current driver JAR package, the method further includes: The historical class loader corresponding to the historical driver JAR is marked as expired, and the historical class loader is reclaimed with a delay after the pending task corresponding to the historical driver JAR is completed.

4. The data processing method according to claim 1, characterized in that, After distributing the data processing task of the message to be processed to the target thread pool bound to the target message topic for execution according to the target message topic, the method further includes: The execution status of each data processing task is checked periodically by a daemon thread; Interrupt timed-out tasks and release the corresponding processing thread resources; Retry the execution of the task that has failed, or transfer the pending message corresponding to the task to a pre-defined exception message topic in the message queue cluster.

5. The data processing method according to claim 4, characterized in that, Before periodically checking the execution status of each data processing task through a daemon thread, the method further includes: writing the task execution context of each data processing task into a timeout task mapping table, and recording the corresponding message topic and the cache location of the corresponding message to be processed in the message queue cluster; The method of periodically checking the execution status of each data processing task through a daemon thread includes: The daemon thread iterates through the task execution context of each data processing task in the timeout task mapping table to determine the execution status of each data processing task and whether each data processing task has timed out. The execution status of each data processing task is recorded in the database.

6. The data processing method according to claim 1, characterized in that, After matching the corresponding target processing unit from the current processor routing mapping, the method further includes: In response to the failure to find a corresponding target processing unit in the processor routing mapping, the message to be processed is marked as an abnormal message and recorded in the database.

7. The data processing method according to claim 1, characterized in that, The IoT driver center system further includes a driver gateway. Before consuming the currently pending messages for the target message topic from the message queue cluster, the method further includes: The driver gateway acquires the data to be processed and converts it into a message, which is then delivered to the message queue cluster.

8. A driving center node, characterized in that, The drive center node includes: The message consumption module is used to consume the currently pending messages of the target message topic from the message queue cluster, wherein the pending messages include pending data. The processor routing module is used to match the corresponding target processing unit from the current processor routing mapping relationship according to the target message topic and the function type of the message to be processed. The processor routing mapping relationship is used to record the mapping relationship between each combination of message topic and message function type and each processing unit in the current driver JAR. The message processing module is used to distribute the data processing task of the message to be processed to the target thread pool bound to the target message topic for execution, based on the target message topic. The target thread pool is used to execute the data processing logic of the target processing unit according to the data processing task.

9. The driving center node according to claim 8, characterized in that, The driver center node also includes a dynamic class loading module; The dynamic class loading module is used to execute the driver loading and update process of the current driver JAR, which includes: In response to receiving a message about a driver configuration change event, the current driver metadata is read from the database based on the driver identifier and version information in the message. The current driver metadata includes the current driver version, remote download address, and checksum. Download the current driver JAR from the remote download address and perform an integrity check based on the checksum. In response to the current driver JAR passing validation, the corresponding current class loader is created. Load the current driver JAR using the current class loader, and instantiate the processing units in the current driver JAR package; Update the current processor routing mapping based on each processing unit in the current driver JAR package.

10. An Internet of Things (IoT) drive center system, characterized in that, The IoT-driven central system includes: The driver gateway is used to maintain and manage the system driver configuration and collect data to be processed, and convert driver configuration change events and data to be processed into messages and deliver them to the message queue cluster. A central node is driven to execute the data processing method as described in any one of claims 1-7; The message queue cluster is used to manage messages under each message topic, including pending messages corresponding to pending data and messages related to driver configuration change events. The database is used to store the latest driver configuration, the data to be processed collected by the driver gateway, and the data processing results of the driver center node.

Images