Sensor data access method for load balancing and parallel optimization in smart park

Abstract

The invention discloses a sensor data access method for load balancing and parallel optimization in a smart park, and the method employs a JMX to monitor the load condition of a Kafka partition, and distributes and deploys temperature and humidity sensor and RFID equipment data to a Kafka cluster in a dynamic load balancing manner through customizing a Kafka Producer. A ForkJoinPool parallel processing mechanism is used, data preprocessing, cleaning and processing tasks are executed in Spark Streaming, and finally data polling is carried out and written into a database node. Through dynamic load balancing and multi-thread parallel processing, the processing efficiency of the sensor data is remarkably improved. The system automatically distributes data according to the partition load, avoids the overload condition, and accelerates the data processing speed by adopting a task partitioning mechanism. The data cleaning strategy ensures the accuracy and reliability of the data, and the polling storage strategy further optimizes the load balance of the database nodes. The whole system has the advantages of high efficiency, stability and expandability, and the management efficiency and the data processing capability of the smart park are effectively improved.

Description

Technical Field

[0001] The present invention relates to a sensor data access method based on load balancing and parallel optimization in a smart park, and belongs to the technical field of data transmission and distributed processing of the Internet of Things. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, smart parks have gradually become the core model of modern park management. Such parks deploy various sensors, such as temperature sensors, humidity sensors, and radio frequency identification (RFID) devices, to achieve real-time monitoring of the environment, equipment, and personnel, and improve management efficiency and safety. However, with the surge in the number of sensors and the increase in the frequency of data collection, the amount of data has increased dramatically. How to efficiently collect, transmit, process, and store these massive amounts of data has become a key challenge in the construction of smart parks. Especially when multiple sensors are working at the same time, the high concurrency and diversity of data increase the complexity of system management. Traditional processing methods often cannot meet the real-time and data quality requirements.

[0003] First, there is the management complexity brought about by the diversity of sensor data and the huge amount of data. Different types of data such as temperature, humidity and location information often require different processing methods, and the surge in data volume can easily lead to system overload. Secondly, the problem of load imbalance is prominent. With the continuous changes in equipment and data traffic, some data partitions or nodes may be overloaded, resulting in system performance degradation. Although the traditional static partitioning method is simple, it lacks dynamic adjustment capabilities and cannot effectively cope with the imbalance of data load in actual applications, causing some partition resources to be idle, while other partitions may have serious data backlogs. Furthermore, smart parks have extremely high requirements for real-time data processing, especially in environmental monitoring and personnel positioning. If the data cannot be processed and responded to in a timely manner, it will directly affect the efficiency and safety of park management. In addition, the quality of data from different sensors varies, and noise data and outliers will affect the accuracy of subsequent data analysis.

[0004] Traditional solutions often have obvious limitations for these pain points. Although data distribution based on static partitioning or hash strategies is simple to implement, it cannot dynamically adapt to fluctuations in data traffic, resulting in load imbalance. Although the random allocation method can avoid overloading of some partitions, it still cannot achieve optimal resource utilization due to the lack of accurate monitoring of real-time loads. Although the dynamic load balancing method based on resource scheduling can be dynamically adjusted according to the actual load of the system, it has high implementation complexity and requires a powerful monitoring and decision-making mechanism to adjust the partition strategy in real time, resulting in high system overhead. Moreover, in the process of sensor data access and processing in smart parks, traditional methods often find it difficult to effectively solve problems such as load imbalance, real-time processing, and data quality. There are many types of sensor devices, data traffic fluctuates greatly, and different types of data have different requirements for timeliness and accuracy. Traditional static partitioning and load distribution methods often lead to overloading of some partitions, while other partition resources are idle, and the processing capacity of the system cannot be fully utilized. Summary of the invention

[0005] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and propose a sensor data access method with load balancing and parallel optimization in a smart park. The method combines streaming data transmission technologies such as Kafka and Spark to achieve more efficient data access and processing in the smart park. Through the integration of the Kafka cluster and the JMX framework, the system can monitor the load of each partition in real time, including the message backlog, throughput and response time of the partition, and calculate the load weight of the partition based on these data. The customized KafkaProducer can call the load aggregation service before sending data, dynamically obtain the latest load weight of the partition, so as to distribute the data to the partition with lower load first, ensure the reasonable distribution of data streams between different partitions, and avoid overload problems. In addition, the parallel optimization mechanism of the ForkJoinPool thread pool and SparkStreaming is adopted, which can split large data sets into multiple subtasks. Multiple threads in the thread pool can process these tasks in parallel and support recursive splitting, which greatly improves the processing efficiency and scalability of the system.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a sensor data access method based on dynamic load balancing and parallel processing optimization in a smart park, the method comprising the following steps:

[0007] Step 1: Deploy temperature sensors, humidity sensors, and radio frequency identification (RFID) devices in the smart park. The devices are identified by a unique Device_ID, and the data type is defined by Data_Type.

[0008] Step 2: Send the Sensor_Data collected by the device to the edge gateway through the MQTT protocol, where the Sensor_Data includes Device_ID, Data_Value and Timestamp;

[0009] Step 3: The edge gateway receives the Sensor_Data and formats it into a JSON object containing Device_ID, Data_Type, Data_Value, and Timestamp, and uses Kafka Producer to send the JSON object to the Kafka topic associated with the device type.

[0010] Furthermore, the present invention includes a load aggregation service. The load aggregation service collects load data of Kafka partitions through JMX, and the load data includes message backlog, partition throughput and partition response time of the partition.

[0011] Furthermore, the present invention includes a load weight. The load weight is calculated according to the following formula:

[0012] Load Weight =α×Partition Backlog +β×Partition Throughput +γ×Partition ResponseTime

[0013] Among them, α, β and γ are adjustment parameters, which are used to adjust the influence of each indicator on the load weight.

[0014] Furthermore, before sending a message, Kafka Producer calls the REST API interface of the load aggregation service to obtain the latest load weight of each partition, and sends the message to the partition with the lowest load weight to achieve load balancing.

[0015] Furthermore, the present invention includes Spark Streaming. The Spark Streaming reads data from multiple topics through Kafka, maps each Kafka partition to a parallel Spark task for processing, and parses the data into structured data through JSON for cleaning.

[0016] Furthermore, the present invention includes data cleaning. The data cleaning includes:

[0017] 1) The temperature sensor data only retains the records with Data_Value in the range of -50℃ to 100℃;

[0018] 2) Humidity sensor data only retains records with Data_Value in the range of 0% to 100%;

[0019] 3) RFID location data only retains records of longitude and latitude within the valid range.

[0020] Furthermore, the Spark task of the present invention uses the ForkJoinPool thread pool for parallel processing. When the data volume of the task exceeds a predefined threshold, the task is divided into multiple subtasks, and recursively split into appropriate scales for parallel processing, and finally the results of each subtask are summarized to generate the final task result.

[0021] Furthermore, the present invention uses JMX to monitor the load of Kafka partitions, and distributes and deploys temperature, humidity sensors and RFID device data to the Kafka cluster in a dynamic load-balanced manner through customizing KafkaProducer. Using the ForkJoinPool parallel processing mechanism, data preprocessing, cleaning and processing tasks are performed in Spark Streaming, and finally the data polling is written into the database node. This belongs to the field of Internet of Things data transmission and distributed processing.

[0022] The sensor data access method based on dynamic load balancing and parallel processing optimization in a smart park of the present invention also includes:

[0023] The sensor sends data to the edge gateway via the MQTT protocol. The gateway formats the data into JSON objects and distributes them to different topics of Kafka according to the device type. The message backlog, throughput, and response time of the Kafka partition are monitored through JMX to calculate the load weight. The custom KafkaProducer regularly calls the REST API of the load aggregation service to obtain the load weight of each partition, and sends the data to the partition with the lowest load to ensure data load balancing. During the data processing process, SparkStreaming reads data from Kafka and maps it to structured data, then cleans the data and filters out abnormal temperature, humidity, and RFID location information. Through ForkJoinPool parallel optimization, the system recursively splits tasks according to the size of the data set, processes subtasks in parallel, and improves processing efficiency. The cleaned data is evenly written to multiple database nodes to ensure storage load balance.

[0024] Beneficial effects:

[0025] 1. The present invention can greatly improve the processing efficiency and data quality of the system by dynamically monitoring partition loads, cleaning data and performing efficient task scheduling.

[0026] 2. The present invention significantly improves the processing efficiency of sensor data through dynamic load balancing and multi-threaded parallel processing. The system automatically distributes data according to the partition load to avoid overload, and adopts the task block mechanism to speed up data processing. The data cleaning strategy ensures the accuracy and reliability of the data, and the polling storage strategy further optimizes the load balancing of the database nodes. The overall system has the advantages of high efficiency, stability and scalability, which effectively improves the management efficiency and data processing capabilities of the smart park. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the deployment of the system of the present invention.

[0028] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0029] The invention will be further described in detail below in conjunction with the accompanying drawings.

[0030] like Figure 1 and Figure 2 As shown, the present invention provides a sensor data access method for load balancing and parallel optimization in a smart park, the method comprising the following steps:

[0031] Step 1: In the smart park, deploy a variety of sensors: temperature sensors, humidity sensors, and radio frequency identification (RFID) devices. Each device is identified by a unique device identifier Device_ID, the type of device is defined by Device_Type, and the type of data collected is described by Data_Type. The data collected by the sensor and RFID device is called Sensor_Data, which includes the specific value collected Data_Value and the timestamp of the data collection.

[0032] Step 2: Connect the device to the edge gateway through a wireless network and communicate data through the message queue telemetry transport protocol (MQTT protocol). After collecting data, each device packages its Sensor_Data into a message containing Device_ID, Data_Value and Timestamp, and sends the message to the edge gateway through the MQTT protocol.

[0033] Step 3: The edge gateway receives Sensor_Data from multiple devices and formats each piece of data received into a unified JSON object. This JSON object contains the following fields: Device_ID: The unique identifier of the device, the value is the Device_ID obtained from the device. Data_Type: The data type collected by the device, the value is the Data_Type of the device. Data_Value: The data value collected by the device, the value is the Data_Value obtained from the device. Timestamp: The timestamp of the data collected by the device, the value is the Timestamp obtained from the device, accurate to seconds. After the edge gateway completes the JSON format conversion of the data, it forwards the JSON_Data to the data stream processing platform through Kafka_Producer for use by subsequent systems.

[0034] Step 4: Define Kafka topics, where the temperature sensor topic is sensor_temperature_topic, the humidity sensor topic is sensor_humidity_topic, and the RFID sensor topic is rfid_location_topic. Associate all topics with Device_Type so that Kafka Producer can send formatted data to the corresponding topic in Kafka according to the device type Device_Type.

[0035] Step 5: Configure the Kafka cluster. The cluster consists of 3 Brokers, and the replication factor is set to 3. The number of partitions per Kafka topic, partition_count, is set to 10. Each partition is managed by a Leader node, and other Broker nodes store its replicas. The retention policy is configured to retain data for a retention_time of 48 hours.

[0036] Step 6: Design the load aggregation service Load_Aggregation_Service. Use JMX to collect the load data of each partition of KafkaBroker, including:

[0037] Partition message backlog (Partition_Backlog): The message backlog is calculated through the JMX indicators kafka.log:type=Log,name=LogEndOffset and kafka.log:type=Log,name=LogStartOffset. The backlog is:

[0038] LogEndOffset-LogStartOffset, which indicates the number of messages in the partition that have not been consumed.

[0039] Partition processing throughput (Partition_Throughput): Through JMX indicators

[0040] kafka.server:type=BrokerTopicMetrics, which collects production throughput (BytesInPerSec) and consumption throughput (BytesOutPerSec), indicating the number of bytes processed per second by the partition during production and consumption.

[0041] Partition response time (Partition_ResponseTime): via JMX metrics

[0042] kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce and

[0043] kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer, measures the average latency of partitions processing production (Produce) requests and consumption (FetchConsumer) requests respectively.

[0044] The load aggregation service Load_Aggregation_Service collects these load data regularly and calculates the load weight Load_Weight of each partition. The load weight is calculated using the following formula:

[0045] Load_Weight=α×Partition_Backlog+β×Partition_Throughput+γ

[0046] ×Partition_ResponseTime

[0047] Among them, α, β and γ are weight coefficients for adjusting the influence of message backlog, throughput and response time in load weight, and the parameters are adjustable.

[0048] Step 7: Design the load aggregation service REST API interface. This REST API allows KafkaProducer to obtain the latest load weight Load_Weight of each partition before sending a message, so as to determine which partition to route the message to. The specific design is as follows:

[0049] URL http: / / {Load_Aggregation_Service_Host} / api / partition_load Request method GET Request Parameters The name of the Kafka topic to query

[0050] The returned content is as follows:

[0051] Partition_Id Partition unique identifier Load_Weight Current load weight of the partition

[0052] Step 8: Customize the Partitioner of KafkaProducer to call the REST API interface of Load_Aggregation_Service every 1 second to obtain the Load_Weight value of each partition and send the message to the partition with the lowest load weight to complete the load-balanced KafkaProducer data distribution. If the loads are the same, they are distributed randomly.

[0053] Step 9: SparkStreaming connects to Kafka. The spark.readStream method is used to read data from different topics from Kafka and map them to Spark's structured data format SparkDataFrame. Data is pulled from Kafka in micro-batch intervals of 1 second. When processing partitions, each partition in the Kafka topic corresponds to a Spark task. For Kafka topics with multiple partitions, SparkStructuredStreaming creates a parallel Spark task for each Kafka partition to process data, so 10 partitions in Kafka are mapped to 10 Spark parallel tasks, each task processing the data of the corresponding partition.

[0054] Step 10: Use Java's ForkJoinPool class to create a thread pool Thread_Pool for each task. This thread pool supports a work stealing mechanism, that is, when a thread completes its assigned subtask, it can automatically steal tasks from other unfinished threads for processing, avoiding idle resources and improving processing efficiency.

[0055] Step 11: Blocking within Spark tasks. For each Spark task, first determine the size of the data set. If the size of the data set exceeds the predefined threshold Data_Threshold, the data set Task_Data is divided into multiple subsets Data_Subset_i. If the data subset that a subtask Task_i is responsible for is still large and exceeds the minimum threshold Data_Threshold that can be processed in parallel, the task can be further recursively split into smaller subtasks until the size of the data set is small enough so that each task can be efficiently executed within the thread. Recursively submit the split subtasks to Thread_Pool again, and the thread pool continues to process them in parallel.

[0056] Step 12: For each divided data subset Data_Subset_i, create a subtask Task_i for it. Submit these subtasks Task_i to the thread pool Thread_Pool, and multiple threads in the thread pool execute these subtasks in parallel. Each subtask will process the data subset it is responsible for and return partial results after completion. When all threads in Thread_Pool complete their respective subtasks, the processing results of all subtasks will be summarized. The summary operation will merge the sub-results Result_i returned by each thread and form the final processing result Final_Result of the entire task.

[0057] Step 13: Spark data preprocessing and cleaning: Use SparkSQL statements to extract fields from JSON_Data into structured data Parsed_DataFrame.

[0058] Cleaning of temperature sensor data: For temperature sensor data, Data_Type is temperature, and its value must be within a reasonable range, specifically -50°C to 100°C. If the value of a record exceeds this range, the data is filtered out. The cleaned DataFrame is recorded as Cleaned_Temperature_DataFrame.

[0059] Cleaning of humidity sensor data: For humidity sensor data, Data_Type is humidity, and value must be between 0% and 100%. Any record with humidity data below 0% or above 100% will be filtered. The cleaned DataFrame is recorded as Cleaned_Humidity_DataFrame.

[0060] Cleaning of RFID location data: For the location data of RFID devices, Data_Type is location coordinates (location), which should contain valid geographic coordinate information. For longitude and latitude data, the range of latitude should be between -90 and 90, and the range of longitude should be between -180 and 180. If the latitude or longitude in the record exceeds this range, the record will be discarded. The cleaned DataFrame is recorded as Cleaned_RFID_Location_DataFrame.

[0061] Step 14: Define the database node set DB_Nodes = {DB_Node_1, DB_Node_2, ..., DB_Node_k}, where k represents the total number of nodes in the database cluster, and each node is a different database shard.

[0062] Step 15: Set the index variable Node_Index with an initial value of 1. Whenever there is processed data that needs to be written to the database, Node_Index points to the current database node DB_Node_{Node_Index} in the set DB_Nodes, and writes the processed data to the node. After the write operation is completed, Node_Index increments by 1. When the value of Node_Index is greater than k, it reaches the end of the set, and Node_Index is set to 1 again, and polling starts again from the first database node.

[0063] Step 16: After each batch of data is processed, Spark Streaming obtains the starting offset `Start_Offset_i` of the current Kafka partition and the processed ending offset `End_Offset_i` to mark the consumption progress of each partition. After performing the data processing operation, Spark Streaming submits the `End_Offset_i` of each partition to Kafka by calling Kafka's `commitAsync()` method to update the consumption progress. If the submission fails, the exception is captured and the error log is recorded, and the retry mechanism is triggered if necessary. In the event of a failure or restart, Spark Streaming uses Kafka's `seek()` method to resume consumption based on the last submitted offset to ensure that each piece of data is processed only once.

[0064] The implementation cases and specific application scenarios of the present invention include:

[0065] In the smart park, temperature sensors, humidity sensors, and RFID devices are deployed. Various sensors transmit the collected data to the edge gateway through the MQTT protocol. After the edge gateway formats the data, it forwards it through the Kafka cluster, and uses the load aggregation service to dynamically monitor the Kafka partition load and distribute the data to the partition with the lightest load. SparkStreaming pulls sensor data from Kafka in real time for cleaning to ensure that the data is within a reasonable range. Subsequently, the system evenly distributes the processed data to multiple database nodes through a polling strategy to ensure the balance and reliability of data storage.

Claims

1. A sensor data access method for load balancing and parallel optimization in a smart park, characterized in that: The following steps are involved: Step 1: Deploy temperature sensors, humidity sensors, and RFID devices in the smart park. The devices are identified by a unique Device_ID, and the data type is defined by Data_Type. Step 2: Send the Sensor_Data collected by the device to the edge gateway through the MQTT protocol, where the Sensor_Data includes Device_ID, Data_Value and Timestamp; Step 3: The edge gateway receives the Sensor_Data and formats it into a format that includes Device_ID, Data_Type, The JSON object of Data_Value and Timestamp is sent to the Kafka topic associated with the device type using Kafka Producer.

2. According to claim 1, a sensor data access method for load balancing and parallel optimization in a smart park is characterized in that: The method includes a load aggregation service; the load aggregation service collects load data of Kafka partitions through JMX, and the load data includes message backlog of the partitions, partition throughput and partition response time.

3. According to claim 2, a sensor data access method for load balancing and parallel optimization in a smart park is characterized in that: The method includes a load weight; the load weight is calculated according to the following formula: Load Weight =α×Partition Backlog +β×Partition Throughput +γ×Partition ResponseTime Among them, α, β and γ are adjustment parameters, which are used to adjust the influence of each indicator on the load weight.

4. According to claim 1, a sensor data access method for load balancing and parallel optimization in a smart park is characterized in that: The Kafka Producer in step 3 calls the REST API interface of the load aggregation service before sending the message to obtain the latest load weight of each partition, and sends the message to the partition with the lowest load weight to achieve load balancing.

5. According to claim 1, a sensor data access method for load balancing and parallel optimization in a smart park is characterized in that: The method includes Spark Streaming; the Spark Streaming reads data from multiple topics through Kafka, maps each Kafka partition to a parallel Spark task for processing, and parses the data into structured data through JSON for cleaning processing.

6. The sensor data access method for load balancing and parallel optimization in a smart park according to claim 1 is characterized in that: The method includes data cleaning; the data cleaning includes: 1) The temperature sensor data only retains the records with Data_Value in the range of -50℃ to 100℃; 2) Humidity sensor data only retains records with Data_Value in the range of 0% to 100%; 3) RFID location data only retains records of longitude and latitude within the valid range. The sensor data access method for load balancing and parallel optimization in a smart park according to claim 5 is characterized in that: Spark tasks use the ForkJoinPool thread pool for parallel processing. When the amount of data in a task exceeds a predefined threshold, the task is divided into multiple subtasks, which are recursively split into appropriate sizes and processed in parallel. Finally, the results of each subtask are aggregated to generate the final task result.

Images