High-concurrency processing system and method for producing area processing equipment

Abstract

The invention provides a high-concurrency processing system and method for producing area processing equipment, the system comprises a sensing layer, a network layer and an intermediate layer, the sensing layer is used for acquiring equipment data concurrently uploaded by an equipment cluster based on a target data center, and sending the equipment data to the network layer, the target data center is determined according to real-time load state information of a server in the data center; the equipment cluster is a hundred thousand-level equipment cluster; the network layer is composed of a preset number of Kafka clusters, and is used for pushing the received equipment data to a corresponding theme based on a message producer, and performing Kafka information storage by a proxy server cluster by taking a partition as a unit; and the middle layer is used for processing the Kafka information subscribed by the Kafka consumer group and writing the processed data into the database in parallel. According to the invention, the processing efficiency of mass agricultural data is improved.

Description

Technical Field

[0001] This invention relates to the field of agricultural Internet of Things (IoT) technology, and in particular to a high-concurrency processing system and method for on-site processing equipment. Background Technology

[0002] In recent years, agricultural IoT technology has developed rapidly, and the real-time data generated by massive agricultural production equipment has exploded. In the scenario of intelligent processing of agricultural products, it is necessary to ensure that hundreds of thousands of devices are online at the same time and upload data in real time to achieve precise control. This poses a severe challenge to the high concurrency, low latency and efficient storage capabilities of the data processing system.

[0003] Existing high-concurrency access architectures struggle to manage hundreds of thousands of device connections and sessions, easily leading to server resource exhaustion and service crashes. Furthermore, data processing and transmission efficiency is low, lacking traffic shaping and load balancing mechanisms, impacting message middleware and causing message backlog and increased latency. Meanwhile, data storage and writing become performance bottlenecks, with traditional writing methods being inefficient and unable to meet real-time monitoring and control requirements.

[0004] Existing technologies lack global traffic scheduling and fine-grained resource optimization strategies, resulting in low resource utilization, poor architectural scalability, and difficulty in fully utilizing distributed system resources to meet the dynamic demands of continuously growing equipment scale. Furthermore, they have no application in controlling IoT devices at the origin of hundreds of thousands of agricultural products. Therefore, there is an urgent need for a high-concurrency processing system and method for origin-based processing equipment to address these issues. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides a high-concurrency processing system and method for on-site processing equipment.

[0006] This invention provides a high-concurrency processing system for on-site processing equipment, comprising a sensing layer, a network layer, and a middleware layer, wherein: The perception layer is used to acquire device data uploaded concurrently by the device cluster based on the target data center, and send the device data to the network layer. The target data center is determined based on the real-time load status information of the servers in the data center; the device cluster is a cluster of hundreds of thousands of devices. The network layer consists of a preset number of Kafka clusters, which are used to push the received device data to the corresponding topic based on the message producer, and the proxy server cluster stores the Kafka information in units of partitions. The intermediate layer is used to process the Kafka information subscribed to by the Kafka consumer group and write the processed data into the database in parallel.

[0007] According to the present invention, a high-concurrency processing system for on-site processing equipment is provided, the system further comprising an application platform layer, wherein: The application platform layer is used to generate corresponding data display content for the user based on the data processed by the intermediate layer.

[0008] According to the high-concurrency processing system for origin processing equipment provided by the present invention, the sensing layer further includes a global load balancing layer, wherein: The global load balancing layer is used to determine the target data center corresponding to the device data to be uploaded by the device cluster through a DNS server, based on the IP address information of the device cluster, the real-time load status of the servers in each data center, and the round-robin algorithm.

[0009] According to the high-concurrency processing system for origin processing equipment provided by the present invention, the sensing layer further includes a front-end load balancing layer, wherein: The front-end load balancing layer is composed of an EMQX message middleware cluster, which is used to preprocess the device data and send the preprocessed device data to the network layer based on a preset distribution strategy. The data preprocessing includes data cleaning, data format conversion and traffic shaping.

[0010] According to the high-concurrency processing system for origin processing equipment provided by the present invention, the network layer is further used to adjust the parameters of the Kafka cluster according to the preset parameter tuning instructions to obtain the Kafka cluster with adjusted parameters.

[0011] According to the present invention, a high-concurrency processing system for origin processing equipment is provided, wherein the network layer manages the metadata of the Kafka cluster through Zookeeper.

[0012] According to the high-concurrency processing system for on-site processing equipment provided by the present invention, the intermediate layer is specifically used for: Based on the database connection pool and the preset thread pool, the Kafka information subscribed by the Kafka consumer group is processed; The processed data is written to the database in parallel. The database connection pool consists of multiple database connections that are continuously maintained. The preset thread pool consists of multiple threads.

[0013] The present invention also provides a high-concurrency processing method for origin processing equipment based on the above-mentioned high-concurrency processing system for origin processing equipment, comprising: Based on the target data center, acquire the device data uploaded concurrently by the device cluster; wherein, the target data center is determined based on the real-time load status information of the servers within the data center; the device cluster is a cluster of hundreds of thousands of devices; The message producer pushes the received device data to the corresponding topic, and the proxy server cluster stores the Kafka information in units of partitions. The Kafka consumer group processes the Kafka messages it subscribes to and writes the processed data to the database in parallel.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the high-concurrency processing method of the origin processing device as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the high-concurrency processing method of the origin processing equipment as described above.

[0016] The high-concurrency processing system and method for on-site processing equipment provided by this invention involves a perception layer that acquires data uploaded concurrently by a cluster of hundreds of thousands of devices through a target data center and transmits it to the network layer. The network layer consists of a preset number of Kafka clusters, which push data to corresponding topics based on message producers, and the proxy server cluster stores the data in partitions. Then, the middle layer processes the subscription information of Kafka consumer groups and writes it to the database in parallel, thereby improving the processing efficiency of massive agricultural data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the high-concurrency processing system of the origin processing equipment provided by the present invention; Figure 2 A schematic diagram of EMQX load balancing provided by the present invention; Figure 3 This is a schematic diagram of the Kafka architecture provided by the present invention; Figure 4 A schematic flowchart illustrating the high-concurrency processing method for the origin processing equipment provided by the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] With the rapid development of agricultural IoT technology, the real-time data generated by massive agricultural processing equipment, such as agricultural machinery and sensors, is growing explosively, posing a severe challenge to the high concurrency, low latency, and efficient storage capabilities of data processing systems. Especially in intelligent agricultural product processing scenarios, it is necessary to ensure that hundreds of thousands of devices are online simultaneously and uploading data in real time to achieve precise control.

[0021] However, existing technical solutions generally suffer from the following bottlenecks: First, the high-concurrency access capability is insufficient. Traditional architectures struggle to effectively manage persistent connections and session persistence for hundreds of thousands of devices. A large number of concurrent connection requests can easily lead to server resource exhaustion, a surge in response latency, or even service crashes.

[0022] Second, the data processing and transmission efficiency is low. With massive amounts of data from devices flooding in instantly, the existing technology lacks efficient traffic shaping and load balancing mechanisms, which can easily impact core message middleware, such as Kafka clusters, leading to message backlog, decreased throughput, and increased average latency.

[0023] Third, data storage and writing become a performance bottleneck. Traditional database writing methods, such as single-threaded synchronous writing, are inefficient and cannot maintain the writing requirements of high-throughput data streams. This becomes a performance bottleneck in the entire data processing chain, leading to data backlog and storage delays, which cannot meet the requirements of real-time monitoring and control.

[0024] Fourth, the system suffers from low resource utilization and poor scalability. It lacks a global perspective on traffic scheduling and fine-grained resource optimization strategies, making it difficult to fully utilize distributed system resources. Furthermore, its architecture lacks scalability, making it difficult to cope with the dynamic demands of continuously growing equipment scale.

[0025] To address the key issues in existing technologies for high-concurrency agricultural IoT device data processing, such as difficulties in managing massive connections, low data transmission efficiency, significant storage and write bottlenecks, and insufficient resource utilization, this invention proposes a high-concurrency system for on-site processing devices based on Kafka two-level load balancing. This system aims to provide a distributed data management solution with high throughput, low latency, fast storage, high resource efficiency, and scalability for the agricultural IoT field, particularly in scenarios involving hundreds of thousands of on-site processing devices. It meets the urgent engineering needs for efficient processing and accurate monitoring of massive real-time device data, thereby helping to improve the level of intelligent management in agricultural production.

[0026] Figure 1 This is a schematic diagram of the high-concurrency processing system of the on-site processing equipment provided by the present invention, as shown below. Figure 1 As shown, the present invention provides a high-concurrency processing system for on-site processing equipment, comprising a sensing layer 101, a network layer 102, and an intermediate layer 103, wherein: The perception layer 101 is used to acquire device data uploaded concurrently by the device cluster based on the target data center, and send the device data to the network layer 102. The target data center is determined based on the real-time load status information of the servers in the data center; the device cluster is a cluster of hundreds of thousands of devices. The network layer 102 is composed of a preset number of Kafka clusters, which are used to push the received device data to the corresponding topic based on the message producer, and the proxy server cluster stores Kafka information in units of partitions. The intermediate layer 103 is used to process the Kafka information subscribed by the Kafka consumer group and write the processed data into the database in parallel.

[0027] In this invention, the on-site processing equipment includes, but is not limited to, sorting and grading equipment, precooling equipment, packaging equipment, and storage equipment, which together form a cluster of hundreds of thousands of devices.

[0028] The perception layer 101 acquires device data uploaded concurrently by the device cluster based on the target data center. In this invention, the target data center is not fixed but dynamically determined according to the real-time load status information of the servers within the data center. This approach allows for more rational allocation of data processing tasks, avoiding overloading of individual servers. After acquiring the device data, the perception layer 101 sends the device data to the network layer.

[0029] Network layer 102 consists of a pre-defined number of Kafka clusters. Kafka is a high-performance distributed stream processing platform used to process large-scale real-time data streams. In network layer 102, message producers push device data sent from perception layer 101 to corresponding topics; different device data can be pushed to different topics. Simultaneously, a broker server cluster stores Kafka information in units of partitions.

[0030] In this invention, the partitioning design improves Kafka's concurrent processing capabilities and data storage scalability, enabling the efficient and orderly storage and management of large amounts of device data. For example, data from different types of devices can be stored in different partitions, facilitating subsequent querying and processing.

[0031] The middleware layer 103 is responsible for processing the Kafka information subscribed to by the Kafka consumer group. In this invention, a Kafka consumer group can consist of multiple consumers that collectively consume data from Kafka topics. After receiving this data, the middleware layer 103 performs a series of processing operations to meet the needs of subsequent business operations. After processing, the middleware layer 103 writes the obtained data to the database in parallel. In this invention, the parallel writing method can greatly improve the efficiency of data storage, ensuring that massive amounts of device data can be stored in the database in a timely and accurate manner, providing support for subsequent data analysis and applications.

[0032] In this invention, the middleware layer 103 provides a stable and secure operating environment for the business server and application server through CentOS (Community Enterprise Operating System). It can effectively manage the server's hardware resources, such as CPU, memory, and disk, ensuring that various applications on the server can run efficiently and stably. For example, CentOS can optimize the allocation of system resources, preventing one application from excessively consuming resources and affecting the normal operation of other programs.

[0033] Meanwhile, in the middle layer 103, Ubuntu provides a runtime environment for the server, making it easier to install and manage various development tools and applications, thus facilitating system development and maintenance.

[0034] At the database level, the middleware layer 103 uses PostgreSQL to store and manage processed data. PostgreSQL can efficiently handle large numbers of data read and write operations, ensuring data consistency and integrity. For example, when writing processed device data to the database, PostgreSQL uses a transaction mechanism to ensure that the data is either all written successfully or rolled back completely in case of an exception, avoiding data inconsistencies. Simultaneously, it also supports complex data queries and analysis, providing accurate data support for the application platform layer.

[0035] In this invention, the business server primarily undertakes processing tasks related to business logic within the system. In the high-concurrency processing system for production site equipment, it is responsible for further processing and analysis of Kafka information obtained from network layer 102. For example, the business server aggregates equipment data, calculates key indicators (such as average equipment uptime, production statistics, etc.), and performs data filtering and transformation operations according to business rules. In this invention, the business server processes data in a targeted manner based on the needs of the application platform layer, providing valuable information for upper-layer applications.

[0036] The application server is primarily responsible for interacting with the application platform layer, providing data services and application support to user devices such as computers and mobile phones. In this invention, the application server receives requests from the application platform layer 104, retrieves relevant data from the database, performs necessary processing and formatting, and then returns it to the user. For example, when a user queries the device's operating status through a mobile application, the application server retrieves relevant data from the PostgreSQL database, converts it into a format suitable for mobile display, and then sends it to the user's phone. Simultaneously, the application server can also process user input and operation commands, feeding them back to the business server for further processing.

[0037] In this invention, the middle layer 103 provides a stable operating environment through CentOS and Ubuntu, uses PostgreSQL for data storage and management, and completes data processing and interaction with the application platform layer with the help of business servers and application servers, thereby realizing efficient processing and flow of data for the entire system and providing a strong guarantee for the efficient operation of the high-concurrency processing system of the production site processing equipment.

[0038] The high-concurrency processing system for origin processing equipment provided by this invention has the following steps: the perception layer obtains data uploaded concurrently by a cluster of hundreds of thousands of devices through the target data center and transmits it to the network layer; the network layer consists of a preset number of Kafka clusters, which push data to the corresponding topics based on message producers, and the proxy server cluster stores the data in partitions; then the middle layer processes the subscription information of Kafka consumer groups and writes it to the database in parallel, thereby improving the processing efficiency of massive agricultural data.

[0039] Based on the above embodiments, you can refer to Figure 1 As shown, the system also includes an application platform layer 104, wherein: The application platform layer 104 is used to generate corresponding data display content for the user terminal based on the data processed by the intermediate layer 103.

[0040] In this invention, the application platform layer 104 encompasses user-end devices such as computers and mobile phones, and is also associated with the agricultural product origin information intelligent management and control big data service platform (hereinafter referred to as the service platform). These user-end devices serve as the window for interaction between the user end and the system, while the service platform is the core backend supporting data display and business logic operation.

[0041] Specifically, the main function of the application platform layer 104 is to generate corresponding data display content for the user based on the data processed by the middle layer 103. The middle layer 103 has already processed the Kafka information obtained from the network layer 102 and written the processed data into the database. This data contains various information about the processing equipment at the production site, such as equipment operating status, working parameters, and production data.

[0042] Application platform layer 104 extracts this processed data from the database and then processes and organizes it according to different user needs and business scenarios. For example, for managers of agricultural production areas, application platform layer 104 may generate real-time monitoring charts of equipment operating status, displaying key indicators such as the uptime and failure rate of each piece of equipment; for production decision-makers, it may generate statistical reports on production data, such as output statistics for different time periods and equipment efficiency analysis.

[0043] In this invention, users can intuitively view the data displays through user-end devices such as computers or mobile phones, thereby gaining timely understanding of the operation and production status of the processing equipment at the production site and making corresponding decisions and management operations. For example, when an abnormal operating state of a piece of equipment is detected, managers can promptly notify maintenance personnel through their mobile phones to handle the situation and ensure normal production.

[0044] The application platform layer 104 transforms the data processed by the intermediate layer 103 into information that users can understand and use, realizing the value output of system data and providing strong support for the intelligent management and control of agricultural product production areas.

[0045] Based on the above embodiments, the perception layer further includes a global load balancing layer, wherein: The global load balancing layer is used to determine the target data center corresponding to the device data to be uploaded by the device cluster through a DNS server, based on the IP address information of the device cluster, the real-time load status of the servers in each data center, and the round-robin algorithm.

[0046] In this invention, the implementation of the global load balancing layer relies on a deployed global load balancing system based on the Domain Name System (DNS). The DNS server can accurately guide data upload requests from devices to the appropriate data center.

[0047] When a device in the cluster is ready to upload data, it sends a request to the DNS server. The global load balancing layer can determine the geographical location of the device by obtaining the IP address information of the cluster. This invention uses the IP address to determine the network area where the device is located, providing basic geographical information for subsequently selecting the nearest data center.

[0048] In this invention, the real-time load status of servers within each data center is also a crucial decision-making factor. The global load balancing layer collects real-time load data on servers in different data centers, including metrics such as CPU utilization, memory usage, and network bandwidth utilization. By understanding the real-time server load, data requests can be avoided from being sent to overloaded data centers, ensuring the efficiency and stability of data processing.

[0049] Simultaneously, a round-robin algorithm is employed to distribute data requests to different data centers in a specific order. When combined with a location-aware strategy, the round-robin algorithm can fairly allocate requests among multiple geographically compatible and reasonably loaded data centers, further optimizing resource utilization.

[0050] In this invention, the DNS server, as a core component of the global load balancing layer, comprehensively considers the IP address information of the device cluster, the real-time load status of servers in each data center, and a round-robin algorithm to determine the target data center corresponding to the device data to be uploaded. Specifically, when a device sends a data upload request, the DNS server first determines the approximate geographical location of the device based on the request source IP and filters out the nearest data center cluster. Then, combining the real-time load status of servers in these data centers, it uses a round-robin algorithm to select from the available data centers, ultimately determining the most suitable target data center. In this way, the global load balancing layer forms an intelligent traffic scheduling hub, which can dynamically allocate the upload target of device data according to the actual situation, avoiding overload of a single data center due to too many requests, while effectively reducing cross-regional transmission latency by more than 40%, thus improving the performance and reliability of the entire system.

[0051] Based on the above embodiments, the perception layer further includes a front-end load balancing layer, wherein: The front-end load balancing layer consists of an EMQX (Erlang / Enterprise / Elastic MQTT Broker) message middleware cluster, which is used to preprocess the device data and send the preprocessed device data to the network layer based on a preset distribution strategy. The data preprocessing includes data cleaning, data format conversion and traffic shaping.

[0052] In this invention, the front-end load balancing layer consists of an EMQX message middleware cluster. EMQX is a high-performance, scalable, open-source MQTT message middleware applicable to the Internet of Things (IoT) field, capable of efficiently handling massive device connections and data transmission.

[0053] In IoT scenarios, the data uploaded by devices may contain a large amount of invalid or erroneous information, such as noise data generated by sensors under abnormal conditions. The rule engine built into the EMQX message middleware cluster can filter this raw data, removing invalid sensor data and other impurities, ensuring that the data processed subsequently is accurate.

[0054] Meanwhile, processing equipment at the production site may come from different manufacturers and have different models, resulting in variations in the data formats uploaded. The EMQX message middleware can convert data uploaded from various heterogeneous devices into a unified format, facilitating subsequent storage and processing. For example, it can standardize the date and time formats used by different devices into a single standard format.

[0055] Furthermore, when a large number of devices upload data simultaneously, it can severely impact the backend Kafka cluster, leading to message backlog and system performance degradation. This invention employs EMQX to implement traffic shaping (such as filtering invalid sensor data) to regulate the data traffic uploaded by devices. For example, data transmission can be limited at a certain rate to avoid excessive instantaneous traffic, ensuring stable data transmission and the stability of the backend system.

[0056] Furthermore, Figure 2 This is a schematic diagram of EMQX load balancing provided by the present invention, which can be referred to. Figure 2 As shown, after completing data preprocessing, the EMQX message middleware cluster sends the processed device data to the network layer based on a preset distribution strategy. For example, through round-robin distribution, data is sent to different Kafka cluster topics in a certain order. In this way, balanced data distribution can be achieved, avoiding excessive data pressure on a single Kafka cluster and improving the overall system's concurrent processing capacity and data transmission efficiency.

[0057] In this invention, the front-end load balancing layer uses the EMQX message middleware cluster to preprocess and distribute device data appropriately, laying a good foundation for subsequent network layer data processing and storage, and effectively improving the system's performance and stability.

[0058] Based on the above embodiments, the network layer is also used to adjust the parameters of the Kafka cluster according to the preset parameter tuning instructions to obtain the Kafka cluster with adjusted parameters.

[0059] In this invention, to improve the efficiency of the Kafka cluster in processing massive amounts of data uploaded from hundreds of thousands of devices, the network layer adjusts the parameters of the Kafka cluster according to preset parameter tuning instructions. These preset instructions are formulated based on the analysis of Kafka cluster performance bottlenecks and consideration of overall system requirements, thereby fully leveraging the high concurrency processing capabilities of the Kafka cluster and ensuring that data can be transmitted and stored in the system efficiently and stably.

[0060] In one embodiment, the Kafka cluster is optimized by optimizing the following parameters, as shown in Table 1: Table 1 Kafka Cluster Parameter Table

[0061] In Table 1, setting linger.ms to 50 indicates that the producer will wait up to 50 milliseconds when sending messages to allow for batch sending of more messages. This reduces the number of network requests, improves network utilization, and thus enhances data transmission efficiency.

[0062] Set `batch.size` to 65536, which is 64KB. When the message size reaches this batch capacity, the producer will send out that batch of messages. Properly setting the batch capacity can further improve the efficiency of batch data processing while ensuring a certain level of real-time performance.

[0063] Setting compression.type=lz4 enables the LZ4 compression algorithm, which can effectively reduce network load, decrease the size of data transmitted over the network, and improve transmission speed without significantly increasing CPU load.

[0064] Setting buffer.memory to 268435456, or 256MB, provides sufficient buffer memory for the producer to store messages to be sent, thus avoiding message sending failures or delays due to insufficient memory.

[0065] Furthermore, a unique broker.id is set for each Kafka broker, such as broker.id=1, to uniquely identify each node in the cluster and ensure the accuracy and stability of cluster management.

[0066] Configure the listening interface as follows: listeners=PLAINTEXT: / / 0.0.0.0:9092; advertised.listeners=PLAINTEXT: / / IP:9092; This specifies the interface address that the Kafka cluster listens on and the address that it advertises to the outside world, ensuring that producers and consumers can connect to the Kafka cluster correctly.

[0067] Setting `num.network.threads=10` and `num.io.threads=20` specifies the number of threads used to handle network requests and input / output operations, respectively. A reasonable number of threads can fully utilize server resources and improve the cluster's concurrent processing capabilities.

[0068] Adjust socket.send.buffer.bytes and socket.receive.buffer.bytes to 1024000, or 1000KB, to set appropriate send and receive buffer sizes and optimize network transmission performance.

[0069] Set `num.partitions` to 5 to determine the number of partitions for each topic. Properly setting the number of partitions can improve the parallel processing capability of the Kafka cluster, allowing data to be distributed more evenly across partitions and improving data read / write efficiency.

[0070] In this embodiment, after adjusting the Kafka cluster parameters, 100,000 messages were sent using the Kafka cluster for testing. By observing metrics such as message transmission throughput, latency, and success rate, the performance improvement of the Kafka cluster after parameter adjustment was verified. If the test results are unsatisfactory, the parameters can be further adjusted until the Kafka cluster can meet the system's requirements for high-concurrency data processing. In this invention, a detection component is used for testing, and the specific operating efficiency of Kafka is viewed. The results are shown in Table 2. Table 2 Performance under different message volumes

[0071] In Table 2, with a five-node Kafka cluster, the producer throughput reached 29,687 messages / second, and the average latency was 188.5ms for 100,000 messages, meeting the needs of 100,000 IoT devices online simultaneously.

[0072] In this invention, the network layer ensures that the Kafka cluster can run efficiently and stably after parameter adjustment and verification, providing reliable data transmission and storage support for the high-concurrency processing system of the entire production site processing equipment.

[0073] Based on the above embodiments, the network layer manages the metadata of the Kafka cluster through Zookeeper.

[0074] In this invention, Zookeeper provides a reliable metadata management framework for the Kafka cluster. Taking a five-node Kafka cluster as an example, Zookeeper plays a core coordinating and management role, ensuring the stable and efficient operation of the Kafka cluster and taking charge of coordinating various resources and information within the Kafka cluster.

[0075] Figure 3 This is a schematic diagram of the Kafka architecture provided by the present invention, which can be referred to. Figure 3 As shown, the Kafka architecture comprises four core components: Broker Information: In a Kafka cluster, the broker is a distributed storage node. ZooKeeper records relevant information for each broker, such as its unique identifier, IP address, and port number. This information is crucial for message producers and consumers to correctly connect to the appropriate broker. For example, when a message producer wants to push data, it needs to obtain a list of available brokers from ZooKeeper to send the message to the correct node.

[0076] Topic Configuration: A Topic is a category or topic of messages in Kafka. Different data can be pushed to different Topics. Zookeeper stores the configuration information of Topics, including the Topic name, number of partitions, number of replicas, etc. Using this configuration information, Kafka can store and manage messages according to predefined rules. For example, messages can be distributed across different brokers based on the number of partitions, improving the system's parallel processing capabilities.

[0077] Partition Status: A partition is the physical unit of message storage in Kafka. Zookeeper monitors and records the status of each partition in real time to ensure reliable message storage and retrieval. When a broker fails, Zookeeper can promptly coordinate other brokers to take over the work of the relevant partition, ensuring data integrity and availability.

[0078] In this invention, Kafka persistently stores the aforementioned metadata in ZooKeeper, ensuring data durability and consistency. Even if some nodes in the Kafka cluster fail, the metadata stored in ZooKeeper can still be used to restore the normal operation of the cluster.

[0079] In addition to metadata management, ZooKeeper also handles coordination functions such as broker status monitoring and consumer load balancing. ZooKeeper monitors the operational status of brokers in real time, and if any broker malfunctions, it promptly notifies the Kafka cluster to make appropriate adjustments. Simultaneously, for consumer groups, ZooKeeper rationally allocates message subscription and retrieval tasks based on the number of consumers and their load, achieving load balancing and improving message consumption efficiency.

[0080] The network layer manages the metadata of the Kafka cluster through Zookeeper, enabling the Kafka cluster to run in an orderly and coordinated environment, providing a solid guarantee for the system to process massive amounts of device data.

[0081] Based on the above embodiments, the intermediate layer is specifically used for: Based on the database connection pool and the preset thread pool, the Kafka information subscribed by the Kafka consumer group is processed; The processed data is written to the database in parallel. The database connection pool consists of multiple database connections that are continuously maintained. The preset thread pool consists of multiple threads.

[0082] In this invention, the middle layer mainly relies on the database connection pool and the preset thread pool to process the Kafka information subscribed by the Kafka consumer group. By making full use of the advantages of multi-threading and connection reuse, the efficiency of data processing and writing is improved to meet the high concurrency processing requirements brought about by massive device data.

[0083] Specifically, in this invention, the database connection pool consists of multiple persistently maintained database connections. During system operation, these connections remain active, avoiding the overhead of frequently creating and closing database connections. For example, a minimum of 5 connections are maintained, and the maximum can be expanded to 20 connections, dynamically adjusting connection usage based on the actual system load.

[0084] In one embodiment, when database operations are required, the middleware layer can directly obtain an established connection from the connection pool without having to re-establish and authenticate the connection, which greatly shortens the database access response time and improves the overall system performance.

[0085] The default thread pool consists of multiple threads. In one embodiment, the system sets up 10 threads for parallel processing of Kafka messages. These threads are in a waiting state and can immediately begin working as soon as a Kafka message needs to be processed.

[0086] This invention designs a thread-pool-based asynchronous write engine for PostgreSQL. It utilizes multi-core CPU resources to achieve multi-threaded parallel processing. The multi-threading advantage of the Kafka consumer is leveraged, allowing received messages to be submitted in batches to worker threads in the thread pool. Each thread can independently process a portion of the messages, thus fully utilizing the system's computing resources and improving message processing speed. For example, through the `process_message_batch` function, after a thread obtains a connection from the connection pool, it performs batch insertion operations on the parsed JSON messages.

[0087] In this invention, the middleware layer first subscribes to Kafka messages through a Kafka consumer group, and then distributes these messages to threads in a pre-defined thread pool for processing. The threads perform operations such as parsing the received Kafka messages to prepare for subsequent data writing.

[0088] The processed data obtains connections from the database connection pool and is written to the database in parallel. During the write process, a transaction mechanism is employed, automatically rolling back in case of exceptions to ensure data consistency. Simultaneously, the system uses a dynamic caching mechanism to balance throughput and latency, continuously monitoring processing speed per second to ensure that remaining messages are fully processed and resources are released when the system is shut down.

[0089] This invention, based on the design of a database connection pool and a pre-defined thread pool, achieves a high degree of parallelism in database write operations through the coordinated optimization of connection reuse, batch I / O, and concurrent processing. Performance tests were conducted for different data write methods to verify that this invention combines performance and stability. Specific performance comparison results can be found in Tables 3 and 4. Table 3 CPU Performance Comparison

[0090] Table 4 Comparison of Transmission Efficiency

[0091] Tables 3 and 4 show the comparative test results, demonstrating that the Kafka consumer employing a thread pool mechanism achieved an average write rate of up to 15,000 records per second, a 5-fold improvement compared to the 3,000 records per second of a single consumer, and over 12.5 times the 1,200 records per second of direct writes without middleware. The write time was also significantly reduced from 80 seconds to 7 seconds. Furthermore, this invention significantly optimized system I / O, increasing TPS (transactions per second), disk write volume, and the number of single I / O operations, while reducing the average I / O wait time, thus achieving a Kafka-PostgreSQL data processing solution that combines performance and stability.

[0092] In one embodiment, the high-concurrency processing system of the origin processing equipment provided by the present invention is verified for high concurrency. Its ability to handle tens of thousands of concurrent requests is verified using the Locust stress testing framework. The test environment is built using Python, and the setup process is as follows: First, import the Locust framework and Kafka producer-related modules, define the KafkaProducerUser class which inherits from the User class, and set a random wait interval of 0.1 to 0.5 seconds.

[0093] During the initialization phase, a Kafka Producer instance is created using a singleton _producer, and multiple Broker addresses, full replica confirmation, idempotency, and failure retry mechanisms are configured.

[0094] In the core task of send_message, a random string is generated as the message payload and sent asynchronously to the load_test topic through the produce method. The delivery_report callback function is bound to it to collect real-time statistics on metrics such as throughput, latency, and message size.

[0095] Calling poll(1) triggers the callback execution, and flush(timeout=10) ensures that all messages are sent when the call stops. The test process displays performance metrics such as P99 latency and error rate through the Locust Web UI, and outputs the success / failure status of each message to the console, realizing a comprehensive evaluation of Kafka producer throughput, latency and reliability, which is suitable for capacity planning and high availability verification scenarios.

[0096] After setup, testing was conducted. Initially, high-concurrency data was not handled using load balancing or Kafka; instead, the system directly simulated 100,000 users online simultaneously. However, to prevent system crashes due to excessive user volume, a batch startup method was adopted, with 5,000 users per batch for long-connection testing. Of all generated user connections, approximately 70,000 connections timed out, approximately 30,000 connections failed, and some failed to connect successfully. Only less than 2,000 connections were successful, with latency exceeding 4000ms. Of all user connections, only 50% were running normally, 35% were waiting due to insufficient threads, and 15% were deadlocked. This indicates that without using the invented load balancing solution, directly managing concurrent connections of 100,000 users is quite difficult.

[0097] After adopting the high-concurrency processing system of the origin processing equipment provided by this invention, users can gradually generate and stably maintain 100,000 long connections. The main indicators are the changes in requests per second (RPS) and response time over the system operation time. According to the change graph obtained from the test results, the relevant indicators fluctuate greatly at the beginning and then gradually stabilize, while the average latency is controlled at around 188.5 milliseconds.

[0098] This invention designs a two-tier load balancing architecture—global DNS load balancing and EMQX front-end load balancing—combining a round-robin algorithm with a geolocation-aware strategy to intelligently schedule cross-datacenter traffic, optimizing the data access path and load distribution of the Kafka cluster and significantly improving message transmission efficiency. Furthermore, the Kafka cluster is used for further load balancing of incoming messages, reducing message load and facilitating subsequent processing. By optimizing core Kafka parameters such as batch size, compression method, buffer, and database write mechanism, hardware resource utilization efficiency, such as CPU and I / O, is maximized. Simultaneously, the distributed architecture design provides a foundation for horizontal system scaling, adapting to future device growth. This invention overcomes the database write bottleneck by proposing and implementing an asynchronous batch write solution based on a thread pool. It deeply integrates Kafka's multi-consumer features, database connection pools, and multi-threaded parallel processing technology, increasing database write speed by an order of magnitude.

[0099] Figure 4 The flowchart illustrates the high-concurrency processing method for the origin processing equipment provided by this invention, as shown below. Figure 4 As shown, the present invention provides a high-concurrency processing method for a production site processing equipment based on the high-concurrency processing system of the production site processing equipment described in the above embodiments, comprising: Step 401: Based on the target data center, obtain the device data uploaded concurrently by the device cluster; wherein, the target data center is determined based on the real-time load status information of the servers within the data center; the device cluster is a cluster of hundreds of thousands of devices.

[0100] In this invention, when processing data uploaded by a cluster of hundreds of thousands of devices, the data is not fixedly transmitted to a specific data center. Instead, the target data center is dynamically determined based on the real-time load status information of the servers within the data center. This real-time server load status information can include metrics such as CPU utilization, memory usage, and network bandwidth utilization. This approach allows for the rational allocation of data processing tasks, preventing a single data center from being overloaded and impacting overall performance.

[0101] The various devices in the equipment cluster (such as sorting and grading equipment, pre-cooling equipment, packaging equipment, and storage equipment) generate a large amount of data during operation. Based on a predetermined target data center, the system acquires the concurrently uploaded device data from these devices. For example, temperature, humidity, and other related data generated by the pre-cooling equipment during operation will be uploaded to the target data center according to the system's guidance.

[0102] Step 402: The message producer pushes the received device data to the corresponding topic, and the proxy server cluster stores the Kafka information in units of partitions.

[0103] In this invention, based on the message producer, the device data received in step 601 is pushed to the corresponding topic. A topic can be understood as a data category or channel; different device data can be pushed to different topics. For example, data from sorting and grading equipment can be pushed to the "Sorting and Grading Topic," and data from packaging equipment can be pushed to the "Packaging Topic," etc.

[0104] Furthermore, the proxy server cluster stores Kafka information in units of partitions. This partitioning design enhances Kafka's concurrent processing capabilities and data storage scalability, enabling efficient and orderly storage and management of large amounts of device data. For example, each topic can be divided into multiple partitions, and different partitions can be stored on different proxy servers, thereby achieving distributed data storage and improving system storage efficiency and reliability.

[0105] Step 403: Process the Kafka information subscribed to by the Kafka consumer group and write the processed data into the database in parallel.

[0106] In this invention, Kafka information subscribed to by a Kafka consumer group is processed. A Kafka consumer group can consist of multiple consumers that collectively consume data from Kafka topics. During processing, operations such as parsing, cleaning, and transformation may be performed on the data to meet subsequent business needs. For example, parsening JSON-formatted device data into a data structure recognizable by the system, and removing invalid or erroneous data.

[0107] Furthermore, the processed data is written to the database in parallel. This invention, through parallel writing, significantly improves data storage efficiency, ensuring that massive amounts of equipment data are stored in the database promptly and accurately. For example, multi-threading technology is used to simultaneously write multiple sets of processed data to the database, reducing data writing wait time and supporting subsequent data analysis and applications. Through this collaborative process, efficient collection, storage, and processing of high-concurrency equipment data generated by hundreds of thousands of processing devices at the production site are achieved, providing strong data support for applications such as intelligent management and control of agricultural product production site information.

[0108] The high-concurrency processing method for origin processing equipment provided by this invention involves the perception layer acquiring data uploaded concurrently by a cluster of hundreds of thousands of devices through the target data center and transmitting it to the network layer; the network layer consists of a preset number of Kafka clusters, which push data to the corresponding topics based on message producers, and the proxy server cluster stores the data in partitions; then the middle layer processes the subscription information of Kafka consumer groups and writes it to the database in parallel, thereby improving the processing efficiency of massive agricultural data.

[0109] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communication bus 504. The processor 501, communications interface 502, and memory 503 communicate with each other via the communication bus 504. The processor 501 can call logical instructions in the memory 503 to execute a high-concurrency processing method for the production processing device. This method includes: acquiring device data uploaded concurrently by the device cluster based on a target data center; wherein the target data center is determined based on real-time server load status information within the data center; the device cluster is a cluster of hundreds of thousands of devices; pushing the received device data to the corresponding topic based on a message producer, and storing the Kafka information in partitions by a proxy server cluster; processing the Kafka information subscribed to by the Kafka consumer group, and writing the processed data into a database in parallel.

[0110] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the high-concurrency processing method for production processing equipment provided by the above methods. The method includes: acquiring device data uploaded concurrently by a device cluster based on a target data center; wherein the target data center is determined based on the real-time load status information of the servers in the data center; the device cluster is a cluster of hundreds of thousands of devices; pushing the received device data to the corresponding topic based on a message producer, and storing Kafka information by a proxy server cluster in units of partitions; processing the Kafka information subscribed to by the Kafka consumer group, and writing the processed data into a database in parallel.

[0112] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the high-concurrency processing method for origin processing equipment provided in the above embodiments. The method includes: acquiring device data concurrently uploaded by a device cluster based on a target data center; wherein the target data center is determined based on real-time load status information of servers within the data center; the device cluster is a cluster of hundreds of thousands of devices; pushing the received device data to the corresponding topic based on a message producer, and storing the Kafka information in partitions by a proxy server cluster; processing the Kafka information subscribed to by the Kafka consumer group, and writing the processed data into a database in parallel.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high concurrency processing system for a production site handling device, characterized by, The system comprises a perception layer, a network layer and an intermediate layer, wherein: The perception layer is configured to acquire device data uploaded concurrently by a device cluster based on a target data center, and send the device data to the network layer, wherein the target data center is determined according to real-time load state information of servers in the data center; and the device cluster is a 100,000-level device cluster. The network layer is composed of a preset number of Kafka clusters, configured to push the received device data to a corresponding topic based on a message producer, and store Kafka information in a partition unit by a proxy server cluster. The intermediate layer is configured to process Kafka information subscribed by a Kafka consumer group, and write data obtained after processing to a database in parallel.

2. The origin processing device high concurrency processing system of claim 1, wherein, The system further comprises an application platform layer, wherein: The application platform layer is configured to generate corresponding data display content for a user end according to data processed by the intermediate layer.

3. The origin processing device high concurrency processing system of claim 1, wherein, The perception layer further comprises a global load balancing layer, wherein: The global load balancing layer is configured to determine the target data center corresponding to device data to be uploaded by the device cluster by a DNS server according to IP address information of the device cluster, real-time load states of servers in each data center and a polling algorithm.

4. The origin processing device high concurrency processing system of claim 1, wherein, The perception layer further comprises a front load balancing layer, wherein: The front load balancing layer is composed of an EMQX message middleware cluster, configured to perform data preprocessing on the device data, and send the device data after data preprocessing to the network layer based on a preset distribution strategy, wherein the data preprocessing comprises data cleaning, data format conversion and traffic shaping processing.

5. The origin processing device high concurrency processing system of claim 1, wherein, The network layer is further configured to adjust parameters of the Kafka cluster according to a preset parameter tuning instruction, and obtain the Kafka cluster after parameter adjustment.

6. The origin processing device high concurrency processing system of claim 1, wherein, The network layer performs metadata management on the Kafka cluster by Zookeeper.

7. The geo-processing device high-concurrency processing system according to claim 1, wherein, The intermediate layer is specifically configured to: process the Kafka information subscribed by the Kafka consumer group according to a database connection pool and a preset thread pool; and write data obtained after processing to the database in parallel, wherein the database connection pool is composed of a plurality of database connections that are continuously maintained; and the preset thread pool is composed of a plurality of threads.

8. A method for high concurrency processing of origin treatment equipment based on the high concurrency processing system of origin treatment equipment according to any one of claims 1 to 7, characterized in that, The method comprises: acquiring device data uploaded concurrently by a device cluster based on a target data center, wherein the target data center is determined according to real-time load state information of servers in the data center; and the device cluster is a 100,000-level device cluster; pushing the received device data to a corresponding topic based on a message producer, and storing Kafka information in a partition unit by a proxy server cluster; processing Kafka information subscribed by a Kafka consumer group, and writing data obtained after processing to a database in parallel.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the high-concurrent processing method for origin processing equipment according to claim 8 when executing the computer program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the high-concurrent processing method for origin processing equipment according to claim 8 when executed by the processor.

Images