Method and system for smoothing data transmission among multiple services driven by non-blocking queue

Abstract

The invention provides a non-blocking queue-driven multi-service data transmission smoothing method and system, and relates to the technical field of Internet of Things, the method comprises the following steps: S1, acquiring data to be transmitted, and smoothly converging the data to be transmitted into a uniform interface service through an external system for data interaction processing; s2, transmitting the data after uniform interface service interaction processing to a non-blocking queue through a gRPC protocol for data analysis; s3, performing data processing on the data after data analysis through an intelligent routing service, and smoothly distributing the processed data to each service system; according to the method, the consumption capability of data is remarkably improved, the time sequence of the data is ensured and the data transmission is smoother on the premise of avoiding data loss, and the overall service capability and reliability of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a method and system for smoothing data transmission between multiple services driven by a non-blocking queue. Background Art

[0002] With the development of IoT technology and the increasing interconnectivity of devices, the demand for data exchange between multiple services has increased dramatically, placing higher demands on data consumption efficiency and transmission and distribution stability. Against this backdrop, traditional data transmission methods, such as polling-based or simple message queue-based communication mechanisms, have gradually exposed their limitations in high-concurrency scenarios. These issues primarily manifest in data loss, processing delays, slow system responses, and high resource utilization, severely impacting the overall performance and user experience of IoT systems. Especially when building large-scale, distributed IoT applications, efficient collaboration between services and smooth data flow are crucial to determining whether a system can respond to and process massive amounts of data in real time. Summary of the Invention

[0003] The purpose of the present invention is to provide a non-blocking queue driven multi-service data transmission smoothing method, which can realize the fast and seamless transmission of data between multiple service components, ensure the efficiency and continuity of data flow, improve the system's response speed and processing capability, and enhance the system's stability.

[0004] The technical solution of the present invention is:

[0005] In a first aspect, the present application provides a method for smoothing data transmission between multiple services driven by a non-blocking queue, which comprises the following steps:

[0006] S1. Obtain the data to be transmitted and smoothly import it into the unified interface service through the external system for data interaction processing;

[0007] S2. Transmit the data processed by the unified interface service interaction to a non-blocking queue through the gRPC protocol for data analysis;

[0008] S3: The analyzed data is processed through intelligent routing services, and the processed data is smoothly distributed to various business systems.

[0009] Furthermore, the data to be transmitted includes hydrological data and wind data uploaded by the gateway and independent nodes in the Internet of Things environment.

[0010] Furthermore, the above-mentioned data interaction processing includes adding metadata tags and data priority identifiers after performing intelligent load balancing processing on the data to be transmitted.

[0011] Furthermore, in step S2, the above data analysis includes: atomic operation, batch processing and cache line optimization.

[0012] Furthermore, step S3 includes:

[0013] S31, distinguishing the types and priorities of the data after data analysis by using the pre-processed embedded metadata;

[0014] S32, combining the differentiated metadata with the requested URL path and query parameters to perform precise matching to form a complete routing context;

[0015] S33. Smoothly distribute the complete routing context to each business system to complete data transmission.

[0016] Furthermore, the metadata includes data source, tracking ID under multiple services, absolute timestamp and target service.

[0017] In a second aspect, the present application provides a non-blocking queue driven multi-service data transmission smoothing system, comprising:

[0018] The data preprocessing module is used to obtain the data to be transmitted and smoothly import the data to be transmitted into the unified interface service through the external system for data interaction processing;

[0019] The data analysis module is used to transmit the data processed by the unified interface service interaction to a non-blocking queue through the gRPC protocol for data analysis;

[0020] The data smoothing distribution module is used to process the data after data analysis through the intelligent routing service, and smoothly distribute the processed data to various business systems.

[0021] In a third aspect, the present application provides an electronic device, comprising:

[0022] a memory for storing one or more programs;

[0023] processor;

[0024] When the one or more programs are executed by the processor, a non-blocking queue driven multi-service data transmission smoothing method as described in any one of the first aspects is implemented.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a non-blocking queue-driven multi-service inter-data transmission smoothing method as described in any one of the first aspects above.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects:

[0027] The present invention provides a non-blocking queue-driven multi-service data transmission smoothing method and system, which sends different data receiving services into a high-performance non-blocking queue for temporary storage through a unified interface service, and then distributes the data extracted from the queue efficiently and correctly through an intelligent routing service, thereby realizing point-to-point data push from the unified interface service to the target business service, significantly improving the data consumption capacity, ensuring the timing of data and making data transmission smoother while avoiding data loss, thereby enhancing the overall business capabilities and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a flow chart of a method for smoothing data transmission between multiple services driven by a non-blocking queue according to the present invention;

[0030] Figure 2 Provides smooth import of data into a unified interface service through external systems;

[0031] Figure 3 A schematic structural diagram of data entering the non-blocking queue service (NBQS);

[0032] Figure 4 The figure is a schematic structural block diagram of an electronic device according to an embodiment of the present invention.

[0033] Icon: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0034] Explanation of terms:

[0035] URL: (Uniform Resource Locator) Uniform Resource Locator, that is, web page address, is the standard resource address on the Internet;

[0036] gRPC: A high-performance, open-source, and general-purpose remote procedure call (RPC) framework.

[0037] Redis (Remote Dictionary Server) is an open-source, network-aware, in-memory and persistent log-based key-value database written in ANSI C. It provides APIs in multiple languages.

[0038] SDK: (Software Development Kit), software development kit;

[0039] CAS: (Compare-And-Swap), a hardware-supported atomic operation for multi-threaded synchronization;

[0040] SegQueue, a lock-free and thread-safe queue, is suitable for high-concurrency scenarios. To ensure that operations on the queue are thread-safe;

[0041] Mutex: (Mutual Exclusion, a synchronization primitive used in multi-threaded or multi-process programming, which ensures that only one thread or process can access shared resources at the same time, preventing multiple threads or processes from reading and writing shared resources at the same time and causing data inconsistency or other concurrency problems.

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0045] It should be noted that, in this document, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprises..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the elements.

[0046] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features thereof may be combined with each other.

[0047] Example 1

[0048] See also Figure 1 , Figure 1 The figure shows a step diagram of a non-blocking queue driven multi-service data transmission smoothing method provided by an embodiment of the present application.

[0049] In a first aspect, the present application provides a method for smoothing data transmission between multiple services driven by a non-blocking queue, which comprises the following steps:

[0050] S1. Obtain the data to be transmitted and smoothly import it into the unified interface service through the external system for data interaction processing;

[0051] S2. Transmit the data processed by the unified interface service interaction to a non-blocking queue through the gRPC protocol for data analysis;

[0052] S3: The analyzed data is processed through intelligent routing services, and the processed data is smoothly distributed to various business systems.

[0053] As a preferred embodiment, the data to be transmitted includes hydrological data and wind data uploaded by the gateway and independent nodes in the Internet of Things environment.

[0054] It should be noted that in the Internet of Things environment, multiple gateways and independent nodes upload data at high frequency, such as urban wind speed, wind direction, hydrological temperature and humidity sensor data. These data not only need to maintain the accuracy and continuity of the time series, but also need to be processed efficiently without being lost.

[0055] As a preferred implementation, the data interaction processing includes adding metadata tags and data priority identifiers after performing intelligent load balancing processing on the data to be transmitted.

[0056] It should be noted that in order to meet the above-mentioned input data requirements, the service must meet sufficient performance requirements to process massive amounts of data. Therefore, the UAS designed in this embodiment must have asynchronous characteristics, and have an intelligent load balancing mechanism and flow control function, and adopt a unified RPC protocol interface for data interaction, with the purpose of receiving massive amounts of real-time data pushed from multiple data receiving services. The service is based on a reliable C / S architecture and uses gRPC to achieve efficient communication interaction. At the same time, it combines the advanced asynchronous characteristics of gRPC and Rust language to greatly improve the utilization of server resources. Among them, the C end is a set of SDKs that fully support Java and Rust languages. It is mainly responsible for asking the S end for load node information and reliably pushing data according to the specified destination. The S end is a highly scalable service group, including primary and secondary gateways and other nodes, which realizes a lightweight distributed extension architecture by writing registration data to Redis and regularly updating heartbeat information. The gateway module in the S end uses weighted polling combined with the least number of connections load balancing algorithm to select the appropriate node, and ensures the health of the node through real-time monitoring and failover mechanisms. It also counts the traffic of each node in real time, sets the upper limit of the traffic, and uses a sliding window current limiting strategy to control the flow and ensure smooth data transmission. When the data passes through the S end, metadata tags are added to each piece of data passing through, including the data source, tracking ID under multiple services, absolute timestamp, target service and other metadata to ensure the correctness and timing of data distribution. According to business rules, priority identification (such as executor operation data) also needs to be added to the data to enable queue selection and subsequent more efficient management and distribution before entering the queue, such as Figure 2 This shows how data is smoothly imported into the unified interface service from external systems.

[0057] As a preferred implementation, in step S2, data analysis includes: atomic operations, batch processing, and cache line optimization.

[0058] It should be noted that the non-blocking queue adopts atomic operations to ensure that thread-safe data sharing can be achieved in a high-concurrency environment. When executing the queue entry or queue exit operation, it will not cause the thread to wait. Specifically, the present embodiment adopts an operation based on Compare-And-Swap (CAS), which is a hardware-supported atomic operation for realizing multi-threaded synchronization, allowing shared data to be updated without using locks. When executing the queue entry or queue exit operation, if the CAS operation fails (i.e., the current value does not match the expected value), the thread will not be blocked, but will be continuously retried by a loop attempt until it succeeds. Although this method increases the burden on the CPU, it avoids the overhead caused by thread switching and improves the response speed and throughput of the system. In addition, the classic non-blocking queue algorithm (Michael-Scott non-blocking queue) is also adopted to manage the queue by maintaining the head node and tail node, and to update these nodes by CAS operations. To further optimize performance, batch processing technology was introduced to reduce the frequency of CAS operations. Open addressing was used in the linked list implementation for compact storage, avoiding the memory fragmentation problem caused by scattered nodes in traditional linked lists. At the same time, Rust-specific annotations were used in the metadata structure to align the data structure to the cache line boundary (usually 64 bytes), ensuring that each instance occupies a cache line. Using the above two methods to optimize the data structure layout reduces cache line conflicts, improves the cache hit rate, and ensures the immediacy and efficiency of data distribution.

[0059] like Figure 3 The following diagram shows a schematic block diagram of the structure after data enters the Non-Blocking Queue Service (NBQS). This system's non-blocking queue is primarily implemented using the features of the Rust language. It also uses a lightweight distributed architecture based on Redis and utilizes SegQueue from the Crossbeam library, a lock-free and thread-safe queue suitable for high-concurrency scenarios. To ensure thread-safe operations on the queue, Mutex is used for mutual exclusion protection, and Arc is used to share ownership across multiple threads. Further optimizations include reducing the scope of locks, handling empty queues, adding error handling, and performance optimization.

[0060] As a preferred embodiment, step S3 includes:

[0061] S31, distinguishing the types and priorities of the data after data analysis by using the pre-processed embedded metadata;

[0062] S32, combining the differentiated metadata with the requested URL path and query parameters to perform precise matching to form a complete routing context;

[0063] S33. Smoothly distribute the complete routing context to each business system to complete data transmission.

[0064] As a preferred implementation, the metadata includes data source, tracking ID under multiple services, absolute timestamp and target service.

[0065] It should be noted that in Example 1 of the present invention, for example, the URL content of an RPC request is " / orders / 123?status=shipped" and carries the metadata "{X-Source:DataHub_182,X-Trace-ID:f47ac10b-58cc-4372-a567-0e02b2c3d479,X-Timestamp:2024-10-01T12:00:00Z,X-Target-Service:ParseGateway}". The routing rule engine will accurately route the request to the data parsing service based on the target service field (ParseGateway), and further match it to a specific business logic branch based on the URL path ( / orders / 171) and query parameters (status=shipped). Through the efficient URL parsing library (url) in the Rust language, the system can quickly parse URLs and ensure efficient distribution of requests, ensuring that data can be quickly and accurately forwarded to the corresponding business processing unit. In addition, IRS also has an asynchronous mechanism for unified interface services and built-in multiple load balancing strategies, such as polling, weighted polling, minimum number of connections and tasks, to cope with the complexity and uncertainty of the network environment.

[0066] Example 2

[0067] In a second aspect, the present application provides a non-blocking queue driven multi-service data transmission smoothing system, comprising:

[0068] The data preprocessing module is used to obtain the data to be transmitted and smoothly import the data to be transmitted into the unified interface service through the external system for data interaction processing;

[0069] The data analysis module is used to transmit the data processed by the unified interface service interaction to a non-blocking queue through the gRPC protocol for data analysis;

[0070] The data smoothing distribution module is used to process the data after data analysis through the intelligent routing service, and smoothly distribute the processed data to various business systems.

[0071] In this second embodiment, a non-blocking queue-driven, multi-service data transmission smoothing system of the present invention is applied to a smart irrigation scenario. In this scenario, farmland is deployed with a variety of IoT devices, water flow and energy, soil moisture sensors, irrigation actuators, etc. These devices need to efficiently exchange data with a central management system and data analysis platform to achieve precision agricultural management. The specific implementation steps are as follows:

[0072] 1. System Deployment and Configuration: Deploying a non-blocking queue-driven data transmission system in a Docker containerized system requires deploying the following components: Unified Interface Service (UAS), Non-Blocking Queue Service (NBQS), Intelligent Routing Service (IRS), and a Redis cache system. These services can be easily started using a pre-prepared docker-compose file, which should include the container definitions and network configuration for each service. Pay attention to the gateway node configuration on the S-side of the UAS service and the master-slave mechanism. In environments with smaller data volumes, the master-slave configuration can be simplified to a single instance. The Redis cache system serves as the foundation for multi-service registration and keepalive functions. Service weights (service priority, reference server resources, peak data, etc.) and data flow are then configured in the system. In actual operation, queue capacity and processing power are adjusted based on data traffic volume.

[0073] 2. Device Data Collection, Upload, and Distribution: In smart irrigation systems, data from water volume, flow, and soil moisture sensors must be regularly uploaded. While the frequency of data from a single node is low, the large number of nodes ensures a large and stable data volume. This data is transmitted to multiple data receiving services via the MQTT and bare socket protocols. The multi-service model is adopted primarily to address limitations on server connections and network bandwidth, ensuring efficient and stable data transmission.

[0074] The Data Receiving Service pushes data to the Non-Blocking Queuing Service (NBQS) for analysis via the Unified Interface Service (UAS) within the system. The Intelligent Routing Service (IRS) then rapidly consumes this data, distinguishing between types and priorities based on metadata embedded during pre-processing. The IRS leverages this metadata (data source, tracking ID, absolute timestamp, and target service) for precise matching, along with the distribution rules configured during pre-processing, to accurately route data to specific business modules, such as data analysis, alarm systems, and control instruction generation.

[0075] 3. Decision-making and execution: The central control system generates precise irrigation instructions based on data analysis results and quickly pushes them to the intelligent irrigation system through a non-blocking queue (to execute the instructions), achieving on-demand watering and saving water resources.

[0076] 4. Monitoring and troubleshooting: The monitoring system continuously tracks the status of data flow. Once a transmission delay or device offline is detected, an alarm is immediately triggered and a backup plan is initiated, such as switching to a backup data channel or rescheduling tasks to ensure that agricultural production activities are not interrupted.

[0077] As can be seen from the above embodiment 2, the non-blocking queue-driven multi-service data transmission smoothing system provided by the present invention has the following advantages:

[0078] (1) Through non-blocking queues and intelligent scheduling mechanisms, data transmission delays are significantly reduced, the real-time nature of inter-service collaboration is improved, and the system can respond to external requests and internal events more quickly;

[0079] (2) Through refined resource management and adaptive optimization strategies, unnecessary resource consumption can be effectively reduced, the overall system processing capacity and efficiency can be improved, and operation and maintenance costs can be reduced;

[0080] (3) The smoothing of data transmission directly improves the service experience of end users, reduces waiting time, and improves service quality, especially in application scenarios such as smart agriculture and the Internet of Things that require immediate feedback;

[0081] (4) The design considers the support of data encryption and security protocols to ensure the security of data transmission, protect user data privacy, and enhance user trust.

[0082] Example 3

[0083] See also Figure 4 , Figure 4 This is a schematic structural block diagram of an electronic device provided in Example 3 of the present application.

[0084] An electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0085] Among them, the memory 101 can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0086] The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] It should be understood that the structure shown in the figure is merely illustrative, and a method for smoothing data transmission between multiple services driven by a non-blocking queue may include more or fewer components than shown in the figure, or have a different configuration than shown in the figure. Each component shown in the figure may be implemented using hardware, software, or a combination thereof.

[0088] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The embodiments described above are merely illustrative. For example, the flowcharts or block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the methods and computer program products according to multiple embodiments of the application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0089] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0090] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0091] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0092] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A non-blocking queue driven multi-service data transmission smoothing method, characterized in that: The following steps are involved: S1. Obtain the data to be transmitted and smoothly import it into the unified interface service through the external system for data interaction processing; S2. Transmit the data processed by the unified interface service interaction to a non-blocking queue through the gRPC protocol for data analysis; S3: The analyzed data is processed through intelligent routing services, and the processed data is smoothly distributed to various business systems.

2. The method for smoothing data transmission between multiple services driven by a non-blocking queue as claimed in claim 1, characterized in that: The data to be transmitted includes hydrological data and wind data uploaded by the gateway and independent nodes in the Internet of Things environment.

3. The method for smoothing data transmission between multiple services driven by a non-blocking queue as claimed in claim 1, characterized in that: The data interaction processing includes adding metadata tags and data priority identifiers after performing intelligent load balancing processing on the data to be transmitted.

4. The method for smoothing data transmission between multiple services driven by a non-blocking queue as claimed in claim 1, characterized in that: In step S2, the data analysis includes: atomic operations, batch processing and cache line optimization.

5. The method for smoothing data transmission between multiple services driven by a non-blocking queue as claimed in claim 1, characterized in that: Step S3 includes: S31, distinguishing the types and priorities of the data after data analysis by using the pre-processed embedded metadata; S32, combining the differentiated metadata with the requested URL path and query parameters to perform precise matching to form a complete routing context; S33. Smoothly distribute the complete routing context to each business system to complete data transmission.

6. The method for smoothing data transmission between multiple services driven by a non-blocking queue as claimed in claim 5, characterized in that: The metadata includes data source, tracking ID under multiple services, absolute timestamp and target service.

7. A non-blocking queue driven multi-service data transmission smoothing system, characterized in that: include: The data preprocessing module is used to obtain the data to be transmitted and smoothly import the data to be transmitted into the unified interface service through the external system for data interaction processing; The data analysis module is used to transmit the data processed by the unified interface service interaction to a non-blocking queue through the gRPC protocol for data analysis; The data smoothing distribution module is used to process the data after data analysis through the intelligent routing service, and smoothly distribute the processed data to various business systems.

8. An electronic device, characterized in that: include: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, a non-blocking queue-driven multi-service data transmission smoothing method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements a non-blocking queue driven multi-service data transmission smoothing method according to any one of claims 1 to 6.

Images