Efficient data transmission method and system based on message queue

Abstract

The invention discloses an efficient data transmission method based on a message queue, which comprises the following steps of: receiving and storing data by adopting a distributed message queue to realize asynchronous transmission; the data transmission rate is dynamically controlled through an intelligent flow scheduling and load balancing mechanism; performing transmission scheduling according to the data priority; an asynchronous transmission and cache mechanism is adopted to deal with a high-load scene; monitoring a system state in real time and dynamically adjusting a transmission strategy; and the data transmission reliability is ensured through an acknowledgement and retransmission mechanism. By adopting a distributed message queue architecture and an intelligent traffic scheduling method, the bottleneck problem of data transmission is solved. According to the invention, under the environment of high concurrency and peak load, the stability and reliability of data transmission can be ensured, the data transmission efficiency is improved, data loss and transmission delay are avoided, the system adapts to complex and changeable application scenes, and the requirements of real-time monitoring and a large-scale data transmission system are met.

Description

TECHNICAL FIELD

[0001] The present application relates to an efficient data transmission method and system, in particular to an efficient data transmission method and system based on message queue. BACKGROUND

[0002] At present, the transmission and processing of massive real-time data are widely used in many industries, especially in large-scale data acquisition, real-time monitoring and high-concurrency systems. Nowadays, traditional data acquisition and transmission methods are mostly used, but when facing network congestion, large-scale data volume and high-concurrency environment, there are many problems. Traditional message queue systems such as Kafka and RabbitMQ, although they improve the concurrency processing capability to a certain extent, but due to the lack of effective flow control and data priority strategy, they cannot provide stable and reliable solutions in high-concurrency data transmission scenarios. Common problems include network congestion and data loss, especially in high-load network environment, data loss, delay and transmission failure frequently occur; the bottleneck of synchronous transmission causes blocking in the data transmission process, increases the system response time; and the transmission efficiency is insufficient, which cannot fully utilize the bandwidth resources, affecting the smooth flow of data stream.

[0003] The invention patent with publication number CN202411475362.6 and the invention name of "Efficient data transmission method and transmission system" discloses an efficient data transmission method and system. Its characteristics are that the data is encrypted, and after obtaining the encrypted data, it is judged whether the data is normal, and then the encrypted data is distributed to the corresponding server. After the data is distributed to the server, one of the servers transfers the stored data, thereby realizing the encryption of the data, avoiding data leakage or theft, and improving the data transmission efficiency. This method optimizes the transmission efficiency by grouping according to the network port transmission speed of the server. However, although this technology improves the data encryption and transmission efficiency, it still has certain defects. In the process of data transmission, due to the performance difference of different servers, it may affect the data transmission efficiency, especially the transmission speed difference between servers will cause low efficiency of data storage and transfer. In addition, with the extension of data storage time, the cold data problem may cause data damage or loss. Therefore, this technology still has room for improvement in handling large-scale data storage and dynamic data transfer, especially in high-load and high-concurrency environments, how to further optimize the data transmission strategy to improve the overall efficiency and stability of the system is the direction of future improvement. SUMMARY

[0004] Invention purpose: The purpose of the present application is to provide a high-efficiency data transmission method and system based on message queue, which ensures the stability and reliability of data transmission, improves the efficiency of data transmission, avoids data loss and transmission delay, adapts to complex and variable application scenarios, and meets the needs of real-time monitoring and large-scale data transmission systems.

[0005] Technical scheme: The high-efficiency data transmission method based on message queue comprises:

[0006] (1) Adopting distributed message queue to receive and store data, realizing asynchronous transmission;

[0007] (2) Dynamically controlling data transmission rate through intelligent flow scheduling and load balancing mechanism;

[0008] (3) Transmission scheduling according to data priority;

[0009] (4) Using asynchronous transmission and cache mechanism to deal with high-load scenarios;

[0010] (5) Real-time monitoring of system state and dynamic adjustment of transmission strategy;

[0011] (6) Guaranteeing data transmission reliability through acknowledgement and retransmission mechanism.

[0012] Further, the message queue receives and stores data, the capacity of the message queue is C, the length of the message queue is L, and the relationship formula between the queue length and the load is:

[0013]

[0014] Further, the intelligent flow scheduling adopts token bucket algorithm for flow control, and controls the data transmission rate by adjusting the token generation rate and the maximum capacity of the token bucket.

[0015] Further, the data priority is a numerical label, and the transmission is carried out from high-priority data to low-priority data in turn, ensuring that critical data is transmitted first in high-load scenarios.

[0016] Further, the cache mechanism includes setting a cache capacity threshold, and automatically pushing data to the message queue when the cache usage rate exceeds the threshold.

[0017] Further, the monitoring system adjustment formula is:

[0018]

[0019] Wherein, CurrentLoad is the current load, and MaximumLoadCapacity is the maximum load capacity.

[0020] Further, the acknowledgement and retransmission mechanism includes setting the acknowledgement time of data as T confirm When the receiving end successfully receives the data, an acknowledgement message is sent, and if there is no acknowledgement within a specified time, the system retries.

[0021] The high-efficiency data transmission system based on a message queue includes:

[0022] The distributed message queue module is used for receiving and storing data and realizing asynchronous transmission.

[0023] The flow scheduling module is used for dynamically controlling the data transmission rate through intelligent flow scheduling and load balancing mechanisms.

[0024] The priority scheduling module is used for transmission scheduling according to data priorities.

[0025] The cache management module is used for coping with high-load scenarios through asynchronous transmission and cache mechanisms.

[0026] The real-time monitoring module is used for real-time monitoring of system states and dynamic adjustment of transmission strategies.

[0027] The retransmission mechanism module is used for guaranteeing data transmission reliability through an acknowledgement and retransmission mechanism.

[0028] A computer device includes one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs are executed by the processors to implement the steps of the high-efficiency data transmission method based on a message queue.

[0029] A computer-readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the high-efficiency data transmission method based on a message queue.

[0030] Advantages: Compared with the prior art, the present application has the following significant advantages:

[0031] (1) By adopting a distributed message queue architecture, single-point failure problems are avoided, and data can still be stably transmitted in a high-concurrency environment; the use of message queue persistent storage and acknowledgement mechanisms greatly improves the reliability of data transmission and reduces data loss and transmission errors.

[0032] (2) Through intelligent flow scheduling and load balancing mechanisms, network congestion during high concurrency is effectively avoided; through token bucket algorithm control flow, the data transmission rate can be dynamically adjusted according to real-time load, ensuring that network bandwidth is fully and reasonably utilized, and improving the overall system throughput.

[0033] (3) Adopting asynchronous data transmission and cache mechanism, non-blocking operation is carried out at the data acquisition end, effectively reducing the response delay of the system, the cache mechanism temporarily stores data at high load, avoiding the transmission bottleneck caused by the high instantaneous traffic peak, and enhancing the processing capacity of the system under high load.

[0034] (4) By setting data priority, it is ensured that critical business data is transmitted preferentially when the network load is high, and the key data in real-time monitoring and business processing is not delayed or lost, ensuring the real-time performance and business continuity of the system.

[0035] (5) Real-time health monitoring and dynamic adjustment mechanism can dynamically adjust the data transmission strategy according to the current network condition and queue load; when the network load is too high, the system automatically starts flow control or adjusts the priority to ensure smooth data transmission, improving the adaptive ability and robustness of the system.

[0036] (6) Data confirmation and retransmission mechanism ensures that data is marked as "transmitted" only after successful transmission, avoiding the problem of data loss caused by network jitter. Retry mechanism can resend data when transmission fails, further reducing the risk of packet loss and improving the integrity of data transmission. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The method flowchart of the application is described.

[0038] Figure 2 The data priority transmission and scheduling flowchart of the application is described. DETAILED DESCRIPTION

[0039] The technical solutions of the application will be further described below with reference to the accompanying drawings.

[0040] As shown in Figure 1 , the application discloses an efficient data transmission method based on message queue, which includes:

[0041] (1) Adopting distributed message queue to receive and store data, realizing asynchronous transmission;

[0042] Specifically, the data acquisition end pushes data to the message queue, and each message queue node is responsible for receiving and storing data to ensure that data is not lost. The relationship between queue length and load is: assuming that the capacity of the message queue is C, the length of the current message queue is L, when L>C, queue blocking or data loss will occur. Therefore, the capacity and load of the queue are controlled by the flow scheduling mechanism. The relationship formula between queue length and load is:

[0043]

[0044] If LF > 1, the queue is overloaded, and the flow control mechanism needs to be started.

[0045] (2) Through intelligent flow scheduling and load balancing mechanism to dynamically control the data transmission rate;

[0046] Introduce intelligent flow scheduling and load balancing mechanism, according to the real-time load, adjust the data transmission rate, and realize priority scheduling. Token bucket algorithm is used to control the flow, and the flow control formula of token bucket model is as follows:

[0047]

[0048] When the queue needs to transmit data, a token is taken out from the bucket. If the bucket is empty, the data cannot continue to transmit until the token is generated. By adjusting the token generation rate r, the data transmission rate is dynamically controlled.

[0049] The basic parameters of token bucket model are: token generation rate r: represents the number of tokens generated by the system per second; The maximum capacity of the bucket B: the maximum number of tokens that can be contained in the token bucket; Time interval t: refers to the time unit of token accumulation; Current token number T(t): at any time t, the number of tokens in the bucket; Data sending demand D: refers to the amount of data that the current queue hopes to send;

[0050] Among them, the system automatically adds tokens to the token bucket according to the rate r every unit time t; The token bucket can only store B tokens at most, and the excess part will be discarded to prevent backlog; When data needs to be sent, the system extracts tokens equivalent to the data amount from the bucket, that is, D≤T(t) is allowed to send; If the number of tokens is insufficient, that is, D>T(t), the data is temporarily suspended until enough tokens are generated.

[0051] The number of available tokens in the token bucket is:

[0052] T(t)=min(B,T(t0)+r·(t-t0))

[0053] Where T(t0) is the number of tokens at the last statistical time; r·(t-t0) is the number of newly added tokens in this time period;

[0054] The formula of token bucket algorithm is:

[0055] Available tokens at time t=min(maxtokens,tokens generated+r·t).

[0056] Where: maxtokens is the maximum capacity of the token bucket; r is the token generation rate; t is the time unit; by adjusting r and maxtokens, dynamically control data transmission according to the current load situation.

[0057] The specific process of dynamically controlling data transmission is as follows:

[0058] (21) Real-time acquisition of system load state, periodic sampling and statistical analysis of the following key indicators through real-time monitoring:

[0059] Including current message queue length L; system load rate λ, such as CPU occupancy, network bandwidth usage; cache usage rate γ; historical packet loss rate and delay fluctuation;

[0060] (22) Calculate the transmission scheduling factor

[0061] According to the above indicators, the current "load factor" or "transmission scheduling factor" α is calculated, and the common formula is as follows:

[0062]

[0063] Where ωi is an empirically set weight parameter; α ∈ [0, 1], the larger the value, the higher the load, the more it needs to limit speed; When α exceeds the set threshold, the system determines that it is currently in a high load or congestion risk state, triggering the speed limiting mechanism.

[0064] (23) Dynamically adjust the token bucket parameters

[0065] According to the scheduling factor α, the system automatically adjusts the following parameters of the token bucket: token generation rate r: the key parameter of adjusting data passing rate; token bucket maximum capacity B: determines the accommodation capacity of instantaneous burst data;

[0066] Priority quota strategy: different levels of data are allocated different rate shares.

[0067] The typical control strategy is as follows:

[0068] r new =r max ·(1-α)

[0069] When the system is idle, α → 0, the transmission rate is the highest; when the system is overloaded, α → 1, it enters the forced speed reduction or suspension state.

[0070] (24) Joint control of data priority scheduling

[0071] At the same time of dynamic speed regulation, the priority scheduling module organizes data transmission according to the following rules:

[0072] High priority data is not affected by throttling and can use excess tokens; low priority data is affected by throttling and is queued; when the system is idle, high priority data is emptied first, and then ordinary data is processed.

[0073] (25) Feedback loop and adaptive adjustment

[0074] Establish a "feedback loop mechanism" to evaluate whether the queue length and packet loss rate are effectively reduced at the end of each control cycle; if an abnormality is detected, immediately force flow limiting or trigger an alarm.

[0075] (3) Transmission scheduling according to data priority;

[0076] During transmission, different priorities are set to ensure that critical data is transmitted first. Set the priority P of each data and sort it by priority.

[0077] Set the priority P of the data i , where a higher P value indicates a higher priority. The order of data transmission follows the following sorting rules: P1>P2>…>P n ; the transmission order is according to the size of P, from high priority data to low priority data, ensuring that critical data is transmitted first in high load.

[0078] (4) Use asynchronous transmission and cache mechanism to deal with high load scenarios;

[0079] Use asynchronous data transmission and cache mechanism. The data acquisition end pushes data to the message queue without waiting for a return, thereby reducing system response time. Assuming the cache capacity of the system is C cache , the current data volume in the cache is L cache , when L cache > C cache , the cache is full, and the system needs to transfer data to the queue for asynchronous processing. The management of the cache can be measured by the following formula:

[0080]

[0081] If the Cache Utilization reaches the threshold, the system automatically pushes the data in the cache to the message queue asynchronously, ensuring that data loss does not occur due to cache overload.

[0082] The definition of the cache threshold is that the total capacity of the system cache is C buf , and the current cache usage is L buf , then the cache utilization is defined as:

[0083]

[0084] (5) Real-time monitoring of system status and dynamic adjustment of transmission strategy;

[0085] By real-time monitoring of the health status of the system, the data transmission strategy is dynamically adjusted. When the system load is too high, automatically adjust the data flow, priority and cache strategy to ensure stable transmission of data. System health status monitoring, system load monitoring and adjustment formula:

[0086]

[0087] Wherein, CurrentLoad is the current load, MaximumLoadCapacity is the maximum load capacity.

[0088] If System Load Factor exceeds the preset threshold, start flow control, priority adjustment or cache expansion mechanism to ensure system stability. The threshold is defined as TSLF, the value range is 0~1

[0089] When SystemLoadFactor≤TSLF, the system is considered to be load controllable and does not need to be intervened.

[0090] When SystemLoadFactor>TSLF, the system enters a high-pressure state and automatically starts the adjustment measures.

[0091] (6) Through the confirmation and retransmission mechanism to ensure the reliability of data transmission.

[0092] Through the persistent storage of message queue and the confirmation mechanism, the reliability of data is ensured. In the process of data transmission, the message queue confirmation mechanism ensures that the data will be marked as "transmitted" only after successful transmission. The confirmation time of data is set as T confirm When the receiving end successfully receives the data, send an acknowledgement message. If there is no confirmation within the specified time, the system will retry.

[0093] Wherein, T confirm is the threshold for determining whether the message is considered "successfully transmitted" or "transmission failed and needs to be retried".

[0094] To realize the message confirmation and retransmission mechanism, the system introduces the following timeout control logic:

[0095]

[0096] Wherein, t send is the original sending time of the data packet; t current is the current system clock; if the confirmation response is not received within T comfirm time, it is considered as confirmation timeout, and the system immediately re-sends the data.

[0097] Wherein, T confirm The setting is generally based on network delay and system response time comprehensive decision:

[0098] T confirm = mu ack + 2 sigma ack

[0099] Wherein, mu ack is the historical average confirmation delay; sigma ack is the standard deviation, used to widen the tolerance bandwidth.

[0100] The application also discloses a kind of efficient data transmission system based on message queue, comprising:

[0101] Distributed message queue module: for receiving and storing data, realizing asynchronous transmission;

[0102] Traffic scheduling module: for dynamically controlling data transmission rate by intelligent traffic scheduling and load balancing mechanism;

[0103] Priority scheduling module: for transmission scheduling according to data priority;

[0104] Cache management module: for using asynchronous transmission and cache mechanism to deal with high-load scenario;

[0105] Real-time monitoring module: for real-time monitoring system state and dynamically adjusting transmission strategy;

[0106] Retransmission mechanism module: for guaranteeing data transmission reliability by acknowledgement and retransmission mechanism.

[0107] The application is based on the functional design of efficient data transmission mechanism and related data acquisition and transmission process based on distributed message queue and the functional interaction between each module. It can effectively solve the problems of network congestion, data loss and transmission delay in mass real-time data transmission. It adopts distributed message queue architecture, intelligent traffic scheduling, priority transmission and asynchronous cache mechanism and other technologies. Through intelligent traffic scheduling and priority strategy, it ensures the priority transmission of critical data, and improves system response speed and processing capacity through asynchronous transmission and cache mechanism. At the same time, the system can dynamically adjust data transmission strategy according to real-time monitoring data, to ensure stable operation under high-load and high-concurrency environment. The method is suitable for various application scenarios and greatly improves the efficiency, stability and reliability of data transmission, and is widely used in large-scale data transmission and real-time monitoring system.

Claims

1. An efficient data transfer method based on message queue, characterized in that, Comprise: (1) Adopting distributed message queue to receive and store data, realizing asynchronous transmission; (2) Dynamically controlling data transmission rate through intelligent flow scheduling and load balancing mechanism; (3) Transmitting according to data priority; (4) Adopting asynchronous transmission and cache mechanism to cope with high load scenario; (5) Real-time monitoring system state and dynamically adjusting transmission strategy; (6) Guaranteeing data transmission reliability through acknowledgement and retransmission mechanism.

2. The efficient data transmission method based on message queue according to claim 1, characterized in that, The message queue receives and stores data, the capacity of the message queue is C, the length of the message queue is L, the formula of the relationship between the queue length and the load is:

3. The efficient data transmission method based on message queue according to claim 1, characterized in that, The intelligent flow scheduling adopts token bucket algorithm to control flow, and controls data transmission rate by adjusting token generation rate and maximum capacity of token bucket.

4. The efficient data transmission method based on message queue according to claim 1, characterized in that, The data priority is a numerical label, from high priority data to low priority data, ensuring that critical data is transmitted first in high load.

5. The efficient data transmission method based on message queue according to claim 1, characterized in that, The cache mechanism includes setting cache capacity threshold, and automatically pushing data to message queue when cache usage exceeds threshold.

6. The efficient data transmission method based on message queue according to claim 1, characterized in that, The monitoring system adjustment formula is: Wherein, CurrentLoad is the current load, MaximumLoadCapacity is the maximum load capacity.

7. The efficient data transmission method based on message queue according to claim 1, characterized in that, The acknowledgement and retransmission mechanism includes setting the acknowledgement time of data as T confirm When the receiving end successfully receives the data, an acknowledgement message is sent, and if there is no acknowledgement within a specified time, the system retries.

8. A high efficient data transmission system based on message queue, characterized in that, Comprise: Distributed message queue module: for receiving and storing data, realizing asynchronous transmission; Flow scheduling module: for dynamically controlling data transmission rate through intelligent flow scheduling and load balancing mechanism; Priority scheduling module: for transmitting according to data priority; Cache management module: for adopting asynchronous transmission and cache mechanism to cope with high load scenario; Real-time monitoring module: for real-time monitoring system state and dynamically adjusting transmission strategy; Re-transmission mechanism module: for guaranteeing data transmission reliability through acknowledgement and retransmission mechanism.

9. A computer device, comprising: One or more processors, memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs are executed by the processor to realize the steps of a kind of efficient data transmission method based on message queue as claimed in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of a kind of efficient data transmission method based on message queue as claimed in any one of claims 1-7.

Images