Big data network communication coordination method
By collecting node hardware performance indicators and request characteristics, and dynamically adjusting request routing and priority, the problem of node overload and resource idleness in high-concurrency scenarios of traditional network communication coordination methods is solved, and efficient and stable request processing is achieved.
Patent Information
- Application Number
- CN202511206386.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-27
- Publication Date
- 2025-12-05
AI Technical Summary
Traditional network communication coordination methods cannot dynamically adjust request allocation in high-concurrency scenarios, leading to node overload or resource idleness. Furthermore, they lack effective request tracking and monitoring mechanisms, which affect system performance and user experience.
By periodically collecting node hardware performance indicators, calculating the overall load value, screening candidate nodes for request distribution, and calculating the distribution ratio using the optimal distribution coefficient formula, combined with the request feature and complexity mapping model, the request priority and queue order are dynamically adjusted to achieve reasonable allocation and processing of requests.
It significantly improves processing efficiency and stability, enabling rapid response and adjustment of request allocation in high-concurrency scenarios, ensuring efficient operation of each node and optimizing user experience.
Smart Images
Figure CN121077972A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of network communication, in particular to a big data network communication coordination method. BACKGROUND
[0002] In today's information age, the rapid development of big data technology has greatly promoted the progress of network communication technology. With the widespread popularity and in-depth application of the Internet, various network services such as cloud computing, the Internet of Things, online social networking, etc. have generated a large amount of data requests, which have characteristics such as diversity, complexity and concurrency, and have put forward very high requirements on the processing capacity and efficiency of network communication systems. Especially in high concurrency scenarios, how to effectively coordinate and manage network communication requests to ensure system stable operation and improve user experience has become a key problem to be solved in the current network communication field.
[0003] However, the traditional network communication coordination method mainly relies on simple load balancing strategy and static priority management mechanism. These methods have obvious limitations when facing large-scale concurrent requests. First, the traditional load balancing strategy is usually based on fixed rules or algorithms, which cannot dynamically adjust request allocation according to real-time load conditions, resulting in overloading of some nodes and idling of other nodes, reducing the overall processing efficiency of the system. Second, the static priority management mechanism cannot accurately reflect the complexity and real-time requirements of requests, which may cause important requests to be delayed or low-priority requests to be waiting for a long time, affecting user experience and service quality. In addition, the traditional method lacks effective request tracking and monitoring mechanism, which makes it difficult to monitor and dynamically adjust the request processing process in real time, further limiting the improvement of system performance, and it is difficult to meet the current demand for efficient, flexible and reliable coordination methods in big data network communication. SUMMARY
[0004] The purpose of the present application is to make up for the shortcomings of the prior art, and to provide a big data network communication coordination method. The present application periodically collects node hardware performance index values, calculates the comprehensive load value of the current node, and compares it with the initial load threshold value. When the comprehensive load value exceeds the initial load threshold value, the load state of the adjacent node is queried, the candidate node lower than the initial load threshold value is screened out, and the shunt ratio is calculated by the optimal shunt coefficient formula to realize the reasonable shunt of the request. This design can dynamically adjust the request allocation according to the real-time load condition, effectively avoid the problem of node overload and resource idling, and significantly improve the processing efficiency and stability. In high concurrency scenarios, it can quickly respond to and adjust the request allocation strategy to ensure that each node can run efficiently within its processing capacity.
[0005] The application provides the following technical scheme to solve the above technical problems: a big data network communication coordination method, and the specific steps of the method are as follows: Model construction: collect historical request data to establish a training sample library, train a mapping relationship model of request features and complexity, and set an initial load threshold and an initial waiting threshold; Request receiving and load monitoring: the node receives concurrent communication requests from different users, generates a unique request ID for each request and creates a request tracking file, and regularly collects node hardware performance index values, calculates the comprehensive load value of the current node, and compares the comprehensive load value of the current node with the initial load threshold to make a judgment. Load shunting and complexity evaluation: when the comprehensive load value of the current node exceeds the initial load threshold, the load state of the adjacent node is queried, the candidate node lower than the initial load threshold is screened out, the optimal shunting coefficient is calculated through the optimal shunting coefficient formula, the shunting ratio is obtained, the request with the target address close to the candidate node is shunted and forwarded, the request retained for processing is analyzed for its feature parameters, and the complexity quantization value is calculated through the mapping relationship model. Priority calculation and dynamic sorting: a priority evaluation mechanism is constructed, the priority score of each request to be processed is calculated through the fair priority index formula, the request processing queue is constructed from high to low according to the score, and a regular reevaluation mechanism is set to dynamically adjust the queue order. Request processing and threshold control: the request is processed according to the queue order, the waiting threshold is dynamically adjusted according to the node load and the request queue length, and the waiting time of each request is monitored.
[0006] Further, in the model construction, historical request data is collected to establish a training sample library, the sample contains request feature parameters and actual processing time, the sample data is trained, a mapping relationship model of request features and complexity is generated, the output of the mapping relationship model is the request complexity value, and the initial load threshold is set according to the node hardware performance and historical load data, wherein the node hardware performance includes CPU utilization, memory occupancy, network interface rate and disk I / O rate, and the initial waiting threshold is determined based on the SLA service level agreement.
[0007] Further, in the request receiving and load monitoring, the node receives concurrent communication requests from different users, each request contains data content, request type, target address and user identification meta-information, a unique request ID is generated for each request, a request tracking file containing receiving time and initial state is created, and the performance index values of CPU utilization, memory occupancy, network interface rate and disk I / O rate of the node are regularly collected, and the comprehensive load value of the current node is calculated through the node load formula.
[0008] Further, in the request receiving and load monitoring, the comprehensive load value of the current node is calculated by a node load formula, which is: wherein, is the comprehensive load value of the current node, is the CPU utilization rate, is the memory occupancy, is the disk I / O rate, is the network interface rate, is the weight coefficient of each index. The comprehensive load value of the current node is compared with the initial load threshold value, if , it is determined that the node is in a high load state, and if , it is determined that the node is in a normal load state.
[0009] Further, in the load shunting and complexity evaluation, the optimal shunting coefficient is calculated by an optimal shunting coefficient formula, which is: wherein, is the optimal shunting coefficient, is the remaining resource proportion of the i-th candidate node, is the network distance to the i-th candidate node, is the comprehensive load value of the i-th candidate node, is the comprehensive load value of the current node, is the average comprehensive load value of the region to which the current node belongs, is the number of candidate nodes, is the index. Further, in the load shunting and complexity evaluation, for the request to be retained, the feature parameters of the data size , the complexity coefficient
[0010] , the data compression ratio , the transmission hop number , the network topology coefficient and the resource requirement level in the request are analyzed, and the extracted feature parameters are input into a mapping relationship model to calculate the complexity quantization value of the request, and the calculation formula is:
[0011] Further, in the priority calculation and dynamic sorting, a priority evaluation mechanism is constructed, based on the factors of request complexity, current waiting time and node load, and the priority score of each request to be processed is calculated through a fair priority index formula, which is: wherein, is the priority score value, is the complexity quantization value, is the initial waiting threshold value, is the current waiting time, is the initial load threshold value, is the current node comprehensive load value, is the weight coefficient.
[0012] Further, in the priority calculation and dynamic sorting, a periodic re-evaluation mechanism is set, in the queue waiting process, according to the change of each request waiting time and the fluctuation of network load, the priority score of all unprocessed requests is recalculated, and the queue order is dynamically adjusted.
[0013] Further, in the request processing and threshold control, the requests are taken out in sequence according to the queue order for processing, and the waiting time of the remaining requests in the queue is automatically updated after the processing of each request is completed, and the waiting threshold value in the fair priority index formula is dynamically adjusted through an adaptive waiting threshold formula according to the node load and the request queue length, which is: wherein, is the dynamically adjusted waiting threshold value, is the initial waiting threshold value, is the current request queue length, is the initial load threshold value, is the current node comprehensive load value. When the node load is high, the waiting threshold value is increased, and when the node load is low, the waiting threshold value is decreased, and at the same time, the waiting time of each request is monitored When the waiting time of a certain request reaches the current dynamically adjusted waiting threshold value , the priority score value is raised to the highest, and the request is processed in priority.
[0014] Compared with the prior art, the big data network communication coordination method has the following beneficial effects: One, the present application can dynamically adjust the request distribution according to the real-time load condition, effectively avoid the problems of node overload and resource idling, thereby significantly improve the processing efficiency and stability, in the high concurrency scene, can quickly response and adjust the request distribution strategy, ensure that each node can run efficiently within its processing capacity.
[0015] Two, the present application can dynamically adjust the request distribution according to the real-time load condition, effectively avoid the problems of node overload and resource idling, thereby significantly improve the processing efficiency and stability, in the high concurrency scene, can quickly response and adjust the request distribution strategy, ensure that each node can run efficiently within its processing capacity.
[0016] Other advantages, objects, and features of the application will be set forth in part by the description which follows, and in part will become apparent to those skilled in the art upon examination of same, or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without any creative effort.
[0018] Figure 1 It is a flow chart of a big data network communication coordination method; Figure 2 It is a framework diagram of a big data network communication coordination method. DETAILED DESCRIPTION
[0019] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined application purpose, the specific embodiments, structures, features and effects according to the present application will be described in detail below combined with the drawings and preferred embodiments.
[0020] Embodiment one: model construction: during the order processing of the e-commerce platform during the big promotion, a large number of order request data during the historical big promotion is collected, covering the order containing the commodity category (such as clothing, home appliances), the number of goods, the payment method (Alipay, WeChat payment, etc.), whether to use the coupon and other request characteristic parameters, and the actual processing time of these orders from submission to completion of inventory deduction, payment verification, logistics single generation, a training sample library is established, a mapping relationship model of order request characteristics and processing complexity is generated through training, and the output of the mapping relationship model is the request complexity value, at the same time, according to the hardware performance of the platform server (including CPU, memory, network interface, disk I / O, etc.), combined with the load peak value during the historical big promotion, the initial load threshold is set, and according to the clauses about order processing delay in the SLA agreement signed with the merchant, the initial waiting threshold is determined.
[0021] Request receiving and load monitoring: the platform server node in a certain area receives concurrent order requests from users in the area, each request contains detailed commodity list, delivery address, user ID, payment status and other meta information, the system generates a unique order number for each order as the request ID, and creates a tracking file to record the receiving time and the initial state as "to be verified inventory", at the same time, the CPU utilization, memory occupancy, network interface rate, disk I / O rate and other performance indicators of the server are collected regularly, and the comprehensive load value of the current node is calculated through the node load formula, and the node load formula is: , wherein, is the comprehensive load value of the current node, is the CPU utilization, is the memory occupancy, is the disk I / O rate, is the network interface rate, is the weight coefficient of each indicator; The comprehensive load value of the current node is compared with the initial load threshold , if , it is determined that the node is in a high load state, if , it is determined that the node is in a normal load state.
[0022] Load shunting and complexity evaluation: when the comprehensive load value of the current node exceeds the initial load threshold, the load states of multiple servers in the same cluster in the surrounding area are queried through the internal network, and the candidate servers with load lower than the threshold are selected, the optimal shunting coefficient is calculated through the optimal shunting coefficient formula to determine the optimal shunting ratio, and the optimal shunting coefficient formula is: , wherein, is the optimal shunting coefficient, is the The proportion of remaining resources for each candidate node. To reach the first Network distance of each candidate node For the first The overall load value of each candidate node. This represents the overall load value of the current node. This represents the average overall load value of the region to which the current node belongs. The number of candidate nodes. For indexing, orders with delivery addresses within the candidate server's coverage area are prioritized for forwarding. For orders that are retained for processing, feature parameters such as data size, complexity coefficient (e.g., overlapping discount rules), data compression ratio, transmission hops, network topology coefficient, and resource requirement level (e.g., requirements corresponding to user membership levels) are analyzed. The extracted feature parameters are then input into a mapping model to calculate a quantified value of the request's complexity. The calculation formula is as follows: ,like Figure 1 As shown.
[0023] Priority Calculation and Dynamic Sorting: A priority evaluation mechanism is constructed, combining factors such as order complexity, current waiting time, and node load. The priority score for each pending request is calculated using a fair priority index formula, which is as follows: ,in, This is the priority score. For complex quantification values, The initial waiting threshold, This is the current waiting time. This is the initial load threshold. This represents the overall load value of the current node. For example, an order with a longer waiting time and a higher complexity metric value will have a higher score than a newly received order with a lower complexity metric value. The order processing queue is built according to the scores from high to low. At the same time, a periodic re-evaluation mechanism is set up. If the waiting time of an order increases, its score will increase accordingly, and the queue order will be dynamically adjusted.
[0024] Request processing and threshold control: Orders are processed sequentially according to the queue order. After each order is completed, the waiting time for the remaining orders is automatically updated. Furthermore, the waiting threshold in the fair priority index formula is dynamically adjusted based on the current node load (e.g., when the load is high) and the queue length using an adaptive waiting threshold formula. The adaptive waiting threshold formula is as follows: ,in, The waiting threshold is dynamically adjusted. The initial waiting threshold, The current request queue length. This is the initial load threshold. the comprehensive load value of the current node; When the node load is low, the waiting threshold is further reduced, and the waiting time of each order is monitored in real time. When the waiting time of an order reaches the adjusted waiting threshold, the priority score of the order is raised to the highest, and the order is inserted at the head of the queue for priority processing.
[0025] In summary, in the order processing scenario during the large promotion period of the e-commerce platform, by constructing the mapping relationship model of the order request features and the processing complexity, setting the initial load threshold and the waiting threshold, efficient processing of concurrent order requests is realized. When receiving an order request, a unique ID and a tracking file are generated for monitoring, and whether the node needs to be shunted is determined according to the node comprehensive load value, and part of the orders are forwarded to the adjacent node with lower load. For the remaining orders, the priority is calculated in combination with the complexity, waiting time and node load, and the processing queue is dynamically adjusted, and the waiting threshold is dynamically adjusted according to the load and the queue length, so as to ensure that the order processing is efficient and meets the requirements of the SLA agreement, and effectively cope with the high concurrency pressure during the large promotion period.
[0026] Example two: model construction: in the real-time transaction settlement scenario of financial institutions, collect the real-time transaction data of the core system of the bank, including a large number of transaction amounts, types (city transfer, cross-border remittance, regular deposit withdrawal, etc.), the number of accounts involved, whether anti-money laundering check is required, and other feature parameters, and the actual settlement time of each transaction from initiation to fund arrival, establish a training sample library, generate a mapping relationship model of transaction request features and settlement complexity through training, and the output of the mapping relationship model is the request complexity value. At the same time, combined with the hardware performance of the settlement server (including CPU, memory, network interface, disk I / O, etc.), the initial load threshold is set according to the historical transaction peak load, and the initial waiting threshold is determined according to the requirements of the SLA agreement signed with the regulatory authority on real-time transaction settlement delay.
[0027] Request receiving and load monitoring: during the morning peak period of fund transactions, a settlement node of a branch bank receives concurrent transaction requests from branch banks in surrounding areas. Each request contains transaction amount, payment account, receiving account, transaction type, and note. A unique transaction number is generated for each transaction as a request ID, and a tracking file is created to record the receiving time and the initial state as "to be cleared". At the same time, the performance indicators of the node, such as CPU utilization, memory occupancy, network interface rate, and disk I / O rate, are collected regularly. The comprehensive load value of the current node is calculated by the node load formula: , and the comprehensive load value of the current node is compared with the initial load threshold . If , it is determined that the node is in a high load state, and if , determine that the node is in a normal load state, as shown in Figure 2 .
[0028] Load shunting and complexity assessment: when the current node comprehensive load value exceeds the initial load threshold, query the load states of other settlement nodes in the branch, filter out candidate nodes with load below the threshold, calculate the optimal shunting coefficient through the optimal shunting coefficient formula: , determine the optimal shunting ratio, and preferentially forward transactions of the payee account in the service area of the candidate node. For transactions remaining for processing, analyze characteristic parameters such as data size, complexity coefficient (whether cross-border), data compression ratio, transmission hop count, network topology coefficient, and resource demand level (such as VIP customers' large intra-city transfers), and input the extracted characteristic parameters into the mapping relationship model to calculate the complexity quantization value of the request, whose calculation formula is: .
[0029] Priority calculation and dynamic sorting: build a priority evaluation mechanism, combine transaction complexity, current waiting time, and node load factors, and calculate the priority score of each pending request through the fair priority index formula, whose formula is: For example, a cross-border transaction with a long waiting time and a high complexity quantization value has a higher score than a small amount of transfer with a low complexity quantization value just received, and an order processing queue is constructed from high to low according to the score. At the same time, a regular re-evaluation mechanism is set up. If the waiting time of a transaction increases, its score will increase accordingly, and the queue order will be dynamically adjusted.
[0030] Request processing and threshold control: process transaction settlement according to the queue order, update the waiting time of the remaining transactions after completing each transaction, and dynamically adjust the waiting threshold in the fair priority index formula through the adaptive waiting threshold formula: when the node load is high; When the node load is low, the waiting threshold is further reduced. At the same time, the waiting time of each transaction is monitored. When the waiting time of a transaction reaches the adjusted waiting threshold, its priority score is raised to the highest, and it is inserted at the beginning of the queue for priority settlement.
[0031] In summary, in the real-time transaction settlement scenario of a financial institution, by constructing a mapping relationship model of transaction request features and settlement complexity, combining the initial threshold set by server hardware performance and SLA protocol, a reliable coordination mechanism is provided for real-time transaction settlement. After receiving the transaction request, the unique transaction number is tracked and managed, and it is determined whether to be distributed to other low-load nodes according to the node comprehensive load value. The remaining transactions are calculated for priority and dynamically sorted according to complexity, waiting time and node load, and the waiting threshold is dynamically adjusted to ensure the timeliness of transaction settlement, balance the load, and improve the stability and efficiency of financial transaction processing.
[0032] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content to obtain equivalent embodiments with equivalent changes, without departing from the technical solution of the present application. Any modification, equivalent change and modification of the above embodiments made according to the technical essence of the present application are still within the scope of the technical solution of the present application.
Claims
1. A method for coordinating communication in a large data network, the method comprising: The specific steps of the method are: Model construction: collect historical request data to establish a training sample library, train a mapping relationship model of request features and complexity, and set initial load threshold and initial waiting threshold; Request receiving and load monitoring: the node receives concurrent communication requests from different users, generates a unique request ID for each request and creates a request tracking file, and periodically collects node hardware performance index values, calculates the comprehensive load value of the current node, and compares the comprehensive load value of the current node with the initial load threshold to determine; Load shunting and complexity evaluation: when the comprehensive load value of the current node exceeds the initial load threshold, query the load state of the adjacent node, select the candidate node below the initial load threshold, calculate the optimal shunting coefficient through the optimal shunting coefficient formula, get the shunting ratio, and forward the request with the target address close to the candidate node, analyze the feature parameters of the request retained for processing, and calculate the complexity quantization value through the mapping relationship model; Priority calculation and dynamic sorting: build a priority evaluation mechanism, calculate the priority score of each request to be processed through the fair priority index formula, build a request processing queue according to the score from high to low, and set a periodic reevaluation mechanism to dynamically adjust the queue order; Request processing and threshold control: process the request according to the queue order, and dynamically adjust the waiting threshold according to the node load and request queue length, and monitor the request waiting time.
2. The method of claim 1, wherein, In the model construction, historical request data is collected to establish a training sample library, the sample contains request feature parameters and actual processing time, and the sample data is trained to generate a mapping relationship model of request features and complexity, and the output of the mapping relationship model is the request complexity value. At the same time, according to the node hardware performance and historical load data, the initial load threshold is set, wherein the node hardware performance includes CPU utilization, memory occupancy, network interface rate and disk I / O rate, and the initial waiting threshold is determined based on the SLA service level agreement.
3. The method of claim 1, wherein, In the request receiving and load monitoring, the node receives concurrent communication requests from different users, each request contains data content, request type, target address and user identification meta information, generates a unique request ID for each request, and creates a request tracking file containing receiving time and initial state. At the same time, periodically collect the performance index values of CPU utilization, memory occupancy, network interface rate and disk I / O rate of the node, and calculate the comprehensive load value of the current node through the node load formula.
4. The method of claim 3, wherein, In the request receiving and load monitoring, the comprehensive load value of the current node is calculated through a node load formula, and the node load formula is: wherein, is the comprehensive load value of the current node, is the CPU utilization rate, is the memory occupancy, is the disk I / O rate, is the network interface rate, is the weight coefficient of each index. comparing the current node aggregate load value to an initial load threshold value if the node is determined to be in a high load state, and if the node is determined to be in a normal load state.
5. The method of claim 1, wherein, In the load balancing and complexity assessment, the optimal load balancing coefficient is calculated using the optimal load balancing coefficient formula, which is: ,in, The optimal diversion coefficient is... For the first The proportion of remaining resources for each candidate node. To reach the first Network distance of each candidate node For the first The overall load value of each candidate node. This represents the overall load value of the current node. This represents the average overall load value of the region to which the current node belongs. The number of candidate nodes. For indexing.
6. The method of claim 1, wherein, The load shunting and complexity evaluation includes the following steps: analyzing the data size in the request of the request for remaining processing , complexity coefficient , data compression ratio , transmission hop count , network topology coefficient and resource requirement level characteristic parameters, and inputting the extracted characteristic parameters into a mapping relationship model to calculate the complexity quantization value of the request , and the calculation formula is as follows: .
7. The method of claim 1, wherein, In the priority calculation and dynamic sorting, a priority evaluation mechanism is constructed, based on the factors of request complexity, current waiting time and node load, the priority score of each request to be processed is calculated through a fair priority index formula, and the fair priority index formula is: wherein, is a priority score value, is a complexity quantization value, is an initial waiting threshold value, is a current waiting time, is an initial load threshold value, is a current node comprehensive load value, is a weight coefficient.
8. The method of claim 1, wherein, In the priority calculation and dynamic sorting, a periodic reevaluation mechanism is set, during the queue waiting process, the priority scores of all unprocessed requests are recalculated according to the request waiting time change and network load fluctuation, and the queue order is dynamically adjusted.
9. The method of claim 1, wherein, In the request processing and threshold control, the requests are taken out in sequence according to the queue order for processing, and the waiting time of the remaining requests in the queue is automatically updated after the processing of each request is completed, and the waiting threshold value in the fair priority index formula is dynamically adjusted according to the node load and the request queue length through the adaptive waiting threshold formula, and the adaptive waiting threshold formula is: wherein, is the dynamically adjusted waiting threshold value, is the initial waiting threshold value, is the current request queue length, is the initial load threshold value, is the current comprehensive load value of the node. When node load is high, increase the waiting threshold; when node load is low, decrease the waiting threshold. Simultaneously, monitor the waiting time of each request. When the waiting time for a certain request Reaching the current dynamically adjusted waiting threshold When, the priority score value will be... Elevate it to the highest priority and process it with due care.