Smart city big data analysis system and method
By using SDN controllers and simulated annealing algorithms in smart city data centers, the congestion problem caused by elephant flow in data center network is solved, data processing performance and load balancing are improved, and high efficiency is achieved , reliable flow control.
Patent Information
- Application Number
- CN202410610945.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-05-16
AI Technical Summary
There are congestion problems caused by elephant flow in smart city data center networks, which affect data processing performance and network load balancing.
Using a combination of SDN controller, network perception module, network monitoring module and path selection module, a simulated annealing algorithm is used to re-rout large-stream data flows, select the optimal path to reduce congestion caused by elephant flow collisions, and increase the load of the data center. Equalization and processing performance.
It realizes a high-efficiency, high-reliability, and low-latency routing method, reduces router port congestion and data flow packet loss rate, has low time complexity and low spatial complexity, and improves the overall performance of the data center.
Smart Images

Figure CN118540264B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data center networks, and in particular to a smart city big data analysis system and method. Background Art
[0002] Smart cities use various information technologies or innovative concepts to connect and integrate urban systems and services based on the network to improve the efficiency of resource utilization, optimize urban management and services, and improve the quality of life of citizens. Make full use of the new generation of information technology in all walks of life in the city to achieve the advanced form of urban informatization based on the next generation of innovation in the knowledge society, realize the deep integration of informatization, industrialization and urbanization, improve the quality of urbanization, realize refined and dynamic management, and improve the effectiveness of urban management and improve the quality of life of citizens.
[0003] When a smart city data center performs data analysis, it first obtains massive amounts of urban data; pre-processes the collected massive amounts of data, including segmentation, cleaning, classification, fusion, etc.; then determines the sensitive data after pre-processing, and performs multiple encryption protection and monitoring on the sensitive data to establish a data protection mechanism; then combines data application scenarios to jointly apply data of different sensitivities, generate intelligent decisions for the target city, and manage the city based on the intelligent decisions. The big data analysis function of smart cities is very complex, and the data processing volume is high. In order to ensure the normal management and service of smart cities, the scale of data centers has grown rapidly, and massive amounts of data need to be processed. The growing scale of data center networks and business needs have posed severe challenges to the load balancing of data center networks. The emergence of a new network architecture based on software-defined networks (SDN) has brought new ideas and opportunities to load balancing research. The controller in the SDN network can obtain the topology, nodes, links, and flow information of the global network, and can then centrally control and schedule network flows.
[0004] Statistics and analysis of traffic in smart city data center networks show that there are two different types of traffic in data center networks, namely rat flows and elephant flows. Among them, rat flows are huge in number in data center networks, accounting for 80%, but carry no more than 10% of the data volume and last for a short time. Rat flows are usually generated by applications such as web search engines and have certain requirements for transmission delay; elephant flows account for a small number in data center networks, no more than 20%, but carry more than 90% of the data volume and last for a long time. Elephant flows are usually generated by applications such as file transfer and backup, and have certain requirements for transmission bandwidth; elephant flows are usually the main cause of congestion in smart city data center networks. Therefore, according to the traffic characteristics of smart city data centers, effective control of traffic, ensuring network utilization, and implementing effective load balancing strategies are the core issues that need to be solved in smart city data center networks. Summary of the invention
[0005] The present invention is conducted to solve the above-mentioned problems and proposes a smart city big data analysis system and method. The purpose is to utilize the immediacy of SDN to obtain global characteristics. Aiming at the characteristics of elephant flows, a large-volume data flow rerouting algorithm is proposed based on the simulated annealing algorithm to select the best traffic path, and finally reduce the congestion caused by elephant flow collisions in the SDN network of the smart city data center, thereby improving the big data analysis and processing performance of the smart city data center.
[0006] In order to achieve the above-mentioned object, the present invention adopts the following scheme: A smart city big data traffic analysis system includes an SDN controller, and a network perception module, a network monitoring module and a path selection module respectively connected to the SDN controller, wherein:
[0007] The network perception module is used to obtain and discover SDN network topology information, and establish the entire initial network topology according to the port correspondence between the router and the terminal, so as to facilitate subsequent path calculation; when the initial network traffic is normal, the SDN controller forwards the traffic according to the shortest path rule established by the initial network topology;
[0008] The network monitoring module is used to send requests to each router at regular intervals and calculate the load capacity of each link, available bandwidth, current traffic of each router, and the flow rate of each port of the router to detect whether there is an elephant flow;
[0009] The path selection module is used to calculate the optimal path for the detected elephant flow, and send the calculated optimal path to the SDN controller, so that the SDN controller generates a new forwarding rule, and the scheduling router forwards the elephant flow according to the new forwarding rule.
[0010] Optionally, after the network perception module establishes the initial network topology, the network monitoring module periodically sends status request messages to the router, receives data flow status messages and port status messages replied by the router, calculates the load capacity of each link, available bandwidth, and current traffic of each router, and the flow rate of each port of the router, and detects data flows with a flow rate greater than 10% of the link capacity as elephant flows.
[0011] Optionally, when the network monitoring module detects an elephant flow, the SDN controller will first determine whether there is an optimal path corresponding to the elephant flow calculated by the path selection module; if not, the SDN controller deletes the forwarding rules of the routers on the original forwarding path of the elephant flow; after deleting the forwarding rules, the routers will forward the data flow because there are no corresponding forwarding rules, and send packetIn event information to the SDN controller. The SDN controller will send the received elephant flow information to the path selection module so as to calculate the optimal path for the detected elephant flow.
[0012] Optionally, based on the elephant flow information sent by the SDN controller, all selectable paths of the elephant flow are determined, and a set of alternative paths with the lowest link cost is calculated. A new solution is generated from the set of alternative paths by iteration, and the adaptation score is compared with the old solution. Then, based on the score, it is determined whether to probabilistically accept a worse new solution. The iteration is performed until the conditions are met, and the solution that meets the conditions is taken as the optimal path.
[0013] In order to solve the above problems, the present invention also provides a smart city big data analysis method, comprising the following steps:
[0014] S1: The network perception module obtains and discovers the SDN network topology information, and establishes the entire initial network topology according to the port correspondence between the router and the terminal, so as to facilitate the subsequent path calculation; when the initial network traffic is normal, the SDN controller forwards the traffic according to the shortest path rule established by the initial network topology;
[0015] S2: The network monitoring module sends requests to each router regularly and calculates the load capacity of each link, available bandwidth, current traffic of each router, and flow rate of each port of the router to detect whether there is an elephant flow;
[0016] S3: If no elephant flow is detected, the SDN controller controls the router to forward traffic according to the shortest path rule; if an elephant flow is detected, the SDN controller first determines whether there is an optimal path corresponding to the elephant flow calculated by the path selection module; if not, proceed to step S4; if yes, proceed to step S6;
[0017] S4: The SDN controller deletes the forwarding rule of the router on the original forwarding path of the elephant flow; after the forwarding rule is deleted, the router forwards the data flow because there is no corresponding forwarding rule, and sends the packetIn event information to the SDN controller. The SDN controller sends the received elephant flow information to the path selection module so as to calculate the optimal path for the detected elephant flow;
[0018] S5: The path selection module calculates the optimal path for the detected elephant flow, and sends the calculated optimal path to the SDN controller, so that the SDN controller generates a new forwarding rule, and the scheduling router forwards the elephant flow according to the new forwarding rule;
[0019] S6: The SDN controller forwards traffic according to the optimal path corresponding to the elephant flow.
[0020] Optionally, step S2 further includes: after the network perception module establishes the initial network topology, the network monitoring module periodically sends a status request message to the router, receives the data flow status message and port status message replied by the router, calculates the load capacity of each link, the available bandwidth and the current traffic of each router, and the flow rate of each port of the router, and detects the data flow with a flow rate greater than 10% of the link capacity as an elephant flow.
[0021] Optionally, the path selection module in step S5 uses an improved simulated annealing algorithm to calculate the optimal path for the detected elephant flow, specifically including:
[0022] S51: Initialize control parameters, including the number of iterations Maxgen, the initial annealing temperature T_MAX, the cooling temperature and the selectivity;
[0023] S52: Determine all the paths that can be selected by the elephant flow, use the lowest link cost as the selection principle, select and calculate K minimum link cost alternative path sets for K times, where K is an integer greater than or equal to 3, and the lowest link cost is: Costpath=LinkCount*(path load+routing cost+average load degree), routing cost=number of links on the path*selection degree, average load degree=average load on the path / path load, where the selection degree of all links is initially 1, and each time it is selected as an alternative path, the record is multiplied by 2;
[0024] S53: Based on the K lowest link cost alternative path sets, the elephant flow that enters the iteration for the first time randomly selects a path from the alternative path set as the historical best solution, and the elephant flow that does not enter the iteration for the first time uses the paths selected by the historical best solutions to form a group of historical best solutions S; all elephant flows randomly select a path from the alternative path set again to form the best solution S' of this iteration and update the selection degree, and then bring all the paths of the historical best solution S and the best solution S' of this iteration into the fitness function to calculate the scores. The fitness function is: AVB ij is the available bandwidth of the jth link in the i-th path, AVB i is the total available bandwidth of the i-th path, Flow_speed is the traffic size of the elephant flow currently to be scheduled, and Li is the number of links in the i-th path;
[0025] S54: If the fitness score F(S′) calculated by the best solution S' of this iteration is lower than or equal to the fitness score F(S) of the best solution S in history, the new solution S' is accepted and the best solution in history is updated; otherwise, if F(S′) is greater than F(S), the process of whether to accept the worse solution S' is triggered;
[0026] S55: randomly draw a number R between 0 and 1, and calculate the acceptance probability P according to the following formula: P = [1 / (F(S′)-F(S)))]×(Now_T / T_MAX). If the P value is greater than or equal to R, the worse new solution is accepted. If the P value is less than R, the current historical best solution is kept unchanged, where Now_T is the current iteration temperature and T_MAX is the initial temperature.
[0027] S56: Determine whether the number of iterations Maxgen is reached. If so, proceed to step S57; if not, subtract the cooling temperature from the current iteration temperature, proceed to step S52 and iterate again until the number of iterations Maxgen is reached, and then proceed to step S57;
[0028] S57: The iteration ends and the current best solution is output as the optimal path.
[0029] In order to solve the above-mentioned problem, the present invention also provides an electronic device, characterized in that the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned big data analysis method.
[0030] In order to solve the above problems, the present invention also provides a computer-readable storage medium storing a computer program, characterized in that the computer program implements the above-mentioned big data analysis method when executed by a processor.
[0031] Functions and Effects of the Invention
[0032] The present invention adopts an SDN controller, and a network perception module, a network monitoring module and a path selection module connected to the SDN controller respectively work together. Aiming at the characteristics of the elephant flow, a large-volume data flow rerouting algorithm is proposed. Based on the improvement of the existing simulated annealing algorithm, the delay consideration of the number of router installations is added. The link load is used to calculate the link cost, the link balance and the link number affect the calculation of the fitness function comparison, and the best flow path is selected. Not only can the link cost be taken into account, but also the delay caused by the number of routers can be taken into account, and finally the congestion caused by the collision of the elephant flow is reduced in the SDN network of the smart city data center. The present invention realizes a high-efficiency, high-reliability and low-delay routing method, which effectively makes the load of the smart city data center balanced, reduces the occurrence of router port congestion, reduces the data flow routing delay, reduces the data flow packet loss rate, and has the characteristics of low time complexity, low space complexity and high performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A schematic diagram of the analysis of big data transmission in smart city data centers in the prior art;
[0034] Figure 2 A schematic diagram of path selection in the prior art;
[0035] Figure 3 A schematic diagram of a system architecture is provided for an embodiment of the present invention;
[0036] Figure 4 A flow chart of a smart city big data analysis method provided by one embodiment of the present invention;
[0037] Figure 5 A schematic diagram of the structure of an electronic device for implementing a smart city big data analysis method provided by one embodiment of the present invention. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when (stated condition or event) is detected" or "in response to detecting (stated condition or event)".
[0040] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.
[0041] Data Center Network Routing Technology Multi-path data transmission is a requirement that data center networks must be able to meet. The diverse characteristics of the data center network topology can provide more convenient conditions for link multi-path transmission. The implementation of multi-path data transmission can effectively reduce the packet loss rate, reduce network service delay and promote the overall improvement of network throughput, so that the utilization of network resources can reach a more reasonable level. At present, the separation of traffic on different paths and the improvement of network performance have become the focus of new routing research. The main algorithms include: (1) Random Equal Cost Mutipath (ECMP) algorithm is to distribute traffic in the network structure in the form of random equivalence. Different flows can be configured on multiple paths according to the flow keywords in the network. The most prominent feature of random equal cost distribution is to ensure the load balancing and traffic backup of multiple paths based on the fact that the value of each link is exactly the same. An important basis of random hierarchical distribution routing is to assume that each link is completely consistent in bandwidth and transmission capacity. However, in the actual operation of the network system, the bandwidth, delay, and stability of each link connected to the data center are significantly different. Once the differences between the paths are obvious, it is difficult to obtain satisfactory results by using random level distribution. (2) Intelligent heuristic algorithm. This algorithm form is the most commonly used algorithm form for dealing with Quality of Service (QoS) routing problems. It mainly includes the more common ant colony algorithm, genetic algorithm, simulated annealing algorithm, etc. The application of intelligent heuristic algorithm can effectively achieve the optimization and improvement of network performance. For example, researchers used the ant colony algorithm to optimize routing in traffic engineering, using the utilization rate of the buffer queue in the router as the node selection standard, and the link utilization rate can serve as the global update pheromone in the ant colony algorithm to select multiple links for data transmission. Although the intelligent heuristic algorithm can significantly improve and optimize the overall network performance, there are still obvious defects between each algorithm. For example, the encoding and decoding of the genetic algorithm are very complicated, and the time consumed for calculation and processing is relatively long.
[0042] The characteristics of the simulated annealing algorithm are that it uses iterative annealing to cool down, uses temperature to calculate probability, and can probabilistically accept differential solutions to break through the local optimal problem. It is an intelligent heuristic algorithm with a certain probability of finding the global optimal solution in a solution set. For those who want to find the best solution for selecting a path in the entire topological network, it is a method that can solve the local optimal problem. However, the annealing algorithm is prone to the situation where the results are difficult to converge due to the huge search space. If the iteration is forced to end, the output result cannot achieve the expected effect, and even worse results may be obtained.
[0043] In addition, from a smart city data center network, effectively detect the elephant flow in the big data traffic, allocate the best path according to the traffic size, and finally successfully install the best path forwarding rules into the router. This is a common flow control process. However, this type of network structure is often more complex, and the number of router jumps in the path will also affect the traffic transmission, which is also a problem that must be considered in the flow control of the smart city data center network.
[0044] like Figure 1 As shown, when a user joins the SDN network for the first time i Want to transmit data packets to user j When the edge router S1 has no user i Therefore, the packetIn event data packet will be sent to the SDN controller C first. After the controller C calculates the best path, it will distribute the new path rule to the corresponding path routers, which are S1, S2, and S3 in the figure. Finally, S1 can complete the forwarding of the data packet to the next router. However, the UL and DL time waiting for C's reply is the delay consumption phenomenon that will occur when the SDN network installs the path rule.
[0045] Currently, in large data center networks, installing router path rules will cause delays depending on the order of installation. Not only will there be additional delays when adding new router path rules, but there will also be delays in the process of installing the path rules on the router. There will be some delays in the transmission of these data packets between routers and between routers and controllers, which will be more significant on large networks. Therefore, the number of routers with path rules installed will affect the performance of the entire data center network.
[0046] like Figure 2 As shown in the figure, a simple network path selection diagram is shown. When the elephant flow of link a from point A to point B is detected, the best allocation path is selected through link cost calculation, which is link c. However, considering the actual installation delay of the router, although the cost of link b is worse than that of link c, the actual router rule installation delay is affected by the c link. This may cause the best path selection result in the actual environment to be inaccurate. Therefore, the number of router installation rules is also one of the factors that must be considered for data center network traffic control.
[0047] In view of the above problems, the embodiment of the present invention proposes a method as follows Figure 3 The smart city big data analysis system shown is characterized by comprising an SDN controller, and a network perception module, a network monitoring module and a path selection module respectively connected to the SDN controller, wherein:
[0048] The network awareness module is responsible for the initiation of various control events, used to obtain and discover SDN network topology information, and establish the entire initial network topology according to the port correspondence between routers and terminals to facilitate subsequent path calculation; when the initial network traffic is normal, the SDN controller forwards the traffic according to the shortest path rule established by the initial network topology;
[0049] The network monitor module is used to send requests to each router regularly (every 2 seconds) and calculate the load capacity of each link, available bandwidth, current traffic of each router, and the flow rate of each port of the router to detect whether there is an elephant flow;
[0050] The path selection module is used to calculate the optimal path for the detected elephant flow after obtaining the network topology information and data flow related information provided by the SDN controller, and send the calculated optimal path to the SDN controller so that the SDN controller generates a new forwarding rule and schedules the router to forward the elephant flow according to the new forwarding rule.
[0051] In detail, after the network perception module establishes the initial network topology, the network monitoring module periodically sends a status request message STATS_REQUEST to the router every 2 seconds, receives the data flow status message FLOW_STATS_REPLY and port status message PORT_STATS_REPLY replied by the router, and uses the interval time and data transmission volume to calculate the router flow rate and the flow rate data of each port of the router as real-time traffic statistics information. The data flow that repeatedly passes through the router will use the latest data as the calculation basis, and the data flow with a flow rate greater than 10% of the link capacity will be detected as an elephant flow.
[0052] In detail, when the network monitoring module detects an elephant flow, the SDN controller will first determine whether there is an optimal path corresponding to the elephant flow calculated by the path selection module; if not, the SDN controller deletes the forwarding rules of the routers on the original forwarding path of the elephant flow; after deleting the forwarding rules, the routers will forward the data flow because there are no corresponding forwarding rules, and send packetIn event information to the SDN controller. The SDN controller will send the received elephant flow information to the path selection module in order to calculate the optimal path for the detected elephant flow.
[0053] In detail, according to the elephant flow information sent by the SDN controller, all the paths that can be selected by the elephant flow are determined, and the set of alternative paths with the lowest link cost is calculated. A new solution is generated from the set of alternative paths in sequence, and the adaptation score is compared with the old solution. Then, according to the score, it is determined whether to probabilistically accept the worse new solution. It is iterated in sequence until the conditions are met, and the solution that meets the conditions is taken as the optimal path.
[0054] In summary, this system is based on traffic statistics detection. For data flows with a current flow rate greater than 10% of the link capacity, it detects them as elephant flows. It will then delete the current path forwarding rules for this data flow, and then import this flow into the path selection module to generate a new optimal path. After waiting for the new router to install the optimal path forwarding rules, it will forward them again to achieve the purpose of scheduling traffic.
[0055] A flow chart of a smart city big data traffic analysis method provided by an embodiment of the present invention is as follows: Figure 4 The specific process is described as follows:
[0056] The overall process starts with the initial topology. The network perception module first obtains and discovers the entire network topology information, and establishes the port correspondence between the router and the terminal to establish the entire network topology for subsequent path and other related calculations. When the traffic is normal at first, the data flow will be forwarded according to the shortest hop path rule established by the initial topology.
[0057] When the network monitoring module detects that the traffic of a data flow meets the elephant flow conditions, it will first determine whether there is an optimal path corresponding to the elephant flow calculated by the path selection module to avoid repeated calculation of the same elephant flow operation in a short time, resulting in redundant resource consumption. If not, the SDN controller deletes the forwarding rules of the router on the original forwarding path of the elephant flow; after deleting the forwarding rules, the router will forward the data flow because there is no corresponding forwarding rule, and send the packetIn event information to the SDN controller. The SDN controller sends the received elephant flow information (such as source IP, MAC, PORT, router, etc.) to the path selection module to calculate the optimal path for the detected elephant flow. Finally, after calculating the optimal path, the forwarding rules are installed on the corresponding new link router, so that the elephant flow to be scheduled can be forwarded using the new path to achieve the purpose of flow control.
[0058] Furthermore, the path selection module will first determine whether the elephant flow has been processed based on the received elephant flow information, to avoid the problem of receiving multiple packetIn event information from the router in a short period of time due to network immediacy. Then, in the iteration, based on all the paths that can be selected by the elephant flow, calculate the set of candidate paths with the lowest link cost, generate a new solution from the candidate set, compare the fitness score with the old solution, and then determine whether to probabilistically accept the worse new solution based on the score, and iterate in sequence until the condition ends.
[0059] At the end of each iteration, no matter whether the best solution in history S or the best solution in the current iteration S' is selected, its path selection degree will be updated, so that the similar path can be avoided from being repeatedly selected in the next iteration. Finally, in order to avoid the loss of continuous waiting caused by failure to converge, the maximum number of iterations is set to limit the number of iterations. When the maximum number of iterations is reached, the current best solution is forced to be accepted to avoid the endless calculation of the algorithm and the consumption of resources.
[0060] In the path selection module, first in the initial network topology stage, the initial path forwarding first calls Dijkstra's algorithm to generate the shortest path selection rule between routers. After completing the initial network topology stage, the network monitoring module is entered to monitor the status regularly until the network monitoring module detects the elephant flow. The corresponding elephant flow information is sent to the path selection module through the SDN controller for optimal path selection processing.
[0061] After receiving the elephant flow information, the path selection module uses the improved simulated annealing algorithm for iterative search. First, the router data flows with the same source and destination are uniformly aggregated and processed, because their paths have certain routing similarities, and they are uniformly aggregated and calculated. The aggregation here does not mean the aggregation of real traffic, but refers to the unified calculation of path selection results for similar traffic. This method can greatly reduce the situation where similar elephant flows are repeatedly calculated, reduce computing costs and the consumption of network resources in smart city data centers.
[0062] Furthermore, in view of the defects of the existing simulated annealing algorithm, which has a large search space and is difficult to converge in calculation, the path selection module in the present invention adopts an improved simulated annealing algorithm to calculate the optimal path selection for the detected elephant flow, uses link cost to improve the path selection method, speeds up the algorithm convergence speed, and uses the link fitness function and the number of routers as reference factors for calculating the path selection strategy, which specifically includes:
[0063] S51: Initialize control parameters, including the number of iterations Maxgen, the initial annealing temperature T_MAX, the cooling temperature and the selectivity.
[0064] S52: Determine all the paths that can be selected by the elephant flow, use the minimum link cost as the selection principle, select K times and calculate the K minimum link cost alternative path sets, where K is an integer greater than or equal to 3, and the minimum link cost is: Costpath = LinkCount * (path load + routing cost + average load degree), routing cost = number of links on the path * selection degree, average load degree = average load on the path / path load, where the selection degree of all links is initially 1, and each time it is selected as an alternative path, the record is multiplied by 2.
[0065] LinkCount refers to the number of links in the path, and the path load is the maximum load on the link router. Because the maximum load is related to the remaining available bandwidth, LinkCount is the delay impact of router installation, so it is calculated as a load penalty. The selection degree of all links is initially 1, and each time it is selected as an alternative path, the record is multiplied by 2. The intention is to exclude similar paths from being repeatedly selected into the alternative path set in iterations. The average load degree is proportional to the link congestion level and inversely proportional to the available bandwidth. The higher the average load degree, the lower the available bandwidth. However, the impact of this parameter is relatively lower than that of the path load and LinkCount, so it is divided by the path load to reduce the calculation ratio.
[0066] S53: Based on the K lowest link cost alternative path sets, the elephant flow that enters the iteration for the first time randomly selects a path from the alternative path set as the historical best solution, and the elephant flow that does not enter the iteration for the first time uses the paths selected by the historical best solutions to form a group of historical best solutions S; all elephant flows randomly select a path from the alternative path set again to form the best solution S' of this iteration and update the selection degree, and then bring all the paths of the historical best solution S and the best solution S' of this iteration into the fitness function to calculate the scores. The fitness function is: AVB ij is the available bandwidth of the jth link in the i-th path, AVB i is the total available bandwidth of the i-th path, Flow_speed is the traffic size of the elephant flow currently to be scheduled, and L i is the number of links in the ith path, and NodeCounts is the number of router nodes.
[0067] The fitness score of the link balance is calculated by taking the average of the square sum of the difference between the link available bandwidth balance and the elephant flow flow size. The smaller this score is, the more balanced the link balance is. When the path averages are different, the smaller the calculated value is, the more balanced it is. If the path averages are the same, the smaller the elephant flow rate is, the less likely the path is to be congested, so the lower this calculated value is, the better. However, in reality, the delay effect of the number of routers installed in the path must also be considered. It is hoped that the lower the link fitness score and the lower the number of routers installed, the better. Therefore, the number of router nodes is multiplied by NodeCounts, and the goal is to find the lowest fitness score of this fitness function as the best solution.
[0068] The available bandwidth can be obtained by subtracting the current load of the link from the initial bandwidth of the link, which is the link flow rate calculated by the network monitoring module. In addition, since network transmission is fast and instant, when there is an opportunity for multiple flows to enter processing at the same time, the alternative path selected by the elephant flow should be processed first, and the available bandwidth calculated by the current statistics should be deducted first to avoid the elephant flows that need to be processed at the same time using the same monitoring statistics to calculate the best solution, which will cause congestion due to the simultaneous selection of the same path.
[0069] S54: If the fitness score F(S′) calculated for the best solution S' of this iteration is lower than or equal to the fitness score F(S) of the best solution S in history, accept the new solution S' and update the best solution in history; otherwise, if F(S′) is greater than F(S), the process S55 of probabilistically accepting the worse solution S' is triggered.
[0070] S55: Randomly draw a number R between 0 and 1, and calculate the acceptance probability P according to the following formula: P = [1 / ((F(S′)-F(S)))]×(Now_T / T_MAX). If the P value is greater than or equal to R, the worse new solution is accepted. If the P value is less than R, the current best historical solution remains unchanged, where Now_T is the current iteration temperature and T_MAX is the initial temperature.
[0071] F(S′) and F(S) are the fitness scores of the new solution S' and the old solution S, respectively. The larger the P value, the easier it is to accept the poor new solution S'. The smaller the difference between the fitness scores of the new solution S' and the old solution S or the higher the current temperature, the larger the P value. In other words, at the beginning of the iteration or when the scores of the new and old solutions are similar, it is easier to probabilistically accept the poor new solution S'. Finally, if the poor new solution S' is probabilistically accepted, the load check must be performed on the current available bandwidth of the link. This poor new solution S' cannot be a poor solution that the bandwidth of the newly selected path link cannot load. Positive solutions can be accepted probabilistically, but the worst path solution that will definitely lead to congestion cannot be accepted.
[0072] S56: Determine whether the number of iterations Maxgen is reached. If so, proceed to step S57; if not, subtract the cooling temperature from the current iteration temperature, proceed to step S52 and iterate again until the number of iterations Maxgen is reached, and then proceed to step S57;
[0073] S57: The iteration ends and the current best solution is output as the optimal path.
[0074] It can be seen that the path selection of the smart city data center found by the present invention has a one-to-one correspondence with the router, and the link load and load bottleneck of the entire data center are obtained according to the selected router. The load bottleneck is reduced by using a routing method based on an improved simulated annealing algorithm to obtain a routing method with better performance within a limited time. This method can be widely used in large tree-shaped data centers with multiple routers to obtain a load-balanced routing method for the data center within a limited time.
[0075] Compared with the traditional traffic scheduling algorithm, the present invention combines the initial routing and rerouting algorithms to enhance the traffic control effect, alleviate network congestion, reduce packet loss, and improve link utilization. The present invention adopts different algorithms for long and short flows, which not only meets the low latency requirements of short flows but also meets the high throughput requirements of long flows. Based on the improved simulated annealing algorithm, starting from the pheromone initialization and update strategy, the relevant concepts and operations are redefined, and the existing traffic scheduling schemes based on the simulated annealing algorithm are optimized, which are prone to fall into local optimality and slow convergence.
[0076] Therefore, the present invention can significantly improve the utilization rate of the routing links of the entire smart city data center, reduce the probability of link congestion and transmission flow packet loss, and can effectively increase the overall throughput of the smart city data center. It overcomes the shortcomings of traditional routing methods that cannot monitor the global state and adjust the routing method according to the real-time concurrency, and obtains a routing solution with low computational complexity and good performance, and has the characteristics of high reliability and low latency, and has high application value in actual engineering.
[0077] like Figure 5 , is a structural diagram of an electronic device for implementing a smart city big data traffic analysis method provided by an embodiment of the present invention.
[0078] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and run on the processor 10, such as a smart city big data fusion analysis cloud program. In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes the programs or modules stored in the memory 11, and calls the data stored in the memory 11 to perform various functions of the electronic device and process data.
[0079] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the smart city big data fusion analysis cloud program, but also can be used to temporarily store data that has been output or is to be output.
[0080] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.
[0081] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0082] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, WiFi modules, etc., which will not be repeated here.
[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0084] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0085] In addition, the functional modules in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules. These computer program instructions may also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce computer-implemented processing, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0086] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A smart city big data analysis system, characterized in that: It includes an SDN controller, and a network perception module, a network monitoring module and a path selection module respectively connected to the SDN controller, wherein: The network perception module is used to obtain and discover SDN network topology information, and establish the entire initial network topology according to the port correspondence between the router and the terminal, so as to facilitate subsequent path calculation; when the initial network traffic is normal, the SDN controller forwards the traffic according to the shortest path rule established by the initial network topology; The network monitoring module is used to send requests to each router at regular intervals and calculate the load capacity of each link, available bandwidth, current traffic of each router, and the flow rate of each port of the router to detect whether there is an elephant flow; The path selection module is used to calculate the optimal path for the detected elephant flow, and send the calculated optimal path to the SDN controller, so that the SDN controller generates a new forwarding rule, and the scheduling router forwards the elephant flow according to the new forwarding rule; the path selection module uses an improved simulated annealing algorithm to calculate the optimal path for the detected elephant flow, specifically including: S5 1: Initialize control parameters, including the number of iterations Maxgen, the initial annealing temperature T_MAX, the cooling temperature and the selectivity; S52: Determine all the paths that can be selected by the elephant flow, use the minimum link cost as the selection principle, select and calculate K minimum link cost alternative path sets for K times, where K is an integer greater than or equal to 3, and the minimum link cost is: Costpath=LinkCount*(path load+routing cost+average load degree), routing cost=number of links on the path*selection degree, average load degree=average load on the path / path load, where the selection degree of all links is initially 1, and each time it is selected as an alternative path, the record is multiplied by 2, and LinkCount refers to the number of links on the path; S53: Based on the K lowest link cost alternative path sets, the elephant flow that enters the iteration for the first time randomly selects a path from the alternative path set as the historical best solution, and the elephant flow that does not enter the iteration for the first time uses the paths selected by the historical best solutions to form a group of historical best solutions S; all elephant flows randomly select a path from the alternative path set again to form the best solution S' of this iteration and update the selection degree, and then bring all the paths of the historical best solution S and the best solution S' of this iteration into the fitness function to calculate the scores. The fitness function is: Among them, AVB ij is the available bandwidth of the jth link in the i-th path, AVB i is the total available bandwidth of the i-th path, Flow_speed is the traffic size of the elephant flow currently to be scheduled, and L i is the number of links in the ith path, NodeCounts is the number of router nodes; S54: If the fitness score F(S′) calculated by the best solution S' of this iteration is lower than or equal to the fitness score F(S) of the best solution S in history, the new solution S' is accepted and the best solution in history is updated; otherwise, if F(S′) is greater than F(S), the process S55 of whether to accept the worse solution S' is triggered probabilistically; S55: Randomly draw a number R between 0 and 1, and calculate the acceptance probability P according to the following formula: P = [1 / ((F(S′)-F(S)))]×(Now_T / T_MAX), if the P value is greater than or equal to R, the worse new solution is accepted, if the P value is less than R, the current historical best solution is kept unchanged, where Now_T is the current iteration temperature and T_MAX is the initial temperature; S56: Determine whether the number of iterations Maxgen is reached. If so, proceed to step S57; if not, subtract the cooling temperature from the current iteration temperature, proceed to step S52 and iterate again until the number of iterations Maxgen is reached, and then proceed to step S57; S57: The iteration ends and the current best solution is output as the optimal path.
2. A smart city big data analysis system according to claim 1, characterized in that: After the network perception module establishes the initial network topology, the network monitoring module periodically sends status request messages to the router, receives data flow status messages and port status messages replied by the router, calculates the load capacity of each link, available bandwidth, and the current traffic of each router and the flow rate of each port of the router, and detects data flows with a flow rate greater than 10% of the link capacity as elephant flows.
3. A smart city big data analysis system according to claim 1, characterized in that: The SDN controller is further configured to: When the network monitoring module detects an elephant flow, the SDN controller first determines whether there is an optimal path corresponding to the elephant flow calculated by the path selection module; If not, the SDN controller deletes the forwarding rules of the router on the original forwarding path of the elephant flow; after deleting the forwarding rules, the router will forward the data flow because there is no corresponding forwarding rule, and send packetIn event information to the SDN controller. The SDN controller sends the received elephant flow information to the path selection module so as to calculate the optimal path for the detected elephant flow.
4. A smart city big data analysis method, the method is based on a smart city big data analysis system according to any one of claims 1 to 3, characterized in that: The following steps are involved: S1: The network perception module obtains and discovers the SDN network topology information, and establishes the entire initial network topology according to the port correspondence between the router and the terminal, so as to facilitate the subsequent path calculation; when the initial network traffic is normal, the SDN controller forwards the traffic according to the shortest path rule established by the initial network topology; S2: The network monitoring module sends requests to each router regularly and calculates the load capacity of each link, available bandwidth, current traffic of each router, and flow rate of each port of the router to detect whether there is an elephant flow; S3: If no elephant flow is detected, the SDN controller controls the router to forward traffic according to the shortest path rule; if an elephant flow is detected, the SDN controller first determines whether there is an optimal path corresponding to the elephant flow calculated by the path selection module; If not, go to step S4, if yes, go to step S6; S4: The SDN controller deletes the forwarding rule of the router on the original forwarding path of the elephant flow; after the forwarding rule is deleted, the router forwards the data flow because there is no corresponding forwarding rule, and sends the packetIn event information to the SDN controller. The SDN controller sends the received elephant flow information to the path selection module so as to calculate the optimal path for the detected elephant flow; S5: The path selection module calculates the optimal path for the detected elephant flow, and sends the calculated optimal path to the SDN controller, so that the SDN controller generates a new forwarding rule, and the scheduling router forwards the elephant flow according to the new forwarding rule; S6: The SDN controller forwards traffic according to the optimal path corresponding to the elephant flow.
5. A smart city big data analysis method according to claim 4, characterized in that: The step S2 further includes: after the network perception module establishes the initial network topology, the network monitoring module periodically sends a status request message to the router, receives the data flow status message and port status message replied by the router, calculates the load capacity of each link, the available bandwidth and the current traffic of each router and the flow rate of each port of the router, and detects the data flow with a flow rate greater than 10% of the link capacity as an elephant flow.
6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a smart city big data analysis method as described in any one of claims 4-5.
7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a smart city big data analysis method as described in any one of claims 4-5 is implemented.
