Congestion control method and device, storage medium and electronic device
Through the duel neural network, the parameter combination advantage value in the switch port state is evaluated, and the congestion control parameter configuration is optimized, which solves the problem of low parameter tuning reliability in the existing technology, and improves the performance and stability of the network in different environments.
Patent Information
- Application Number
- CN202411210908.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-30
AI Technical Summary
In the prior art, when the deep reinforcement learning model is tuned with parameters of the congestion control algorithm, due to too many or too few parameter combinations, the action space during the decision-making process is too large or the optimal parameters cannot be selected, which affects network performance and parameter tuning. The results are less reliable.
Introduce a duel neural network, by obtaining the status characteristics of the switch port, using the specified advantage network in the preset decision network to evaluate the advantage value of each parameter value combination, and selecting a more favorable parameter value combination under the current port state for congestion control parameter configuration , ensure efficient and stable operation of the network.
It improves the reliability and efficiency of network parameter tuning, enhances the convergence performance and convergence rate of deep reinforcement learning models, and ensures the optimization performance of the network under different topology structures and environments.
Smart Images

Figure CN118740738B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of deep learning technology, and more specifically, to a congestion control method and device, a storage medium, and an electronic device. Background Art
[0002] The congestion control algorithm has many parameters that affect its performance. When it faces different topologies, network environments, and business requirements, it takes a lot of manpower and time to tune the parameters of the congestion control algorithm in the entire data center network.
[0003] At present, in the relevant technology, the switch is regarded as an intelligent entity and the entire data center network is regarded as the external environment. The idea of reinforcement learning can be used to use neural networks to determine the most suitable congestion control algorithm parameters for the switch under the current topology environment and traffic scenario mode.
[0004] However, when the number of combinations of parameters of the congestion control algorithm is too large, DQN (Deep Q-Network, a deep reinforcement learning algorithm used to solve the value function approximation and action selection problems in reinforcement learning problems. DQN uses neural networks to approximate Q value functions and maximizes cumulative rewards by learning the optimal action strategy) faces too large an action space in the final decision-making process. Most actions may not have an impact on the current state of the data center network, so the convergence rate and convergence performance of the entire deep reinforcement learning model will be affected. If the number of combinations of parameters of the congestion control algorithm is too small, it is impossible for the switch to choose the most suitable set of parameters for itself, that is, it is difficult to optimize the choice of parameters of the congestion control algorithm through DQN.
[0005] It can be seen that the congestion control method in the related art has the problem of affecting network performance due to the low reliability of parameter tuning results. Summary of the invention
[0006] The embodiments of the present application provide a congestion control method and device, a storage medium and an electronic device to at least solve the problem of affecting network performance due to low reliability of parameter tuning results in the congestion control method in the related art.
[0007] According to an embodiment of the present application, a congestion control method is provided, comprising: obtaining a state feature corresponding to port data of a switch port of a specified switch within a first time period to obtain a target port state feature; inputting the target port state feature into a preset decision network to obtain a target parameter value combination, wherein the target parameter value combination is one of a plurality of parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch, the preset decision network comprises a specified advantage network of a specified duel network, the specified advantage network is used to evaluate an advantage value corresponding to each parameter value combination in the plurality of parameter value combinations under a port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the plurality of parameter value combinations output by the specified advantage network under a target port state corresponding to the target port state feature is a target advantage value, the target parameter value combination is a parameter value combination selected from the plurality of parameter value combinations based on the target advantage value corresponding to each parameter value combination; configuring the set of congestion control parameters according to the target parameter value combination, and performing congestion control according to the configured set of congestion control parameters.
[0008] According to another embodiment of the present application, a congestion control device is provided, comprising: a first acquisition unit, configured to acquire a state feature corresponding to port data of a switch port of a specified switch within a first time period, and obtain a target port state feature; an input unit, configured to input the target port state feature into a preset decision network, and obtain a target parameter value combination, wherein the target parameter value combination is one of a plurality of parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch, the preset decision network comprises a specified advantage network of a specified duel network, the specified advantage network is configured to evaluate an advantage value corresponding to each parameter value combination in the plurality of parameter value combinations under a port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the target port state corresponding to the target port state feature output by the specified advantage network is a target advantage value, and the target parameter value combination is a parameter value combination selected from the plurality of parameter value combinations based on the target advantage value corresponding to each parameter value combination; an execution unit, configured to configure the set of congestion control parameters according to the target parameter value combination, and perform congestion control according to the configured set of congestion control parameters.
[0009] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein the steps of any of the above method embodiments are executed when the program is run.
[0010] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the steps in any one of the above method embodiments through the computer program.
[0011] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.
[0012] Through the embodiments of the present application, an action advantage branch is introduced in the duel neural network, and based on the action advantage in the current port state of the switch, that is, the relative advantage between different parameter value combination configurations of the congestion control parameters, a parameter value combination that is more advantageous (for example, a larger target advantage value) for the switch output port in the current port state is selected, and the parameter value configuration of the congestion control parameter is performed based on the selected parameter value combination, and congestion control is performed based on the configured congestion control parameters. This can achieve parameter adjustment and optimization according to the specific situation of the network to ensure efficient and stable operation of the network, thereby solving the problem of affecting network performance due to the low reliability of parameter tuning results in the congestion control method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a hardware structure block diagram of a congestion control method according to an embodiment of the present application.
[0014] Figure 2 It is a flow chart of a congestion control method according to an embodiment of the present application.
[0015] Figure 3 It is a schematic diagram of the network structure of a preset decision network according to an embodiment of the present application.
[0016] Figure 4 It is a flowchart of another congestion control method according to an embodiment of the present application.
[0017] Figure 5 It is a flowchart of another congestion control method according to an embodiment of the present application.
[0018] Figure 6 It is a flowchart of another congestion control method according to an embodiment of the present application.
[0019] Figure 7 It is a structural block diagram of an optional congestion control device provided in an embodiment of the present application.
[0020] Figure 8 It is a structural block diagram of a computer system of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0022] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0024] The method embodiments provided in the embodiments of the present application can be applied to network devices. Figure 1 is a hardware structure block diagram of a congestion control method according to an embodiment of the present application. Figure 1 As shown, congestion control may involve the interaction between the sender, the receiver, and the designated switch (or router), and the switch may be responsible for forwarding received data packets from one network interface to another. When the switch forwards data packets, if it finds that the queue length on a link or interface exceeds a preset threshold, it will consider the network to be in a congested state. Through the collaboration of the sender, the receiver, and the designated switch (or router), the stability and efficiency of the network can be ensured. Specifically, the sender responds to network congestion by adjusting the sending rate, the receiver controls the sending rate of the sender through confirmation and receiving window management, and the designated switch (or router) is responsible for forwarding and congestion control of data packets. Through this application, the network can avoid or alleviate congestion problems to a certain extent.
[0025] In this embodiment, a congestion control method is provided. Figure 2 is a flow chart of a congestion control method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:
[0026] Step S202: Acquire a state feature corresponding to the port data of the switch port of the designated switch in the first time period to obtain a target port state feature.
[0027] The congestion control method in this embodiment can be applied to the scenario of configuring the congestion control parameters of the switch port in the data center network. The algorithm involved in this application only needs to be deployed at the switch egress end, has low complexity, can be quickly calculated using the CPU on the switch, and is easy to implement.
[0028] Data center networks usually carry a large number of data transmission tasks, including cloud computing, big data analysis, and real-time communication. In these applications, the quality of network performance directly affects user experience and business efficiency. Congestion control algorithms optimize the allocation and use of network resources to ensure smooth transmission of data packets in the network, thereby improving the overall performance of the network. At the same time, congestion control algorithms monitor the network status in real time and dynamically adjust the data transmission rate to avoid packet loss and retransmission caused by network congestion, thereby improving data transmission efficiency. This is particularly important for business scenarios that require rapid processing of large amounts of data. In addition, network congestion not only leads to performance degradation and reduced transmission efficiency, but may also cause system crashes or failures, such as PFC (Priority Flow Control) deadlock and PFC storms. Congestion control algorithms prevent and handle congestion problems, reduce the risk of network failures, and enhance system stability and reliability. This is of great significance for ensuring the continuous and stable operation of data center networks. Finally, the resources in data center networks include bandwidth, buffers, computing resources, etc. The limited nature of these resources makes it particularly important to allocate and utilize resources reasonably. Congestion control algorithms optimize resource utilization and improve the overall efficiency of the network by intelligently scheduling and managing network resources.
[0029] In current data center networks, the congestion control algorithm widely deployed on switches is DCQCN (Data Center Quantized Congestion Notification). It combines the features of QCN (Quantized Congestion Notification) and DCTCP (Data Center Transmission Control Protocol). By finely controlling network congestion, it not only significantly improves the throughput and fairness of RoCEv2-RDMA (Remote Direct Memory Access over Converged Ethernet version 2) traffic, but also effectively avoids unfairness and victim flow problems in the network. As a rate-based congestion control algorithm, DCQCN uses the ECN (Explicit Congestion Notification) mechanism to adjust the sending rate in real time and quickly respond to network congestion conditions, thereby improving data transmission efficiency and stability. However, the DCQCN algorithm has too many critical hyperparameters that affect its performance. When it faces different topologies, different network environments and business requirements, it takes a lot of manpower and time to tune the DCQCN parameters in the entire data center network. Among them, the three most important parameters are Kmin, Kmax and Pmax located on the data center network switch.
[0030] Among them, Kmin usually refers to the minimum value of the ECN marking threshold at which a switch or router starts marking packets when network congestion occurs. In other words, it is the minimum congestion level that triggers ECN marking. When the queue length or delay reaches Kmin, the network device starts marking the ECN bit on the packets instead of dropping them immediately;
[0031] Kmax refers to the maximum value of the ECN marking threshold at which the network device marks packets. This is the maximum degree to which the network device can mark before congestion becomes too severe. If the queue length or latency exceeds Kmax, the network device may start dropping packets even if they have not yet been marked.
[0032] Pmax refers to the maximum marking probability or maximum packet loss rate that can be tolerated in the network. This is the maximum proportion of packets that a network device can mark in ECN marking mode. If the marking probability exceeds Pmax, the network may need to take more aggressive measures to control congestion, such as starting to drop packets.
[0033] If the queue length is less than the lower limit, the message is not discarded, and the ECN field is not identified and marked.
[0034] If the queue length is between the upper and lower limits, when the device calculates that a message needs to be discarded based on the discard probability, it will check the ECN field of the message. If the ECN field shows that the message is sent by a terminal that supports ECN, the device will mark both the ECT bit and the CE bit of the message as 1 and then forward the message; if the ECN field shows that the message transmission path has experienced congestion (that is, both the ECT and CE bits are 1), the device directly forwards the message without re-marking the ECN field; if both the ECT bit and the CE bit are 0, the device will discard the message.
[0035] If the queue length exceeds the upper limit, the ECN fields of all packets in the queue will be marked as 11. When the queue length reaches the queue tail discard threshold, the packets will be discarded.
[0036] The rise of artificial intelligence technology has brought new opportunities and challenges to the development of congestion control algorithms. In particular, the intelligent congestion control model that combines deep reinforcement learning algorithms and DCQCN algorithms has provided new impetus for the development of data center networks. The DCQCN algorithm has been criticized for its large number of parameters. It regards the switch as an intelligent agent and the entire data center network as the external environment. Therefore, the idea of reinforcement learning can be used to use neural networks to select the most suitable DCQCN parameters for the switch in the current topology environment and traffic scenario mode. However, when the number of combinations of Kmin, Kmax and Pmax is too large, DQN faces too large an action space in the final decision-making process, and most actions may not have an impact on the current state of the data center network. Therefore, the convergence rate and convergence performance of the entire deep reinforcement learning model will be affected. If the number of combinations of Kmin, Kmax and Pmax is too small, the switch cannot choose the most suitable set of ECN parameters for itself. Combining the two factors, it is difficult to optimize the selection of DCQCN parameters through DQN (Deep Q-Network, DQN uses deep neural networks to approximate Q functions, so that it can learn complex strategies).
[0037] For example, using the Double-DQN technology, when Kmin, Kmax, and Pmax each have n choices, even if you set The total preset selection method of ECN parameter configuration is also close to When the selection range is increased and the parameter granularity is refined, will increase, and the total number of ECN parameter configuration types will also increase dramatically, seriously affecting the convergence performance of the deep reinforcement learning model.
[0038] In order to at least partially solve the above problems, in this embodiment, the dueling network Dueling-DQN is used to decouple the irrelevant value and Q value of the action, so that the learning is more robust, thereby completing the selection of the switch ECN configuration parameters.
[0039] Optionally, in this embodiment, a state feature corresponding to port data of a switch port of a designated switch within a designated time period is acquired to obtain a target port state feature.
[0040] Optionally, the switch can collect and store port data under multiple timestamps from the chip and use it as input state features through calculation. The chip here may refer to the forwarding chip in the switch. Considering that it is time-consuming to perform statistics from the upper-level software, in this embodiment, the port data is directly counted from the underlying chip.
[0041] In an exemplary embodiment, before obtaining the state feature corresponding to the port data of the switch port of the designated switch in the first time period, the method further includes:
[0042] Collecting chip bottom layer data of the switch port of the current switch in the first time period through a software development kit to obtain port data of the switch port of the current switch in the first time period, wherein the designated switch is the current switch; or
[0043] In response to receiving port status characteristics transmitted by a group of upstream switches, chip bottom layer data of the switch port of the current switch in a first time period is collected through a software development kit to obtain port data of the switch port of the current switch in the first time period, wherein the port status characteristics transmitted by the group of upstream switches are status characteristics corresponding to the port data of the switch port of a group of upstream switches in the first time period, and the designated switches include a group of upstream switches and the current switch.
[0044] During the data collection process, you can first determine the collection target: determine the type of data to be collected, such as traffic statistics, network status, device performance indicators, etc. Select collection points: set up data collection points at key locations in the network (such as routers, switches, servers, etc.). For chip underlying data, the collection point will be located on the network interface card (NIC) or network processor. Configure collection tools: use protocols or tools such as SNMP (Simple Network Management Protocol), NetFlow, sFlow, IPFIX, etc. for data collection.
[0045] The collection of chip bottom-level data can be based on specific hardware support and the use of specific software tools to ensure the accuracy and real-time nature of the data. Optionally, for the chip bottom layer, a specific software development kit or application programming interface can be used to access the data of the underlying hardware. The software development kit usually includes a series of library functions, sample codes, documents, and debugging tools, while the application programming interface defines the interface specifications for interaction between software and hardware. With these tools, software developers can write programs to control and monitor the behavior of the underlying hardware to implement advanced network management and congestion control functions.
[0046] The collected data may include but is not limited to at least one of the following:
[0047] Queue length: collects the lengths of queues of different priorities inside the chip, which reflects the processing status of data packets inside the chip.
[0048] Cache utilization: monitors the occupancy of the chip's internal cache to understand the real-time status of data packet processing.
[0049] Processing latency: Measures the time it takes for a data packet to be processed within the chip, including latency in receiving, processing, and sending.
[0050] Error Statistics: Collects the number of error packets due to hardware failure or transmission errors.
[0051] Bandwidth allocation: Get information about how the chip allocates bandwidth to different traffic flows.
[0052] Hardware events: record specific events at the hardware level, such as interrupts, buffer overflows, etc.
[0053] Temperature and Power Consumption: Monitor chip temperature and power consumption, which can affect performance and reliability.
[0054] The underlying hardware can refer to the physical components that are directly involved in data transmission, processing, and storage. For the chip bottom layer, the underlying hardware mainly refers to the following types of components: Network Interface Card (NIC): The network interface card is a hardware device that connects the computer to the network. It is responsible for sending and receiving data packets. Network Processor (NP): A network processor is a chip specifically used to process network data packets. It usually has multiple processing cores and hardware acceleration functions. Switching chip: The switching chip is used to implement data packet switching and routing functions in network devices (such as switches), etc.
[0055] In order to ensure the accuracy of the target port state characteristics, the port data of the switch ports of a group of upstream switches in the first time period can be merged with the port data of the switch ports of the current switch in the first time period to obtain the state characteristics corresponding to the port data of the switch ports of the designated switch in the first time period.
[0056] Step S204: input the target port state characteristic into a preset decision network to obtain a target parameter value combination, wherein the target parameter value combination is one of multiple parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch, the preset decision network includes a specified advantage network of the specified duel network, the specified advantage network is used to evaluate the advantage value corresponding to each parameter value combination in the multiple parameter value combinations under the port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the target port state corresponding to the target port state characteristic output by the specified advantage network is the target advantage value, and the target parameter value combination is a parameter value combination selected from the multiple parameter value combinations based on the target advantage value corresponding to each parameter value combination.
[0057] Optionally, multiple parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch can be pre-stored in a preset decision network, or multiple parameter value combinations and target port state characteristics can be input into the preset decision network together to obtain a target parameter value combination.
[0058] The target parameter value combination is a parameter value combination selected from the plurality of parameter value combinations based on a target advantage value corresponding to each of the plurality of parameter value combinations.
[0059] The preset decision network includes a specified advantage network of the specified duel network, the specified advantage network is used to evaluate the advantage value corresponding to each parameter value combination under the port state of the switch port of the specified switch, and the advantage value corresponding to each parameter value combination under the target port state corresponding to the target port state feature output by the specified advantage network is the target advantage value.
[0060] For example, in this embodiment, the output of the designated advantage network is a vector, in which each element represents the advantage value of taking different actions under a given state. The state here can be a state feature corresponding to the port data of the switch port of the designated switch in the first time period, and the action can be a certain parameter value combination of a set of congestion control parameters corresponding to the switch port of the designated switch.
[0061] The current state and a set of actions are input into a preset decision network, and the action to be performed in the current state is selected based on the target advantage value output by the specified advantage network, that is, the target parameter value combination is selected from multiple parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch.
[0062] By presetting the decision network, a more fine-grained feedback signal is provided to help the switch egress port understand which ECN configuration is more beneficial in a specific state.
[0063] Step S206: configure a set of congestion control parameters according to the target parameter value combination, and perform congestion control according to the configured set of congestion control parameters.
[0064] According to the selected target parameter value combination, a set of congestion control parameters corresponding to the switch port of the designated switch is configured, and congestion control is performed according to the configured set of congestion control parameters.
[0065] In an exemplary embodiment, a set of congestion control parameters includes a lower ECN marking threshold, an upper ECN marking threshold, and a maximum ECN marking probability.
[0066] Congestion control parameters can include one or more marking thresholds (such as the lower limit of the ECN marking threshold Kmin and the upper limit of the ECN marking threshold Kmax), which determine when the switch starts marking packets. Kmin is the minimum congestion level to start marking, and Kmax is the maximum congestion level for marking. By setting these thresholds, the triggering timing of congestion notification can be finely controlled.
[0067] Congestion control parameters may also include a maximum marking probability (such as the maximum ECN marking probability Pmax), which limits the proportion of packets that can be marked in a congested state. This helps prevent too many packets from being marked, thereby avoiding unnecessary network performance degradation.
[0068] Properly configuring congestion control parameters can improve the overall performance of the network because it allows network devices to handle congestion in a more graceful manner rather than simply dropping packets.
[0069] Through the above steps of the embodiment provided by the present application, a state feature corresponding to the port data of the switch port of the specified switch in the first time period is obtained to obtain a target port state feature; the target port state feature is input into a preset decision network to obtain a target parameter value combination, wherein the target parameter value combination is one of multiple parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch, the preset decision network includes a specified advantage network of a specified duel network, the specified advantage network is used to evaluate the advantage value corresponding to each parameter value combination in the multiple parameter value combinations under the port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the target port state corresponding to the target port state feature output by the specified advantage network is a target advantage value, the target parameter value combination is a parameter value combination selected from multiple parameter value combinations based on the target advantage value corresponding to each parameter value combination; a set of congestion control parameters is configured according to the target parameter value combination, and congestion control is performed according to the configured set of congestion control parameters, thereby solving the problem of affecting network performance due to low reliability of parameter tuning results in the congestion control method in the related art.
[0070] In an exemplary embodiment, acquiring a state feature corresponding to port data of a switch port of a designated switch in a first time period to obtain a target port state feature includes:
[0071] S11, determining a set of normalized port states of the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period, wherein the target port state is a set of normalized port states, and the set of normalized port states includes at least one of the following: a normalized packet sending rate, a normalized queue length, and a normalized ECN marking rate;
[0072] S12, performing feature conversion on the normalized port states in the group of normalized port states to obtain a group of port state features, wherein the target port state features are a group of port state features.
[0073] The port data of the switch port of the specified switch in the first time period may include: the packet sending rate of the switch port, which reflects the speed of sending data in a specific time period; the ECN marking rate, which shows the application frequency of the ECN marking; the switch port queue length, which indicates the queue backlog of the switch port at the current moment.
[0074] To ensure that each feature is on the same scale, the port data of the switch ports of the specified switch in the first time period may be normalized.
[0075] Normalization is a data processing method that is mainly used to change the range of data distribution so that it falls within a specific interval, usually [0, 1] or [-1, 1].
[0076] Performing feature conversion on the normalized port states in a set of normalized port states to obtain a set of port state features may refer to organizing the normalized data into a vector form. Each state feature may be represented as a vector, wherein each element corresponds to a port data.
[0077] In an exemplary embodiment, determining a set of normalized port states of the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period includes:
[0078] S21, when a set of normalized port states includes a normalized packet sending rate, determining the number of unit bytes forwarded by the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period, and obtaining a target number of bytes;
[0079] S22, dividing the target number of bytes by the bandwidth rate of the switch port of the designated switch, and determining the result as the normalized packet sending rate.
[0080] For example, in this embodiment, the calculation formula of the normalized packet transmission rate tx_rate is shown in formula (1).
[0081]
[0082] Among them, tx_bytes refers to the number of bytes forwarded by the switch port in the first time period time_interval. It refers to the number of bytes forwarded by the switch port of the specified switch in the first time period, that is, the target number of bytes. Bandwidth refers to the bandwidth rate of the switch forwarding port.
[0083] In an exemplary embodiment, determining a set of normalized port states of the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period includes:
[0084] S31, when a set of normalized port states includes a normalized queue length, determining a queue length of the switch port of the designated switch in the first time period according to port data of the switch port of the designated switch in the first time period, and obtaining a target queue length;
[0085] S32, based on the target queue length, determining a target value from 1 to a specified value, wherein the target value is from 1 to a specified value such that 2 n≥ the minimum value of n for which the target queue length is satisfied. The specified value is such that 2 n ≥ the minimum n value of the maximum queue length of the switch port of the specified switch;
[0086] S33, determining the difference between 1 and the target ratio as the normalized queue length, wherein the target ratio is the ratio of the target value to the specified value.
[0087] For the normalization of queue length, phased inverse exponential interval normalization can be used.
[0088] For example, in this embodiment, the specified value is N, the target value is n, the target queue length is Que_length, and the calculation formula for the normalized queue length que_len is shown in formulas (2), (3), and (4).
[0089]
[0090] Among them, argmin refers to 2 n The minimum value of n that is ≥ Que_length, where N is 2 n The minimum value of n that is ≥ max(Que_length) and the target ratio is .
[0091] In an exemplary embodiment, determining a set of normalized port states of the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period includes:
[0092] S41, in a case where a set of normalized port states includes a normalized ECN marking rate, determining, based on port data of a switch port of a designated switch in a first time period, a number of data packets sent by the switch port of the designated switch in the first time period to obtain a first number of data packets, and determining a number of data packets ECN marked by the switch port of the designated switch in the first time period to obtain a second number of data packets;
[0093] S42: Divide the number of the second data packets by the number of the first data packets to determine the result as a normalized ECN marking rate.
[0094] For example, in this embodiment, the calculation formula of the normalized ECN marking rate ECN_marking_ratio is shown in formula (5).
[0095]
[0096] The first data packet number Tx_pkts and the second data packet number ECN_pkts refer to the number of data packets sent and the number of data packets marked with ECN at the switch port within the first time period time_interval, respectively.
[0097] In an exemplary embodiment, the preset decision network further includes: an input layer and a convolution layer, the input layer is used to transfer the input port state features to the convolution layer, and the convolution layer is used to perform a convolution operation on the port state features transferred by the input layer through weight connection parameters to perform high-dimensional feature mapping on the port state features transferred by the input layer;
[0098] Input the target port status characteristics into the preset decision network to obtain the target parameter value combination, including:
[0099] S51, inputting the target port state feature into the input layer, so that the input layer transmits the target port state feature to the convolution layer;
[0100] S52, performing a convolution operation on the target port state feature through the weight connection parameter to perform high-dimensional feature mapping on the target port state feature to obtain a high-dimensional intermediate feature;
[0101] S53, inputting the high-dimensional intermediate features into the designated advantage network, and obtaining the target advantage value corresponding to each parameter value combination output by the designated advantage network;
[0102] S54, selecting a target parameter value combination from a plurality of parameter value combinations based on the target advantage value corresponding to each parameter value combination.
[0103] The input layer receives the port state features, which can be original state data or pre-processed state representations, similar to the above embodiments. The function of the input layer is to pass the port state features to the next layer, namely the convolutional layer.
[0104] The convolution layer contains multiple convolution operations of the convolutional neural network. The convolution layer performs convolution operations on the port state features transmitted by the input layer through weight connection parameters.
[0105] The convolution operation is a special linear operation that extracts features by performing a sliding window operation on the input features and applying a convolution kernel (also called a filter).
[0106] The purpose of the convolution layer is to perform high-dimensional feature mapping on the state features passed by the input layer. This means that the convolution layer can extract more complex and abstract feature representations from the original low-dimensional features, which can better capture the essence and key information of the state.
[0107] The duel network includes a specified advantage network. By inputting high-dimensional intermediate features into the specified advantage network, a target advantage value corresponding to each parameter value combination output by the specified advantage network can be obtained, which is used to represent the advantage over other actions; based on the target advantage value corresponding to each parameter value combination, a target parameter value combination is selected from multiple parameter value combinations.
[0108] For example, in this embodiment, the switch collects and stores port data at multiple timestamps from the chip and enters the input layer of the duel neural network as input state features through calculation, and performs convolution operations through weight connection parameters to perform high-dimensional feature mapping. The obtained high-dimensional intermediate features are used as independent variables of the advantage function to calculate the advantage value under each ECN configuration, and based on the target advantage value corresponding to each parameter value combination, the target parameter value combination is selected from multiple parameter value combinations.
[0109] In an exemplary embodiment, selecting a target parameter value combination from a plurality of parameter value combinations based on a target advantage value corresponding to each parameter value combination includes:
[0110] S61, determining, among multiple parameter value combinations, a parameter value combination corresponding to the maximum target advantage value as a target parameter value combination.
[0111] Optionally, in this embodiment, when a preset decision network is used, only a specified dominant network branch in a specified duel network may be included, based on which terminal computing resources may be saved.
[0112] Similar to the aforementioned embodiment, the target advantage value is used to indicate the advantage of a parameter value combination of a set of congestion control parameters relative to other parameter value combinations.
[0113] A parameter value combination with the largest target advantage value among multiple parameter value combinations is determined as the target parameter value combination, that is, a group of congestion control parameters is configured according to the parameter value combination with the largest relative advantage value.
[0114] Here, the target advantage value is the relative advantage corresponding to the current port state and parameter value combination, which can represent the difference in expected returns of congestion control based on the target parameter value combination and congestion control based on other parameter value combinations under the current port state.
[0115] Through this embodiment, the parameter value combination with the largest action advantage value is determined as the optimal parameter value combination under the current port state, which can more accurately evaluate the relative advantages and disadvantages of each action, reduce decision bias, and thus maximize long-term returns.
[0116] In an exemplary embodiment, the designated duel network further includes a designated value network, the designated value network being used to evaluate a state value of a port state of a switch port of the designated switch, the state value of the port state of the switch port of the designated switch being an average expected return of a set of congestion control parameters taking each parameter value combination of a plurality of parameter value combinations under the port state of the switch port of the designated switch;
[0117] Input the target port state characteristics into the preset decision network to obtain the target parameter value combination, which also includes:
[0118] S71, when the preset decision network includes a designated value network, inputting the high-dimensional intermediate feature into the designated value network to obtain a target state value corresponding to the target port state feature output by the designated value network;
[0119] Based on the target advantage value corresponding to each parameter value combination, a target parameter value combination is selected from the plurality of parameter value combinations, including:
[0120] S72, determining a reference Q value corresponding to each parameter value combination according to the target state value and the target advantage value corresponding to each parameter value combination;
[0121] S73, determining the parameter value combination with the largest corresponding reference Q value among the multiple parameter value combinations as the target parameter value combination.
[0122] The designated duel network also includes a designated value network, which is used to evaluate the state value of the port state of the switch port of the designated switch, and the state value of the port state of the switch port of the designated switch is the average expected return of each parameter value combination of a set of congestion control parameters taking multiple parameter value combinations under the port state of the switch port of the designated switch.
[0123] In duel networks, the purpose of the state value branch is to predict the state value, which is a scalar that represents the expected return from following a particular strategy in a given state. The state value quantifies the expected value of the long-term return that can be obtained by starting from the current state and following a certain strategy. The state value does not consider a specific action, but rather evaluates the goodness of the state itself. The expected return is the average or expected cumulative reward calculated for a specific state, considering all possible future actions and state transitions, and their corresponding rewards.
[0124] For example, in this embodiment, the duel network can be divided into two branches: the state value branch (i.e., the specified value network branch) and the action advantage branch (i.e., the specified advantage network branch). The specified value network is used to output a single value, representing the state value. The state value is the expected return corresponding to a given state and does not depend on any specific action. It is the expected return that can be obtained by following a given strategy from the current state to the final state.
[0125] The reference Q-value (also called action value) is a combination of the target state value and the target advantage value corresponding to each combination of parameter values, which gives the expected reward of taking an action in the state.
[0126] Optionally, the action with the largest Q value can be determined as the target action, which means that the action has the highest expected return in the current state and has the greatest advantage over other actions. This means that the action is the most likely choice to bring the highest return in the current state. In this embodiment, the action with the largest Q value is usually selected as the optimal action in the current state.
[0127] Through this embodiment, the parameter value combination with the largest Q value is determined as the optimal parameter value combination under the current port state, which can improve the decision quality.
[0128] In an exemplary embodiment, determining a reference Q value corresponding to each parameter value combination according to the target state value and the target advantage value corresponding to each parameter value combination includes:
[0129] S81, determine the sum of the target state value, the target advantage value corresponding to each parameter value combination, and the reference advantage value as the reference Q value corresponding to each parameter value combination, wherein the reference advantage value is the maximum value among the target advantage values corresponding to each parameter value combination, or is the average value of the target advantage values corresponding to each parameter value combination.
[0130] Combination Figure 3 , the duel network includes two neural networks. One neural network (i.e., the designated advantage network) is denoted as It is the target advantage value corresponding to each combination of parameter values Another neural network (i.e., the specified value network) is denoted as , which is the value function of the target state The reference Q value function corresponding to each parameter value combination is recorded as , where s represents the state, a represents the action, and A represents the action set represents the network parameters of the specified dominant network, Represents the network parameters of the specified value network.
[0131] The switch collects and stores port data at multiple timestamps from the chip and enters the input layer of the duel neural network as input state features through calculation, and performs convolution operations through weight connection parameters to perform high-dimensional feature mapping. The obtained high-dimensional intermediate features need to be copied, one for the independent variable of the state value function to calculate the state value corresponding to the current switch state, and the other for the independent variable of the advantage function to calculate the advantage value under each ECN configuration.
[0132] Combined with formula (6), in order to stabilize the training, the maximum value among the target advantage values corresponding to each parameter value combination can be determined as the reference advantage value, and the reference Q value can be adjusted based on the reference advantage value.
[0133]
[0134] Combined with formula (7), in order to ensure an unbiased estimation of the Q value, the average of the target advantage values corresponding to each parameter value combination can be determined as the reference advantage value, and the reference Q value can be adjusted based on the reference advantage value.
[0135]
[0136] In an exemplary embodiment, the designated duel network further includes a designated value network, the designated value network being used to evaluate a state value of a port state of a switch port of the designated switch, the state value of the port state of the switch port of the designated switch being an average expected return of a set of congestion control parameters taking each parameter value combination of a plurality of parameter value combinations under the port state of the switch port of the designated switch;
[0137] Combination Figure 4 After inputting the target port status feature into the preset decision network, the method further includes:
[0138] S402, obtaining a set of training state features, wherein each training state feature in the set of training state features is a state feature corresponding to port data of a switch port of a reference switch in a historical time period;
[0139] S404, training the designated duel network to be trained using a set of training state features to obtain a trained designated duel network;
[0140] S406: Use the network parameters of the designated dominant network in the trained designated duel network to update the network parameters of the designated dominant network in the preset decision network.
[0141] During the training process of the designated duel network, both the designated value network and the designated advantage network in the designated duel network can be trained, and the network parameters can be updated. During use, only the branch corresponding to the designated advantage network can be configured on the terminal, and the network parameters of the designated advantage network in the preset decision network can be regularly updated based on the network parameters of the designated advantage network in the trained designated duel network.
[0142] Exemplarily, duel network training may include the following steps:
[0143] Step 1: Collect state-action pairs (s, a) and their corresponding rewards r. Here, the state refers to the state characteristics corresponding to the port data of the switch port in the first time period, the action refers to the parameter value combination of a set of congestion control parameters corresponding to the switch port, and the reward refers to the reward value obtained from the environment immediately after the action is executed, which is used to reflect the direct feedback given by the environment after the action is executed.
[0144] Step 2: Store the collected experience (state-action pairs and their rewards) in an experience pool; during the training process, randomly extract a batch of experience from the experience pool to train the network.
[0145] Step 3, convert the collected states into a format suitable for network input.
[0146] Step 4, the duel network contains a shared feature extraction layer for processing state features. The shared feature extraction layer is followed by two parallel branches: the state value branch and the action advantage branch. The state value branch outputs the state value, while the action advantage branch outputs the action advantage. The outputs of the two branches are finally merged to obtain the action value (Q value) of each action.
[0147] Step 5: The loss function is used to evaluate the difference between the predicted action value and the actual action value. You can use the mean squared error (MSE) as the loss function. Calculate the gradient of the loss function through backpropagation. Use an optimization algorithm such as gradient descent to update the weights of the network.
[0148] Step 6: Repeat the above steps to improve the performance of the network by continuously interacting with the environment, collecting experience, and training the network.
[0149] Step 7: As the training progresses, the strategy will gradually optimize. In order to improve the stability and convergence speed of training, a target network can be used to store the weights of the current network and regularly update the weights of the target network.
[0150] Optionally, in this embodiment, a set of training state features may include state features corresponding to port data of a switch port of a specified switch in a first time period. A set of training state features is used to train a specified duel network to be trained, to obtain a trained specified duel network; and network parameters of a specified advantageous network in the trained specified duel network are used to update network parameters of a specified advantageous network in a preset decision network. Here, the specified duel network to be trained and the preset decision network may be configured on different terminals.
[0151] In an exemplary embodiment, the designated duel network is a duel deep Q network; the designated duel network to be trained is trained using a set of training state features to obtain a trained designated duel network, including:
[0152] S91, in the process of performing a round of training on the designated duel network, each training state feature in a set of training state features is sequentially used as a current training state feature to perform the following processing operations to obtain an updated designated duel network:
[0153] Input the current training state feature into the specified duel network to obtain the current predicted Q value corresponding to each parameter value combination output by the specified duel network;
[0154] Determine a current loss value corresponding to the specified loss function and the current training state feature based on the current predicted Q value corresponding to each parameter value combination and the current target Q value corresponding to each parameter value combination;
[0155] Based on the current loss value corresponding to the current training state feature, adjust the network parameters of the specified duel network to reduce the current loss value corresponding to the current training state feature.
[0156] The current target Q value corresponding to each parameter value combination may be pre-marked, may be set based on experience, or may be determined based on information such as changes in port data after the parameter value combination is adopted.
[0157] The loss function evaluates the difference between the current predicted Q-value corresponding to each combination of parameter values and the current target Q-value corresponding to each combination of parameter values.
[0158] The loss value is the actual calculation result of the loss function on the given data. During the training process, by optimizing the loss value, the network can learn a more accurate action value function, so that the intelligent agent can make better decisions.
[0159] During the training process, the gradient descent algorithm (and its variants, such as Adam, RMSprop, etc.) can be used to update the weights of the network to reduce the loss value. By iterating this process, the network can gradually learn the strategy of taking the best action in a given state.
[0160] For example, in this embodiment, the loss function may be MSE (Mean Squared Error) loss.
[0161] In an exemplary embodiment, the current training state feature is input into a specified duel network to obtain a current predicted Q value corresponding to each parameter value combination output by the specified duel network, including:
[0162] S101, inputting the current training state feature into the designated duel network, and obtaining the current state value output by the designated value network, and the current advantage value corresponding to each parameter value combination output by the designated advantage network;
[0163] S102, the sum of the current state value, the current advantage value corresponding to each parameter value combination and the current reference advantage value is determined as the current predicted Q value corresponding to each parameter value combination, wherein the current reference advantage value is the maximum value among the current advantage values corresponding to each parameter value combination, or the average value of the current advantage values corresponding to each parameter value combination.
[0164] It should be noted that the calculation method of the current predicted Q value may be consistent with the calculation method of the reference Q value in the aforementioned embodiment, which will not be elaborated here.
[0165] In an exemplary embodiment, before determining the current loss value corresponding to the specified loss function and the current training state feature according to the current predicted Q value corresponding to each parameter value combination and the current target Q value corresponding to each parameter value combination, the method further includes:
[0166] S111, adding the current reward value corresponding to each parameter value combination to the discount coefficient multiplied by the next predicted Q value corresponding to each parameter value combination, and determining the current target Q value corresponding to each parameter value;
[0167] Among them, the current reward value corresponding to each parameter value combination is the reward value corresponding to each parameter value under the current port state corresponding to the current training state feature, the next predicted Q value corresponding to each parameter value combination is the predicted Q value corresponding to each parameter value combination output by the specified duel network after the next port state feature is input into the specified duel network, the next port state feature is the state feature corresponding to the next port data, and the next port data is the port data in the second time period after a set of congestion control parameters are configured according to the corresponding parameter value combination with the largest current predicted Q value in the current port state of the reference switch port.
[0168] For example, in this embodiment, the current reward value r corresponding to each parameter value combination, the discount coefficient , the next predicted Q value Q(s′,a′), the current target Q value The calculation formula of is shown in formula (8).
[0169]
[0170] Among them, r is the reward value given by the system after the agent takes action in the current state, is the discount coefficient, which means the size of the close correlation between the state features at the next moment and the state features at the current moment.
[0171] The next port data is the port data in the second time period after a set of congestion control parameters are configured according to the corresponding parameter value combination with the largest current predicted Q value when the switch port of the reference switch is in the current port state, or the next port data is the port data in the second time period after a set of congestion control parameters are configured according to the corresponding parameter value combination with the largest target advantage value when the switch port of the reference switch is in the current port state.
[0172] In an exemplary embodiment, before determining the result of adding the current reward value corresponding to each parameter value combination to the discount coefficient and multiplying the next predicted Q value corresponding to each parameter value combination as the current target Q value corresponding to each parameter value, the method further includes:
[0173] S121, determining a weighted sum of a reference throughput corresponding to each parameter value combination and a reference queue length corresponding to each parameter value combination as a current reward value corresponding to each parameter value combination;
[0174] Among them, the reference throughput corresponding to each parameter value combination is the throughput of the switch port of the reference switch in the third time period after a set of congestion control parameters are configured according to each parameter value combination in the current port state, and the reference queue length corresponding to each parameter value combination is the queue length of the switch port of the reference switch in the third time period after a set of congestion control parameters are configured according to each parameter value combination in the current port state.
[0175] In this embodiment, the reward value can be determined by the throughput and congestion length obtained by the next sampling. Here, the reference throughput T corresponding to each parameter value combination can refer to the packet sending rate at the switch egress end after normalization, and the calculation method of the congestion queue length que_len after normalization can be similar to the above embodiment, which is not repeated here. The calculation formula of the reward value r is as shown in formula (9).
[0176]
[0177] Among them, ω1 and ω2 are the proportions of the normalized packet sending rate and the normalized value of the congestion queue length at the export end of the reward switch in the reward function, so ω1+ω2=1.
[0178] As an optional exemplary embodiment, the congestion control method is explained by taking the preset decision network as a duel network as an example.
[0179] Based on the introduction of duel neural network, it is possible to optimize and improve the intelligent congestion control model, and enhance the convergence performance and convergence speed of the network model in the multi-dimensional strategy decision space.
[0180] As an optional exemplary embodiment, the congestion control model framework can be as follows Figure 5 As shown:
[0181] Step 501: The switch collects chip bottom layer data through SDK (Software Development Kit);
[0182] Step 502: preprocessing the collected data using formulas (1) to (5);
[0183] Step 503: The state features after preprocessing are input into the duel neural network; here, a fixed traffic scenario is adopted, and the duel neural network needs to be retrained if the scenario changes.
[0184] Step 504: The action with the highest Q value output by the duel neural network is the optimal ECN configuration under the current state characteristics.
[0185] As another optional exemplary embodiment, the congestion control model framework can be as follows: Figure 6 As shown:
[0186] Step 601: Data collection: the switch collects chip bottom layer data through SDK;
[0187] Step 602: Data preprocessing: using formula (1) to formula (5) to perform data preprocessing on the collected data;
[0188] Step 603: Switch state transmission: the state characteristics after preprocessing are transmitted to the downstream switch, and the downstream switch uniformly processes the state of the upstream switch and its own state;
[0189] Step 604: the switch status is updated. Each intelligent agent of the switch updates its own status using the interaction result of step 3.
[0190] Step 605: repeat step 603 and step 604;
[0191] Step 606: The final updated state is input into the duel neural network;
[0192] Step 607: The action with the highest Q value output by the duel neural network is the optimal ECN configuration under the current state characteristics.
[0193] Through this embodiment, a duel neural network is introduced to separate the state value and action value of the current state of the switch, and two data streams are used to independently estimate these two values, one stream focuses on the state value, and the other stream focuses on the advantage function, that is, the relative advantages and disadvantages between different ECN configurations. Through this embodiment, a more fine-grained feedback signal is provided to help the switch egress port understand which ECN configuration is more advantageous in a specific state. Applying the congestion control method in this embodiment to the intelligent congestion control model can improve the convergence performance of the model and make the learning more robust.
[0194] Although the implementation methods described in this application are as above, the above descriptions and definitions are only for the purpose of facilitating the understanding of the implementation methods of this application, and are not intended to limit this application. Any modifications and changes made without departing from the spirit and scope of this application, especially any software and hardware implementation solutions based on this algorithm, are within the scope of protection of the present invention.
[0195] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0196] Figure 7 is an optional congestion control device provided in an embodiment of the present application, such as Figure 7 As shown, the congestion control device comprises:
[0197] The first acquisition unit 702 is used to acquire a state feature corresponding to the port data of the switch port of the designated switch in the first time period to obtain a target port state feature;
[0198] an input unit 704, configured to input the target port state characteristic into a preset decision network to obtain a target parameter value combination, wherein the target parameter value combination is one of a plurality of parameter value combinations of a set of congestion control parameters corresponding to a switch port of a specified switch, the preset decision network includes a specified advantage network of a specified duel network, the specified advantage network is used to evaluate an advantage value corresponding to each parameter value combination in the plurality of parameter value combinations under a port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the target port state corresponding to the target port state characteristic output by the specified advantage network is a target advantage value, and the target parameter value combination is a parameter value combination selected from the plurality of parameter value combinations based on the target advantage value corresponding to each parameter value combination;
[0199] The execution unit 706 is used to configure a set of congestion control parameters according to the target parameter value combination, and perform congestion control according to the configured set of congestion control parameters.
[0200] It should be noted that the first acquisition unit 702 in this embodiment can be used to execute the above step S202, the input unit 704 in this embodiment can be used to execute the above step S204, and the execution unit 706 in this embodiment can be used to execute the above step S206.
[0201] Through the embodiment of the present application, a state feature corresponding to the port data of the switch port of the specified switch in the first time period is obtained to obtain a target port state feature; the target port state feature is input into a preset decision network to obtain a target parameter value combination, wherein the target parameter value combination is one of multiple parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch, the preset decision network includes a specified advantage network of the specified duel network, the specified advantage network is used to evaluate the advantage value corresponding to each parameter value combination in the multiple parameter value combinations under the port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the target port state corresponding to the target port state feature output by the specified advantage network is a target advantage value, the target parameter value combination is a parameter value combination selected from the multiple parameter value combinations based on the target advantage value corresponding to each parameter value combination; a set of congestion control parameters is configured according to the target parameter value combination, and congestion control is performed according to the configured set of congestion control parameters. Through the present application, the problem of affecting network performance due to low reliability of parameter tuning results in the congestion control method in the related art is solved.
[0202] The device is used to implement the congestion control method provided in the above embodiments, and the description has been made no further. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0203] In an exemplary embodiment, the first acquiring unit includes:
[0204] A determination module, configured to determine a set of normalized port states of the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period, wherein the target port state is a set of normalized port states, and the set of normalized port states includes at least one of the following: a normalized packet sending rate, a normalized queue length, and a normalized ECN marking rate;
[0205] The conversion module is used to perform feature conversion on the normalized port states in a group of normalized port states to obtain a group of port state features, wherein the target port state features are a group of port state features.
[0206] In an exemplary embodiment, the determining module includes:
[0207] A first determination submodule is used to determine the number of unit bytes forwarded by the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period, and obtain a target number of bytes when a set of normalized port states includes a normalized packet sending rate;
[0208] The second determination submodule is used to determine the result of dividing the target number of bytes by the bandwidth rate of the switch port of the designated switch as the normalized packet sending rate.
[0209] In an exemplary embodiment, the determining module includes:
[0210] A third determination submodule is used to determine the queue length of the switch port of the specified switch in the first time period according to the port data of the switch port of the specified switch in the first time period, and obtain the target queue length when a set of normalized port states includes a normalized queue length;
[0211] The fourth determination submodule is used to determine a target value from 1 to a specified value based on the target queue length, wherein the target value is from 1 to a specified value such that 2 n ≥ the minimum value of n for which the target queue length is satisfied. The specified value is such that 2 n ≥ the minimum n value of the maximum queue length of the switch port of the specified switch;
[0212] The fifth determination submodule is used to determine the difference between 1 and the target ratio as the normalized queue length, wherein the target ratio is the ratio of the target value to the specified value.
[0213] In an exemplary embodiment, the determining module includes:
[0214] a sixth determination submodule, configured to determine, when a set of normalized port states includes a normalized ECN marking rate, a number of data packets sent by the switch port of the designated switch in the first time period based on the port data of the switch port of the designated switch in the first time period to obtain a first number of data packets, and determine a number of data packets ECN marked by the switch port of the designated switch in the first time period to obtain a second number of data packets;
[0215] The seventh determination submodule is configured to determine a result of dividing the second number of data packets by the first number of data packets as a normalized ECN marking rate.
[0216] In an exemplary embodiment, the preset decision network further includes: an input layer and a convolution layer, the input layer is used to transfer the input port state features to the convolution layer, and the convolution layer is used to perform a convolution operation on the port state features transferred by the input layer through weight connection parameters to perform high-dimensional feature mapping on the port state features transferred by the input layer;
[0217] The input unit includes:
[0218] A first input module, used for inputting the target port state feature into the input layer, so that the input layer transmits the target port state feature to the convolution layer;
[0219] An operation module, used for performing a convolution operation on the target port state feature through a weight connection parameter, so as to perform high-dimensional feature mapping on the target port state feature to obtain a high-dimensional intermediate feature;
[0220] The second input module is used to input the high-dimensional intermediate features into the specified advantage network to obtain the target advantage value corresponding to each parameter value combination output by the specified advantage network;
[0221] The selection module is used to select a target parameter value combination from multiple parameter value combinations based on the target advantage value corresponding to each parameter value combination.
[0222] In an exemplary embodiment, the selection module includes:
[0223] The eighth determination submodule is used to determine, among the multiple parameter value combinations, the parameter value combination with the largest corresponding target advantage value as the target parameter value combination.
[0224] In an exemplary embodiment, the designated duel network further includes a designated value network, the designated value network is used to evaluate the state value of the port state of the switch port of the designated switch, the state value of the port state of the switch port of the designated switch is the average expected return of each parameter value combination of a set of congestion control parameters in the port state of the switch port of the designated switch; the designated duel network is a duel deep Q network; the input unit further includes:
[0225] A third input module is used to input the high-dimensional intermediate features into the designated value network when the preset decision network includes a designated value network, so as to obtain the target state value corresponding to the target port state feature output by the designated value network;
[0226] The selected modules include:
[0227] a ninth determination submodule, for determining a reference Q value corresponding to each parameter value combination according to the target state value and the target advantage value corresponding to each parameter value combination;
[0228] The tenth determination submodule is used to determine, among the multiple parameter value combinations, the parameter value combination with the largest corresponding reference Q value as the target parameter value combination.
[0229] In an exemplary embodiment, the ninth determining submodule includes:
[0230] A determination subunit is used to determine the sum of the target state value, the target advantage value corresponding to each parameter value combination, and the target reference advantage value as a reference Q value corresponding to each parameter value combination, wherein the target reference advantage value is the maximum value among the target advantage values corresponding to each parameter value combination, or the average value of the target advantage values corresponding to each parameter value combination.
[0231] In an exemplary embodiment, the designated duel network further includes a designated value network, the designated value network being used to evaluate a state value of a port state of a switch port of the designated switch, the state value of the port state of the switch port of the designated switch being an average expected return of a set of congestion control parameters taking each parameter value combination of a plurality of parameter value combinations under the port state of the switch port of the designated switch;
[0232] The above device also includes:
[0233] a second acquisition unit, configured to acquire a set of training state features after inputting the target port state feature into a preset decision network, wherein each training state feature in the set of training state features is a state feature corresponding to port data of a switch port of a reference switch in a historical time period;
[0234] A training unit, used for training a designated duel network to be trained using a set of training state features to obtain a trained designated duel network;
[0235] The updating unit is used to update the network parameters of the specified dominant network in the preset decision network by using the network parameters of the specified dominant network in the trained specified duel network.
[0236] In an exemplary embodiment, the duel network is designated as a duel deep Q network; the training unit includes:
[0237] The execution module is used to perform the following processing operations on each training state feature in a set of training state features as the current training state feature in a process of performing a round of training on the specified duel network, so as to obtain an updated specified duel network:
[0238] Input the current training state feature into the specified duel network to obtain the current predicted Q value corresponding to each parameter value combination output by the specified duel network;
[0239] Determine a current loss value corresponding to the specified loss function and the current training state feature based on the current predicted Q value corresponding to each parameter value combination and the current target Q value corresponding to each parameter value combination;
[0240] Based on the current loss value corresponding to the current training state feature, adjust the network parameters of the specified duel network to reduce the current loss value corresponding to the current training state feature.
[0241] In an exemplary embodiment, the execution module includes:
[0242] An input submodule, used to input the current training state feature into the specified duel network, and obtain the current state value output by the specified value network, and the current advantage value corresponding to each parameter value combination output by the specified advantage network;
[0243] The eleventh determination submodule is used to determine the sum of the current state value, the current advantage value corresponding to each parameter value combination, and the current reference advantage value as the current predicted Q value corresponding to each parameter value combination, wherein the current reference advantage value is the maximum value among the current advantage values corresponding to each parameter value combination, or the average value of the current advantage values corresponding to each parameter value combination.
[0244] In an exemplary embodiment, the apparatus further comprises:
[0245] A first determining unit is used to determine, before determining the current loss value corresponding to the specified loss function and the current training state feature according to the current predicted Q value corresponding to each parameter value combination and the current target Q value corresponding to each parameter value combination, the current reward value corresponding to each parameter value combination plus the discount coefficient multiplied by the next predicted Q value corresponding to each parameter value combination as the current target Q value corresponding to each parameter value;
[0246] Among them, the current reward value corresponding to each parameter value combination is the reward value corresponding to each parameter value under the current port state corresponding to the current training state feature, the next predicted Q value corresponding to each parameter value combination is the predicted Q value corresponding to each parameter value combination output by the specified duel network after the next port state feature is input into the specified duel network, the next port state feature is the state feature corresponding to the next port data, and the next port data is the port data in the second time period after a set of congestion control parameters are configured according to the corresponding parameter value combination with the largest current predicted Q value in the current port state of the reference switch port.
[0247] In an exemplary embodiment, the above device further comprises:
[0248] a second determining unit, configured to determine, as a current reward value corresponding to each parameter value combination, a weighted sum of a reference throughput corresponding to each parameter value combination and a reference queue length corresponding to each parameter value combination, before determining a result of adding a current reward value corresponding to each parameter value combination to a discount coefficient and multiplying the next predicted Q value corresponding to each parameter value combination as a current target Q value corresponding to each parameter value;
[0249] Among them, the reference throughput corresponding to each parameter value combination is the throughput of the switch port of the reference switch in the third time period after a set of congestion control parameters are configured according to each parameter value combination in the current port state, and the reference queue length corresponding to each parameter value combination is the queue length of the switch port of the reference switch in the third time period after a set of congestion control parameters are configured according to each parameter value combination in the current port state.
[0250] In an exemplary embodiment, the above device further comprises:
[0251] a first collecting unit, configured to collect chip bottom layer data of the switch port of the current switch in the first time period through a software development kit before acquiring the state feature corresponding to the port data of the switch port of the designated switch in the first time period, so as to obtain the port data of the switch port of the current switch in the first time period, wherein the designated switch is the current switch; or
[0252] The second collection unit is used to collect chip bottom layer data of the switch port of the current switch in a first time period in response to receiving the port status characteristics transmitted by a group of upstream switches, through a software development kit, to obtain port data of the switch port of the current switch in the first time period, wherein the port status characteristics transmitted by the group of upstream switches are status characteristics corresponding to the port data of the switch port of a group of upstream switches in the first time period, and the designated switches include a group of upstream switches and the current switch.
[0253] In an exemplary embodiment, a set of congestion control parameters includes a lower ECN marking threshold, an upper ECN marking threshold, and a maximum ECN marking probability.
[0254] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.
[0255] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk, or an optical disk.
[0256] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.
[0257] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.
[0258] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.
[0259] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flowchart. Figure 8The computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions provided by the embodiments of the present application are executed. The serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.
[0260] refer to Figure 8 , Figure 8 It is a structural block diagram of a computer system of an optional electronic device provided in an embodiment of the present application.
[0261] Figure 8 The computer system structure block diagram of the electronic device used to implement the embodiment of the present application is schematically shown. Figure 8 As shown, the computer system 800 includes a central processing unit 801 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 802 (ROM) or the program loaded from the storage part 808 to the random access memory 803 (RAM). Various programs and data required for system operation are also stored in the random access memory 803. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output interface 805 (Input / Output interface, referred to as I / O interface) is also connected to the bus 804.
[0262] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.
[0263] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions defined in the system of the present application are executed.
[0264] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0265] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0266] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0267] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0268] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.
[0269] The above are only preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.
Claims
1. A congestion control method, characterized in that: include: Acquire a state feature corresponding to the port data of the switch port of the designated switch in the first time period to obtain a target port state feature; Inputting the target port state feature into a preset decision network to obtain a target parameter value combination, wherein the target parameter value combination is one of a plurality of parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch, the preset decision network includes a specified advantage network of a specified duel network, the specified advantage network is used to evaluate the advantage value corresponding to each parameter value combination in the plurality of parameter value combinations under the port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the target port state corresponding to the target port state feature output by the specified advantage network is a target advantage value, and the target parameter value combination is a parameter value combination selected from the plurality of parameter value combinations based on the target advantage value corresponding to each parameter value combination; configuring the set of congestion control parameters according to the target parameter value combination, and performing congestion control according to the configured set of congestion control parameters; Wherein, the designated duel network also includes a designated value network, and the designated value network is used to evaluate the state value of the port state of the switch port of the designated switch, and the state value of the port state of the switch port of the designated switch is the average expected return of each parameter value combination of the plurality of parameter value combinations taken by the set of congestion control parameters under the port state of the switch port of the designated switch; after the target port state feature is input into the preset decision network, the method also includes: obtaining a set of training state features, wherein each training state feature in the set of training state features is a state feature corresponding to the port data of the switch port of a reference switch within a historical time period; using the set of training state features to train the designated duel network to be trained to obtain the trained designated duel network; using the network parameters of the designated advantage network in the trained designated duel network to update the network parameters of the designated advantage network in the preset decision network.
2. The method according to claim 1, characterized in that: The step of acquiring a state feature corresponding to port data of a switch port of a designated switch in a first time period to obtain a target port state feature includes: Determine a set of normalized port states of the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period, wherein the target port state is the set of normalized port states, and the set of normalized port states includes at least one of the following: a normalized packet sending rate, a normalized queue length, and a normalized ECN marking rate; Performing feature conversion on the normalized port states in the group of normalized port states to obtain a group of port state features, wherein the target port state features are the group of port state features.
3. The method according to claim 2, characterized in that The determining, according to the port data of the switch port of the designated switch in the first time period, a set of normalized port states of the switch port of the designated switch in the first time period includes: In a case where the set of normalized port states includes a normalized packet sending rate, determining the number of unit bytes forwarded by the switch port of the designated switch in the first time period according to the port data of the switch port of the designated switch in the first time period, and obtaining a target number of bytes; The result of dividing the target number of bytes by the bandwidth rate of the switch port of the designated switch is determined as the normalized packet sending rate.
4. The method according to claim 2, characterized in that: The determining, according to the port data of the switch port of the designated switch in the first time period, a set of normalized port states of the switch port of the designated switch in the first time period includes: In a case where the set of normalized port states includes a normalized queue length, determining a queue length of the switch port of the designated switch in the first time period according to port data of the switch port of the designated switch in the first time period to obtain a target queue length; Based on the target queue length, a target value is determined from 1 to a specified value, wherein the target value is from 1 to the specified value such that 2 n ≥ the minimum value of n for which the target queue length is established, and the specified value is such that 2 n ≥ the minimum n value of the maximum queue length of the switch port of the designated switch; The difference between 1 and the target ratio is determined as the normalized queue length, wherein the target ratio is the ratio of the target value to the specified value.
5. The method according to claim 2, characterized in that: The determining, according to the port data of the switch port of the designated switch in the first time period, a set of normalized port states of the switch port of the designated switch in the first time period includes: In a case where the set of normalized port states includes a normalized ECN marking rate, determining, based on the port data of the switch port of the designated switch in the first time period, a number of data packets sent by the switch port of the designated switch in the first time period to obtain a first number of data packets, and determining a number of data packets ECN marked by the switch port of the designated switch in the first time period to obtain a second number of data packets; The result of dividing the second number of data packets by the first number of data packets is determined as the normalized ECN marking rate.
6. The method according to claim 1, characterized in that The preset decision network further includes: an input layer and a convolution layer, wherein the input layer is used to transfer the input port state features to the convolution layer, and the convolution layer is used to perform a convolution operation on the port state features transferred by the input layer through a weight connection parameter, so as to perform high-dimensional feature mapping on the port state features transferred by the input layer; The step of inputting the target port status feature into a preset decision network to obtain a target parameter value combination includes: Inputting the target port state feature into the input layer, so that the input layer passes the target port state feature to the convolution layer; Performing a convolution operation on the target port state feature through the weight connection parameter to perform high-dimensional feature mapping on the target port state feature to obtain a high-dimensional intermediate feature; Inputting the high-dimensional intermediate features into the designated advantage network to obtain a target advantage value output by the designated advantage network corresponding to each parameter value combination; The target parameter value combination is selected from the plurality of parameter value combinations based on the target advantage value corresponding to each parameter value combination.
7. The method according to claim 6, characterized in that The selecting the target parameter value combination from the multiple parameter value combinations based on the target advantage value corresponding to each parameter value combination includes: The parameter value combination with the largest corresponding target advantage value among the multiple parameter value combinations is determined as the target parameter value combination.
8. The method according to claim 6, characterized in that The designated duel network further includes a designated value network, the designated value network being used to evaluate a state value of a port state of a switch port of the designated switch, the state value of the port state of the switch port of the designated switch being an average expected return of the set of congestion control parameters taking each of the plurality of parameter value combinations under the port state of the switch port of the designated switch; The designated duel network is a duel deep Q network; The step of inputting the target port state feature into a preset decision network to obtain a target parameter value combination further includes: when the preset decision network includes the specified value network, inputting the high-dimensional intermediate feature into the specified value network to obtain a target state value output by the specified value network corresponding to the target port state feature; The method of selecting the target parameter value combination from the multiple parameter value combinations based on the target advantage value corresponding to each parameter value combination includes: determining a reference Q value corresponding to each parameter value combination according to the target state value and the target advantage value corresponding to each parameter value combination; and determining the parameter value combination with the largest corresponding reference Q value among the multiple parameter value combinations as the target parameter value combination.
9. The method according to claim 8, characterized in that The step of determining a reference Q value corresponding to each parameter value combination according to the target state value and the target advantage value corresponding to each parameter value combination includes: The sum of the target state value, the target advantage value corresponding to each parameter value combination, and the target reference advantage value is determined as a reference Q value corresponding to each parameter value combination, wherein the target reference advantage value is the maximum value among the target advantage values corresponding to each parameter value combination, or the average value of the target advantage values corresponding to each parameter value combination.
10. The method according to claim 1, characterized in that The designated duel network is a duel deep Q network; and the using the set of training state features to train the designated duel network to be trained to obtain the trained designated duel network includes: In the process of performing a round of training on the designated duel network, each training state feature in the set of training state features is sequentially used as a current training state feature to perform the following processing operations to obtain an updated designated duel network: Inputting the current training state feature into the designated duel network to obtain a current predicted Q value output by the designated duel network corresponding to each parameter value combination; Determine a current loss value corresponding to a specified loss function and the current training state feature according to a current predicted Q value corresponding to each parameter value combination and a current target Q value corresponding to each parameter value combination; Based on the current loss value corresponding to the current training state feature, the network parameters of the designated duel network are adjusted to reduce the current loss value corresponding to the current training state feature.
11. The method according to claim 10, characterized in that The step of inputting the current training state feature into the designated duel network to obtain the current predicted Q value output by the designated duel network corresponding to each parameter value combination includes: Inputting the current training state feature into the designated duel network, respectively obtaining the current state value output by the designated value network and the current advantage value corresponding to each parameter value combination output by the designated advantage network; The sum of the current state value, the current advantage value corresponding to each parameter value combination, and the current reference advantage value is determined as the current predicted Q value corresponding to each parameter value combination, wherein the current reference advantage value is the maximum value among the current advantage values corresponding to each parameter value combination, or the average value of the current advantage values corresponding to each parameter value combination.
12. The method according to claim 10, characterized in that Before determining the current loss value corresponding to the specified loss function and the current training state feature according to the current predicted Q value corresponding to each parameter value combination and the current target Q value corresponding to each parameter value combination, the method further includes: The current target Q value corresponding to each parameter value is determined by adding the current reward value corresponding to each parameter value combination to the discount coefficient and multiplying the result by the next predicted Q value corresponding to each parameter value combination; Among them, the current reward value corresponding to each parameter value combination is the reward value corresponding to each parameter value under the current port state corresponding to the current training state feature, the next predicted Q value corresponding to each parameter value combination is the predicted Q value corresponding to each parameter value combination output by the designated duel network after the next port state feature is input into the designated duel network, the next port state feature is the state feature corresponding to the next port data, and the next port data is the port data of the switch port of the reference switch in the second time period after the set of congestion control parameters are configured according to the parameter value combination with the largest corresponding current predicted Q value under the current port state.
13. The method according to claim 12, characterized in that Before determining the result of adding the current reward value corresponding to each parameter value combination to the discount coefficient and multiplying the next predicted Q value corresponding to each parameter value combination as the current target Q value corresponding to each parameter value, the method further includes: Determine a weighted sum of a reference throughput corresponding to each parameter value combination and a reference queue length corresponding to each parameter value combination as a current reward value corresponding to each parameter value combination; The reference throughput corresponding to each parameter value combination is the throughput of the switch port of the reference switch in the third time period after the set of congestion control parameters are configured according to each parameter value combination in the current port state, and the reference queue length corresponding to each parameter value combination is the queue length of the switch port of the reference switch in the third time period after the set of congestion control parameters are configured according to each parameter value combination in the current port state.
14. The method according to claim 1, characterized in that Before acquiring the state feature corresponding to the port data of the switch port of the designated switch in the first time period, the method further includes: Collecting chip bottom layer data of the switch port of the current switch in the first time period through a software development kit to obtain port data of the switch port of the current switch in the first time period, wherein the designated switch is the current switch; or In response to receiving port status characteristics transmitted by a group of upstream switches, chip bottom layer data of the switch port of the current switch within the first time period is collected through a software development kit to obtain port data of the switch port of the current switch within the first time period, wherein the port status characteristics transmitted by the group of upstream switches are status characteristics corresponding to the port data of the switch port of the group of upstream switches within the first time period, and the designated switches include the group of upstream switches and the current switch.
15. The method according to any one of claims 1 to 14, characterized in that The set of congestion control parameters includes a lower limit of an ECN marking threshold, an upper limit of an ECN marking threshold, and a maximum value of an ECN marking probability.
16. A congestion control device, characterized in that: include: A first acquisition unit, configured to acquire a state feature corresponding to port data of a switch port of a designated switch within a first time period, and obtain a target port state feature; an input unit, configured to input the target port state feature into a preset decision network to obtain a target parameter value combination, wherein the target parameter value combination is one of a plurality of parameter value combinations of a set of congestion control parameters corresponding to the switch port of the specified switch, the preset decision network includes a specified advantage network of a specified duel network, the specified advantage network is configured to evaluate an advantage value corresponding to each parameter value combination in the plurality of parameter value combinations under a port state of the switch port of the specified switch, the advantage value corresponding to each parameter value combination in the target port state corresponding to the target port state feature output by the specified advantage network is a target advantage value, and the target parameter value combination is a parameter value combination selected from the plurality of parameter value combinations based on the target advantage value corresponding to each parameter value combination; an execution unit, configured to configure the set of congestion control parameters according to the target parameter value combination, and perform congestion control according to the configured set of congestion control parameters; Wherein, the designated duel network also includes a designated value network, and the designated value network is used to evaluate the state value of the port state of the switch port of the designated switch, and the state value of the port state of the switch port of the designated switch is the average expected return of each parameter value combination of the plurality of parameter value combinations taken by the set of congestion control parameters under the port state of the switch port of the designated switch; the device also includes: a second acquisition unit, which is used to acquire a set of training state features after inputting the target port state feature into the preset decision network, wherein each training state feature in the set of training state features is a state feature corresponding to the port data of the switch port of a reference switch within a historical time period; a training unit, which is used to train the designated duel network to be trained using the set of training state features to obtain the trained designated duel network; an updating unit, which is used to update the network parameters of the designated advantage network in the preset decision network using the network parameters of the designated advantage network in the trained designated duel network.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 15 when executed by a processor.
18. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.
19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.
