Method and system for multi-modal network bandwidth adaptive allocation based on meta-reinforcement learning

Abstract

The application discloses a kind of multi-modal network bandwidth self-adaptive allocation method and system based on meta reinforcement learning.The method is for the competition problem generated when sharing limited network bandwidth resources in the same network coexisting multiple heterogeneous network modalities, design a centralized scheduling agent, first by periodically collecting the real-time bandwidth demand, queue state, delay sensitivity and priority information of each network modal, and simultaneously simulate the introduction of traffic burst, link state change, new network modal access and other network dynamic events, then the agent is based on meta reinforcement learning framework in a small number of sample tasks Rapid adaptation, realize strategy generalization through cross-task meta update, finally output the optimal or near-optimal bandwidth allocation ratio of each business modal.The method does not need manual parameter adjustment, can significantly improve the bandwidth resource utilization and key business service continuity in multiple network modal scenarios, and has the ability of rapid self-healing to traffic burst or network topology change.

Description

Technical Field

[0001] This invention relates to the field of communication network resource management technology, and more particularly to a multimodal network bandwidth adaptive allocation method and system based on meta-reinforcement learning. Background Technology

[0002] In multimodal networks, multiple heterogeneous network technologies operate simultaneously, each constituting an independent network mode. These network modes share underlying physical bandwidth resources. Without dynamic coordination mechanisms and generalization capabilities, competition for bandwidth resources among network modes can lead to a decline in the quality of service for some network modes. When new network modes are accessed or the topology changes, traditional methods typically rely on manual readjustment of resource allocation strategies, resulting in response delays and high adjustment costs. This further degrades the performance of critical business modes and restricts the overall effectiveness of multiple network modes.

[0003] While some existing solutions have introduced scheduling strategies based on static priority or single-modal reinforcement learning, which can improve bandwidth utilization and service quality of some services to a certain extent, they still have problems such as insufficient generalization ability, slow adaptation to new network modalities, and reliance on manual parameter adjustment. It is difficult to maintain the optimal scheduling effect in a dynamic and complex network modal environment for a long time. Summary of the Invention

[0004] The purpose of this application is to provide a reinforcement learning agent training method, a network modality bandwidth resource scheduling method and apparatus to solve the problems in the prior art where static allocation strategies cannot adapt to dynamic network modality changes, new network modality access response lags, and sudden traffic leads to a decline in the quality of service of critical network modalities.

[0005] This invention provides a method for adaptive bandwidth allocation in multimodal networks based on meta-reinforcement learning, comprising the following steps: S1. Construct a deep neural network for global network feature states, actions, and training meta-reinforcement learning, which includes a multimodal feature encoding network, an integrated fusion network, a policy network, and a value network; S2. Set the training hyperparameters, including the number of training rounds, the number of fast adaptation steps in the inner loop, the learning rate in the inner loop, the learning rate in the outer loop, the meta-batch size, the outer loop parameters, and the inner loop parameters; S3. Input the currently collected global network feature status and corresponding disturbance task labels into the meta-reinforcement learning agent; the agent first outputs bandwidth allocation action according to the task-specific strategy, and then sends and executes it through the SDN switch; then monitors the network status and reward feedback after the switch executes the interaction action, and stores the interaction experience in the buffer of the corresponding task. S4. In the rapid adaptation phase of the inner loop, after the data in the task buffer reaches the predetermined batch size and several steps of interaction are completed, the network state and reward feedback in the task buffer are used to fine-tune the agent's policy network parameters by the rapid adaptation steps set in step S2, so as to obtain the dedicated policy network parameters for the task. S5. In the outer loop meta-update phase, for each task in the meta-batch, based on the task-specific policy network parameters obtained after the rapid adaptation in step S4, the adaptation parameters of all tasks are compared with the meta-parameters, and the globally shared meta-parameters are updated by gradient descent MAML. S6: Repeat steps S3–S5 until the meta-parameters converge, the bandwidth allocation of each network mode is stable, and there is no overload at the network egress end, thereby completing the continuous adaptive optimization of multiple network modes.

[0006] Further, step S1 specifically involves: the global network characteristic state includes the number of active connections, the total number of transmitted packets, and the average size of a single packet for each network modality and its subordinate flows; for each data flow, the average round-trip time, the number of data packets within the flow, the cumulative number of bytes and the average packet length during the flow duration are periodically collected; the queue backlog of each network modality and its subordinate flows, changes in egress bandwidth utilization, and records of historical abnormal behaviors are also collected; its action space is defined as: the bandwidth allocation ratio output by the agent for each network modality m. ,satisfy , where M is the number of network modes.

[0007] Furthermore, step S3 specifically includes the following sub-steps: S31: The neural network adopts a unified backbone neural network structure. First, it collects the multi-dimensional features of each modal network at the current moment, and then performs feature representation and splicing through the embedding layer to form a global network state vector. S32: Input the concatenated global features into the modality coding layer. Its temporal features are encoded by a multi-head Transformer, and its static features are processed by an MLP. The output is a representation vector of uniform length for each network modality. S33: Input the encoding results of all network modalities into the self-attention mechanism to achieve global feature fusion, and finally output the global network feature state; S34: Input the fused global features into a multilayer MLP and output the bandwidth allocation coefficients for each network mode; perform softmax normalization on the bandwidth allocation coefficients to obtain the final allocation ratio for each network mode; S35: The meta-reinforcement learning agent takes the current global network feature state and the corresponding perturbation task label as input, outputs the bandwidth allocation action according to the final allocation ratio generated in step S34, and sends the final decision through the SDN switch to execute the bandwidth allocation. S36: The agent interacts with the environment in several steps, and each interaction is recorded as a quintuple, including the pre-action state, the perturbation task label, the selected action, the post-action state, and the reward value; the experience data of this interaction is stored in the task buffer corresponding to the current task for subsequent policy network to quickly adapt and update.

[0008] Furthermore, step S35 specifically involves: if a disturbance task exists, then at each step, the current network state is input. and Disturbance Task Tags Among them, the disturbance label If the value is not 0, and there is no perturbation task, the perturbation label is... The value is 0, and the meta-reinforcement learning agent follows the strategy. Output Action The actions described therein It is the sum of the action vector selected in the global state corresponding to the input and the noise, used to adjust the bandwidth of the network mode to schedule the communication resources that each network mode can occupy. Its physical meaning is the proportion of each flow of each network mode to the exit; assuming that... Indicates the first One network mode, Indicates the first If the number of streams in a network modality during a time interval t is given, then the dimension of the output action space is: Then the first In each time slot, only Each element has a corresponding stream, so its value is between 0.1 and 1, while the other elements have a value of 0; actions are performed using an SDN switch. Post-feedback status and rewards Record data ( ) to task buffer The reward function is defined as follows: ; Where P represents the number of network modes. Representing the The weights of each network mode. This represents the first time slot t. Number of flows per network mode Representing network modes The i-th stream in time slot The demand for traffic at that time Indicates the first The i-th flow of a network mode is in The percentage of data that actually successfully reaches the server at any given moment. This indicates the maximum total traffic the system can handle; exceeding this limit results in overload. LOAD represents the sum of actual flows arriving from all network modes. The specific formula is as follows: The above reward function is designed to meet the communication transmission requirements of different service types in multiple network modes.

[0009] Furthermore, the inner loop rapid adaptation phase in step S4 involves rapidly updating the policy network, specifically including: The latest interaction data, including the pre-action state, selected action, post-action state, and reward value, is written into the task buffer. In the next inner loop or subsequent outer loop, historical samples are extracted from the task buffer at a fixed ratio and mixed with the current task samples to balance rapid adaptation and cross-task experience accumulation. Input the global network feature state before all actions in the buffer into the policy network, calculate the action distribution or allocation probability in each state, combine the actual reward signal of each data and the discount factor to calculate the discounted reward of each state, and calculate the loss function of each sample based on the current policy output and the actual executed action. The total loss under the current perturbation task is obtained by summing up the losses of all samples, and the policy network parameters are optimized through backpropagation to make them more suitable for the current task environment.

[0010] Furthermore, the calculation of the cumulative discount reward is specifically as follows: The quintuples from multiple sampling steps can be pieced together to form a complete interaction record from start to finish; for each step of the interaction, the cumulative reward of the discount needs to be calculated. Used to evaluate the trade-off between current rewards and future rewards: ; in The discount factor has a value range of (0, 1). The larger the value, the more the agent values ​​future rewards; conversely, the more it values ​​current rewards. Discounted cumulative rewards transform current reward signals into a global evaluation of future gains, enabling meta-reinforcement learning agents to adapt to complex environments and weigh short-term and long-term goals.

[0011] Furthermore, the total loss under the current perturbation task is obtained by accumulating all the mixed sample losses, and the policy network parameters are optimized through backpropagation; specifically, for each perturbation task, historical data is first sampled from the experience pool when updating parameters in the inner loop. Then, the current perturbation task data is mixed. Obtain a mixed dataset ,in =K+n, where K represents the amount of newly sampled data in this task, i.e., the inner loop sampling length, and n represents the amount of historical experience pool samples, i.e., the number of historical trajectories used for mixed sampling. Represents the global network characteristic state. This represents a disturbance label. Represents the reward value; calculates the mixed loss, i.e., for the mixed set The policy gradient loss of each sample is averaged to obtain the inner loop average loss of this perturbation task, and its expression is as follows: ;in For the discount rewards of each item in the mixed data, historical data can be used for reward normalization. Represents the current global network feature state. Represents the state The action selected by the next strategy This represents the reward after performing the action. This represents the next state after the action is performed. Finally, the policy network parameters in the inner loop are updated: ,in This is the learning rate for the inner loop.

[0012] Specifically, in step S5, the globally shared meta-parameters are updated using the gradient descent method MAML. The tasks completed in multiple inner loops are archived, that is, the policy network parameters obtained after fine-tuning in the inner loop of each perturbation task are compared with the initial outer loop parameters. The fine-tuned parameters of all tasks are then differiated from the meta-parameters to calculate the parameter update direction. A weighted average is used, and the meta-policy parameters are uniformly adjusted using the outer loop learning rate. The updated meta-parameters are used as the starting point for a new round of inner loops. Since the outer loop update is based on the gradient descent method MAML, within the MAML framework, gradients are calculated for each task completed in multiple inner loops, accumulated in the outer loop, and then updated all at once. Specifically: for one round of the outer loop, i rounds of inner loops have already been performed, N tasks have been sampled, and the fine-tuned parameters of each inner loop are obtained. Then the outer loop parameters are aggregated, as shown in the following expression: ; Where β is the outer loop learning rate.

[0013] This invention also discloses a multimodal network bandwidth adaptive allocation system based on meta-reinforcement learning, comprising the following modules: (1) State collector: Actively and periodically acquires network feature data of each network mode and its subordinate flow, and performs feature normalization and preprocessing; (2) Dynamic event injection module: used to simulate and inject disturbance tasks such as traffic bursts, link quality changes, and new network mode access, and perform adaptive scheduling; (3) Intelligent agent module: Based on the MAML and other meta-reinforcement learning algorithms, it integrates multiple network modal features to realize bandwidth allocation strategy reasoning, rapid adaptation of the inner loop and aggregation of meta-parameters in the outer loop; (4) Quota execution module: Translates the bandwidth allocation results of the agent's decision into control commands that can be recognized by the underlying network devices; (5) Performance monitoring module: end-to-end real-time monitoring of key indicators such as resource utilization, latency satisfaction rate and bandwidth allocation jitter of each network mode, and generating reward feedback for intelligent agent optimization decision-making; (6) Inner loop update module: Based on the rewards and status in the current perturbation task buffer, quickly and adaptively fine-tune the policy network parameters; (7) Outer loop update module: collects the parameter results after fine-tuning of the inner loop of multiple perturbation tasks and updates the meta-strategy parameters in a unified manner; (8) Repeat module: Used to repeat the above process until the bandwidth allocation of each network mode meets the communication quality and there is no overload at the egress end.

[0014] Furthermore, it also includes performance self-healing and automatic adjustment for fault detection, specifically including the following modules: Performance monitoring module: used to collect the operating status of various service modes and links in real time, and monitor core indicators, including bandwidth utilization, latency and packet loss; Adaptive optimization decision module: Used to diagnose the detection results of the performance monitoring module and dynamically generate adjustment strategies; Strategy execution module: Used to issue adjustment instructions based on the adjustment strategy generated by the adaptive optimization decision module, and continuously monitor the effect.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) Rapid adaptation to network changes. By explicitly injecting sudden events such as traffic bursts, link quality changes, and the access of new network modes during training, combined with rapid fine-tuning based on task buffers in the inner loop and meta-parameter aggregation in the outer loop, the strategy can quickly recover and maintain the stability of the allocation results in dynamic environments, reducing the cost of manual intervention and repeated parameter tuning; 2) Adaptive bandwidth allocation based on network modality. The system periodically collects key state features of each network modality and its subordinate flows. The agent integrates multimodal features to make bandwidth allocation decisions, thereby forming the optimal bandwidth configuration for different modalities. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. Figure 1This is an overall flowchart illustrating a multimodal network bandwidth adaptive allocation method based on meta-reinforcement learning, according to an exemplary embodiment. Figure 2 This is a diagram illustrating the neural network decision-making process for adaptive bandwidth allocation in a multimodal network according to an exemplary embodiment; Figure 3 This is a flowchart illustrating the inner and outer loops of a meta-reinforcement learning agent according to an exemplary embodiment; Figure 4 This is a functional flowchart illustrating a multimodal network bandwidth adaptive allocation system according to an exemplary embodiment; Figure 5 This is a flowchart illustrating a dynamic network perturbation task construction process according to an exemplary embodiment; Figure 6 This is a flowchart illustrating fault detection and automatic adjustment according to an exemplary embodiment; Figure 7 This is a schematic diagram of an electronic device structure suitable for edge deployment, according to an exemplary embodiment. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0018] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0019] The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0021] Example 1: Figure 1This is a flowchart illustrating a method for training a meta-reinforcement learning agent in a multimodal network, based on an exemplary embodiment. The method, applied to a meta-reinforcement learning agent, may include the following steps: S1: Intelligent Agent Architecture Construction and Initialization: Construct the deep neural network required for global network feature state, actions, and training element reinforcement learning, including a multimodal feature encoding network, an integrated fusion network, a policy network, and a value network; used to perceive the global network state and generate bandwidth allocation actions; each network adopts a fully connected neural network and recurrent neural network architecture, and the initial values ​​of the parameters are assigned through random distribution; S2: Meta-training hyperparameter settings: Set the number of training rounds K, and the number of fast adaptation steps in the inner loop. Inner loop learning rate and outer loop learning rate Meta-batch size and outer loop parameters Inner loop parameters And other hyperparameters; S3: Multi-task parallel interaction and data collection: For each task perturbation, the current global network feature state and the corresponding perturbation task label are input into the meta-reinforcement learning agent; the agent first outputs bandwidth allocation actions according to its current task-specific policy, and distributes and executes the actions through the SDN switch; then it monitors the network state and reward feedback after the switch executes the actions, stores the experience data of this interaction (state before action, selected action, state after action and reward value) into the task buffer corresponding to the task, and quickly adapts and updates the policy network. S4: Inner Loop Rapid Adaptation: After completing several interaction steps, the agent's policy network parameters are rapidly adapted and fine-tuned based on the network status and reward feedback in the task buffer to obtain dedicated parameters adapted to the current task. S5: Outer loop meta-parameter aggregation and update: Collect parameter results obtained from multiple perturbation tasks after fine-tuning in the inner loop, compare the adaptive parameters of all tasks with the meta-parameters, and perform unified aggregation and update of the meta-parameters using gradient descent MAML. S6: Iterative training and performance convergence: Repeat steps S3–S5 until the meta-parameters converge, the bandwidth allocation of each network mode tends to stabilize under the premise of ensuring communication quality, and the network egress end no longer experiences overload, thereby completing the continuous adaptive optimization of the multimodal network.

[0022] Specifically, the global network feature state constructed in step 1 This includes the number of active connections, total number of transmitted packets, and average size of individual packets for each network mode and its subordinate flows. For each data flow, the average round-trip time, number of packets within the flow, cumulative number of bytes over the flow duration, and average packet length are collected periodically. Queue backlog and egress bandwidth utilization changes for each network mode and its subordinate flows are also recorded, along with historical anomalies such as traffic surges and new network mode access. These characteristics constitute the current time interval. The state of global variables. (Used) Represents the t-th Global network features within seconds, the action space is defined as: the bandwidth allocation ratio of each network mode m output by the meta-reinforcement learning agent. ,satisfy: , where M is the number of network modes.

[0023] Specifically, such as Figure 2 As shown, step S3 specifically includes the following sub-steps: S31: Multidimensional Feature Acquisition The neural network adopts a unified backbone neural network structure. First, it collects multi-dimensional features of each modality network at the current moment, specifically including dynamic traffic features: traffic volume, latency, queue length, queue backlog, and number of active connections; historical service features: historical abnormal behavior and total number of transmitted packets; and data stream-level features: for each data stream, it collects its average round-trip latency, number of packets within the stream, cumulative number of bytes over the stream duration, and average packet length. All features are initially represented and concatenated through an embedding layer to form an initial global network state vector.

[0024] S32: Modal Feature Encoding The concatenated global features are input into the modality coding layer. Its temporal features (such as traffic and latency) are encoded by a multi-head Transformer to capture their temporal dependencies, while its static features (such as service type) are processed by a multilayer perceptron (MLP). The output is a uniform-length representation vector for all network modalities.

[0025] S33: Feature Fusion and Interaction The encoding results of all network modalities are input into the self-attention mechanism; the self-attention mechanism is responsible for capturing the inherent dependencies and cooperation relationships between different network modalities (such as traffic, latency, queues), realizing deep feature fusion, and finally outputting a fused global network feature state rich in contextual information.

[0026] S34: Bandwidth Allocation Decision Generation The fused global features are input into a multi-layer MLP; the MLP outputs the initial bandwidth allocation coefficients for each network mode; these coefficients are then Softmax normalized to obtain the final bandwidth allocation ratio for each network mode.

[0027] S35: Decision Implementation and Feedback The agent takes the current global network characteristic state and the corresponding disturbance task label as input, and outputs a bandwidth allocation action based on its current task-specific policy (i.e., the allocation ratio generated in step S34). This action is issued and executed through the SDN switch, completing one bandwidth allocation; the system then monitors the new network state and reward feedback after the action is executed. Specifically: If there are perturbation tasks, input the current network state at each step. and Disturbance Task Tags Among them, the disturbance label If the value is not 0, and there is no perturbation task, the perturbation label is... The value is 0, and the meta-reinforcement learning agent follows the strategy. Output Action The actions described therein It is the sum of the action vector selected in the global state corresponding to the input and the noise, used to adjust the bandwidth of the network mode to schedule the communication resources that each network mode can occupy. Its physical meaning is the proportion of each flow of each network mode to the exit. Assuming that... Indicates the first One network mode, Indicates the first Each network mode in The number of streams in the time interval determines the dimension of the output action space. Since the action value is defined as a bandwidth allocation ratio and normalized using softmax, therefore... Actions with flowing elements in a time slot have values ​​between [0,1], while those without flowing elements have values ​​of 0. Actions are executed using an SDN switch. Post-feedback status and rewards Record data ( , , , , ) to task buffer .

[0028] In one round of training, the original reinforcement learning AI performs step S3 k times. During this process, the network parameters are not updated. The designed reward function is defined as follows: ; Where P represents the number of network modes. Representing the The weights of each network mode. This represents the first time slot t. Number of flows per network mode Representing network modes The i-th stream in time slot The demand for traffic at that time (i,t) represents the i-th The i-th flow of a network mode is in The percentage of data that actually successfully reaches the server at any given moment. This represents the maximum total traffic that can be handled; exceeding this limit results in overload. LOAD represents the sum of the actual flows arriving from all network modes. The specific formula is as follows: .

[0029] Through several steps of interaction between the agent and the environment, each interaction is recorded as a quintuple (...). , , , , );in This is the state before the action. Label for the perturbation task. For the selected action, This is the state after the action. This is the reward value. The experience data from this interaction is stored in the task buffer corresponding to the current task for subsequent rapid updates and adaptation by the policy network.

[0030] After each interaction, the experience data ( , , , , Simultaneously, a global experience pool is created. During the inner loop (intra-task update) or subsequent outer loop (cross-task update), historical samples are drawn from the global experience pool at a fixed ratio and mixed with new samples in the current task buffer. This balances rapid adaptation with cross-task experience accumulation. This helps the agent utilize general knowledge learned from previous tasks to accelerate the adaptation process on new tasks and avoid overfitting to the local dynamics of the current task.

[0031] The aforementioned reward function design enables dynamic allocation and efficient utilization of bandwidth resources based on the communication transmission needs of different service types across multiple network modes, preventing global resource allocation imbalances caused by abnormal demands from individual network modes or network congestion. During bandwidth resource scheduling, the effective traffic from each network mode to the server is statistically analyzed, and the actual transmission volume is weighted using the service weight coefficient of each network mode to measure the overall communication quality compliance level.

[0032] When the transmission demand of a certain network mode increases significantly or a sudden event occurs in a particular service, even if that network mode accounts for a large proportion, the reward function will drive subsequent actions to adjust towards services with balanced multi-network modes or high priority, so as to make full use of the remaining bandwidth of the network.

[0033] If multiple service modalities simultaneously experience abnormally high demand, the reward value will be significantly reduced because the overall transmission load exceeds the threshold, thereby inhibiting the agent from making decisions to over-allocate bandwidth and effectively preventing network overload. Furthermore, it will prioritize the communication needs of high-weight critical services while guiding low-weight modal traffic to automatically yield when resources are scarce.

[0034] When sudden anomalies or extreme high loads occur, the negative reward mechanism will guide the agent to reduce the probability of high-risk actions and reduce the risk of network overload.

[0035] Therefore, the design of the reward function can ensure the stable operation of the network and intelligently adjust the bandwidth allocation scheme according to the dynamic evolution of the transmission requirements of each network mode.

[0036] After completing several interactive steps, based on all reward values ​​in the task buffer and the state before the action, the parameters of the intelligent internal loop and external loop are updated; steps S4 and S5 are the rapid adaptation stage of the internal loop and the meta-parameter update stage of the external loop of the present invention. Figure 3 As shown, the details are as follows: Step S4 specifically includes the following sub-steps: S41. For all pre-action states and selected actions, use the current policy network to calculate the probability distribution, action probability density, and the difference between the discounted reward and state value estimate for each sample. S42. Take a weighted average of the loss functions of all samples to form the target loss of this inner loop. Finally, use the optimizer to adjust the network parameters according to the inner loop learning rate to complete this downsizing.

[0037] The inner loop fast adaptation phase in step S4 involves rapidly updating the policy network, specifically including: Input the global network feature state before all actions in the buffer into the policy network, calculate the action distribution or allocation probability in each state, combine the actual reward signal of each data and the discount factor to calculate the discounted reward of each state, and calculate the loss function of each sample based on the current policy output and the actual executed action. The calculation of the cumulative discount reward is as follows: The quintuples from the multi-step sampling can be pieced together to form a complete interaction record episode from start to finish; for each step of the interaction, the cumulative reward of the discount needs to be calculated. Used to evaluate the trade-off between current rewards and future rewards: ; in The discount factor has a value range of (0, 1). The larger the value, the more the agent values ​​future rewards; conversely, the more it values ​​current rewards. Discounted cumulative rewards transform current reward signals into a global evaluation of future gains, enabling meta-reinforcement learning agents to adapt to complex environments and weigh short-term and long-term goals.

[0038] The total loss under the current perturbation task is obtained by summing up the losses of all samples, and the policy network parameters are optimized through backpropagation to make them more suitable for the current task environment.

[0039] Step S5 specifically includes the following sub-steps: S51. Archive the policy parameters and initial outer loop parameters obtained after fine-tuning the inner loop of each disturbance task, and perform differential calculation on all task fine-tuning parameters and meta-parameters to calculate the parameter update direction. S52. Using a weighted average and the learning rate of the outer loop, the meta-policy parameters are uniformly adjusted to improve the global generalization ability. The updated meta-parameters are then used as the starting point for a new round of inner loop training.

[0040] Since the outer loop update is based on the gradient descent method MAML, that is, in the MAML framework, the gradients of the tasks completed by multiple inner loops are calculated separately, and then accumulated in the outer loop before updating the meta-parameters all at once.

[0041] Specifically, for each disturbance task, when updating parameters in the inner loop, historical data is first sampled from the experience pool. Then, the current perturbation task data is mixed. Obtain a mixed dataset ,in =K+n, where K represents the amount of newly sampled data in this task, i.e., the inner loop sampling length, and n represents the amount of historical experience pool samples, i.e., the number of historical trajectories used for mixed sampling. Represents the global network characteristic state. This represents a disturbance label. Represents the reward value. Calculate the mixture loss, i.e., for the mixture set... The average policy gradient loss of each sample is taken to obtain the inner loop average loss of this perturbation task: ;in For the discount rewards of each item in the mixed data, historical data can be used for reward normalization. Represents the current global network feature state. Represents the state The action selected by the next strategy This represents the reward after performing the action. It represents the next state after the action is performed.

[0042] Finally, update the policy network parameters in the inner loop: ,in This is the learning rate for the inner loop.

[0043] After several iterations of step S6, when the resource utilization, latency satisfaction rate, and jitter cost indicators of the meta-reinforcement learning agent in the verification scenario all reach the preset thresholds, it is considered to have converged and the inner and outer loops are stopped. The final meta-reinforcement learning agent is then exported and deployed to the online scheduler. After deployment, it is periodically or as needed to trigger self-fine-tuning based on the latest running logs to maintain long-term performance.

[0044] Figure 4 This is a functional flowchart illustrating a multimodal network bandwidth adaptive allocation system according to an exemplary embodiment. The system utilizes a meta-reinforcement learning agent and includes the following modules: Status acquisition module: Actively and periodically acquires network characteristic data of each network mode and its subordinate flows, including active connection number, traffic, queue status, latency sensitivity, priority, etc., and performs feature normalization and preprocessing; Dynamic event injection module: used to simulate and inject disturbance tasks such as traffic bursts, link quality changes, and new network mode access, to promote adaptive scheduling in complex dynamic environments; The intelligent agent module is based on meta-reinforcement learning algorithms such as MAML, and integrates features of multiple network modalities to achieve bandwidth allocation strategy reasoning, rapid adaptation of the inner loop, and aggregation of meta-parameters in the outer loop. Quota execution module: Translates the bandwidth allocation results decided by the agent into control commands that can be recognized by the underlying network devices; Performance monitoring module: Real-time end-to-end monitoring of key indicators such as resource utilization, latency satisfaction rate and bandwidth allocation jitter of each network mode, generating reward feedback for intelligent agent optimization decisions; Inner loop update module: Based on the rewards and status in the current perturbation task experience pool, quickly and adaptively fine-tune the policy network parameters; Outer loop update module: Collects parameter results after fine-tuning of the inner loop of multiple perturbation tasks, updates the meta-policy parameters in a unified manner, and improves the overall generalization ability of the model; Repeat module: Used to repeat the above process until the bandwidth allocation of each network mode meets the communication quality and there is no overload at the egress end.

[0045] Example 2: Figure 5The diagram shows a flowchart for constructing a dynamic network disturbance task, involving the disturbance generation process such as new network mode access, link capacity changes, and priority adjustments; specifically, it includes the following steps: (a) Based on a preset strategy, dynamic disturbance tasks are automatically generated, including but not limited to typical scenarios such as traffic bursts, access of new network modes, changes in link capacity, or adjustments to service priorities; (b) For the perturbation task, the corresponding perturbation event is injected into the network environment in real time according to the set parameters, and the perturbation type, parameters and effective time period are marked to drive the adaptive scheduling of the meta-reinforcement learning agent. (c) Monitor the network state changes after the injection of disturbance events in real time, collect relevant performance indicators, and provide data support for subsequent agent training and reward function calculation; Specifically, the disturbance task parameters include the type, bandwidth requirement, access time and priority of the newly accessed network mode; the bandwidth adjustment range, affected links and duration of the link capacity change disturbance; the target mode identifier, new priority value and change duration of the priority adjustment disturbance, etc., which are used to accurately control the traffic burst type and scope of the disturbance scenario.

[0046] The disturbance effect monitoring step collects in real time indicators including bandwidth utilization of each network mode, end-to-end latency, packet loss rate, queue length, latency jitter and service satisfaction rate, and stores the above data in association with the corresponding disturbance event labels for agent training and performance evaluation.

[0047] Figure 6 This is a flowchart illustrating the self-healing and automatic adjustment of multimodal network system performance and fault detection according to an exemplary embodiment. The flowchart includes the following components: Performance monitoring module: used to collect the operating status of various service modes and links in real time, and monitor core indicators such as bandwidth utilization, latency, and packet loss; Adaptive optimization decision module: Used to diagnose the detection results of the performance monitoring module and dynamically generate adjustment strategies; Strategy execution module: Used to issue adjustment instructions based on the adjustment strategy generated by the adaptive optimization decision module, and continuously monitor the effect.

[0048] As can be seen from the above embodiments, this invention is based on the idea of ​​meta-reinforcement learning algorithm. First, a global network feature state vector suitable for multimodal networks is constructed, and bandwidth allocation actions and immediate reward functions are defined on this basis. Then, through rapid adaptation in the inner loop, the agent interacts online and fine-tunes policy parameters in a single network task. Then, through meta-update in the outer loop, multi-task feedback is integrated to achieve cross-task policy generalization. Finally, the agent can output the optimal bandwidth allocation action according to changes in network state and reward, so that the multimodal network resource allocation can meet the needs of each modality service and ensure the overall network operation performance. This method does not require retraining for every network scenario. For typical scenarios of network mutations (such as traffic bursts, link quality jitter, and new network mode access; where traffic bursts are tested by injecting peak packet traffic into the basic load to examine short-term capacity; link quality jitter is tested by simulating changes in link bandwidth upper and lower limits and packet loss rate to verify model robustness; new network mode access is tested by accessing new network modes to test the adaptability to new network mode types and traffic characteristics); this invention still has fast adaptation and good generalization capabilities, which has strong practical significance for promoting the intelligent management and control of multimodal networks.

[0049] Furthermore, the intelligent agent module further supports GPU parallel training, model version management, and online rollback functions to improve training efficiency and reliability.

[0050] Furthermore, the quota execution module communicates securely with the underlying network device through different protocols and supports atomic updates to token bucket parameters and queue weights. Regarding the apparatus in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0051] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0052] like Figure 7 As shown in the diagram, this embodiment also provides a schematic diagram of an electronic device structure suitable for edge deployment. The electronic device includes: Processor: Used to load and execute instructions for the multimodal network bandwidth adaptive allocation method; Memory: Used for temporary storage of intermediate data, network status characteristics, etc., generated by the processor during the execution of a method; Non-volatile memory: used to store pre-deployed multimodal network bandwidth adaptive allocation algorithm programs, network status data, policy model parameters, and historical logs, etc. Communication interface: used to enable data interaction, task assignment and execution feedback between the device and other nodes or control center in the network; Internal bus: Used to connect the above components and enable the transmission of data and control signals.

[0053] In this embodiment, the electronic device can be deployed as an edge node in a multi-network modal environment. By calling the allocation method program stored in non-volatile memory, the processor can complete operations such as state acquisition, event injection, agent decision-making, policy distribution, and performance monitoring in real time, thereby realizing dynamic adaptive allocation of bandwidth for access edge service flows and significantly improving the resource utilization and service elasticity of the edge network.

[0054] Furthermore, the electronic device supports multiple communication protocols, making it compatible with the access and management needs of heterogeneous network modes.

[0055] The device's hardware resources can be flexibly expanded according to the actual scenario. For example, it can support high-performance GPU / TPU computing modules or be equipped with low-power processing units to meet the deployment needs of multi-network modal scenarios of different scales and complexities.

[0056] Those skilled in the art, after reading this specification, can appropriately expand or trim the functional modules of the electronic device according to actual needs without exceeding the essential content of this invention.

[0057] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, the program being loaded and run by a processor of an electronic device to implement all or part of the functions of a multi-network modal system performance self-healing and intelligent fault handling device.

[0058] Specifically, the computer-readable storage medium can be any type of storage medium commonly found in the art, including but not limited to hard disks, solid-state drives, USB flash drives, SD cards, smart memory cards, and flash cards. The program instructions in the storage medium enable the electronic device to automatically execute functional modules such as performance monitoring, anomaly detection, diagnostic analysis, adaptive optimization decision-making, strategy execution, and closed-loop verification according to the process described in Embodiment 3 above, thereby achieving rapid fault detection, diagnosis, and self-healing in various network modal environments.

[0059] Furthermore, the computer-readable storage medium is applicable to various computing device forms, such as edge computing nodes, data center servers, and network switching equipment. The core functions of this invention can be easily implemented simply by loading the storage medium onto the corresponding device and running the program therein, without requiring any additional hardware modifications.

[0060] Those skilled in the art can flexibly expand the storage structure, program calling flow, and hardware adaptation method of the storage medium according to actual needs, without having to make any creative effort, to achieve the purpose of the present invention.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0062] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for adaptive bandwidth allocation in multimodal networks based on meta-reinforcement learning, characterized in that, Includes the following steps: S1. Construct a deep neural network for global network feature states, actions, and reinforcement learning of training elements, including a multimodal feature encoding network, an integrated fusion network, a policy network, and a value network; specifically, the global network feature states include the number of active connections, the total number of transmitted packets, and the average size of a single packet for each network modality and its subordinate flows; for each data flow, periodically collect its average round-trip delay, the number of data packets within the flow, the cumulative number of bytes and the average packet length during the flow duration, the queue backlog of each network modality and its subordinate flows, changes in outgoing bandwidth utilization, and records of historical abnormal behaviors; its action space is defined as: the bandwidth allocation ratio output by the agent for each network modality m. ,satisfy Where M is the number of network modes; S2. Set the training hyperparameters, including the number of training rounds, the number of fast adaptation steps in the inner loop, the learning rate in the inner loop, the learning rate in the outer loop, the meta-batch size, the outer loop parameters, and the inner loop parameters; S3. Input the currently collected global network feature status and corresponding perturbation task label into the meta-reinforcement learning agent; the agent first outputs bandwidth allocation action according to the task-specific strategy, and sends and executes it through the SDN switch; then monitors the network status and reward feedback after the switch executes the interaction action, and stores the interaction experience in the buffer of the corresponding task; if there is a perturbation task, the perturbation task label is not 0, and if there is no perturbation task, the perturbation task label is 0. S4. In the rapid adaptation phase of the inner loop, after the data in the task buffer reaches the predetermined batch size and several steps of interaction are completed, the network state and reward feedback in the task buffer are used to fine-tune the agent's policy network parameters by the rapid adaptation steps set in step S2, so as to obtain the dedicated policy network parameters for the task. S5. In the outer loop meta-update phase, for each task in the meta-batch, based on the task-specific policy network parameters obtained after the rapid adaptation in step S4, the adaptation parameters of all tasks are compared with the meta-parameters, and the globally shared meta-parameters are updated by gradient descent MAML. S6: Repeat steps S3–S5 until the meta-parameters converge, the bandwidth allocation of each network mode is stable, and there is no overload at the network egress end, thereby completing the continuous adaptive optimization of multiple network modes.

2. The multimodal network bandwidth adaptive allocation method according to claim 1, characterized in that, Step S3 specifically includes the following sub-steps: S31: The neural network adopts a unified backbone neural network structure. First, it collects the multi-dimensional features of each modal network at the current moment, and then performs feature representation and splicing through the embedding layer to form a global network state vector. S32: Input the concatenated global features into the modality coding layer. Its temporal features are encoded by a multi-head Transformer, and its static features are processed by an MLP. The output is a representation vector of uniform length for each network modality. S33: Input the encoding results of all network modalities into the self-attention mechanism to achieve global feature fusion, and finally output the global network feature state; S34: Input the fused global features into a multilayer MLP and output the bandwidth allocation coefficients for each network mode; perform softmax normalization on the bandwidth allocation coefficients to obtain the final allocation ratio for each network mode; S35: The meta-reinforcement learning agent takes the current global network feature state and the corresponding perturbation task label as input, and outputs the bandwidth allocation action according to the final allocation ratio generated in step S34. The final decision will be issued and bandwidth allocation executed via the SDN switch. S36: The agent interacts with the environment in several steps, and each interaction is recorded as a quintuple, including the pre-action state, the perturbation task label, the selected action, the post-action state, and the reward value; the experience data of this interaction is stored in the task buffer corresponding to the current task for subsequent policy network to quickly adapt and update.

3. The multimodal network bandwidth adaptive allocation method according to claim 2, characterized in that, Specifically, step S35 involves: if there is a disturbance task, then inputting the current network state at each step. and Disturbance Task Tags Among them, the disturbance task label If the value is not 0, and there is no perturbation task, the perturbation task tag will be used. The value is 0, and the meta-reinforcement learning agent follows the strategy. Output Action The actions described therein It is the sum of the action vector selected in the global state corresponding to the input and the noise, used to adjust the bandwidth of the network mode to schedule the communication resources that each network mode can occupy. Its physical meaning is the proportion of each flow of each network mode to the exit; assuming that... Indicates the first One network mode, Indicates the first Each network mode in The number of streams in the time interval determines the dimension of the output action space. Then in the t-th time slot, only Each element has a corresponding stream, so its value is between 0.1 and 1, while the other elements have a value of 0; actions are performed using an SDN switch. Post-feedback status and rewards Record data ( , , , , ) to task buffer The reward function designed is defined as follows: ； in Represents the number of network modes. Representing the The weights of each network mode. Represents time slot In the middle, the first Number of flows per network mode Representing network modes The i-th stream in time slot The demand for traffic at that time Indicates the first The i-th flow of a network mode is in The percentage of data that actually successfully reaches the server at any given moment. This indicates the maximum total traffic the system can handle; exceeding this limit results in overload. LOAD represents the sum of actual flows arriving from all network modes. The specific formula is as follows: The above reward function is designed to meet the communication transmission requirements of different service types in multiple network modes.

4. The multimodal network bandwidth adaptive allocation method according to claim 1, characterized in that, The inner loop fast adaptation phase in step S4 involves rapidly updating the policy network, specifically including: The latest interaction data, including the pre-action state, selected action, post-action state, and reward value, is written into the task buffer. In the next inner loop or subsequent outer loop, historical samples are extracted from the task buffer at a fixed ratio and mixed with the current task samples to balance rapid adaptation and cross-task experience accumulation. Input the global network feature state before all actions in the buffer into the policy network, calculate the action distribution or allocation probability in each state, combine the actual reward signal of each data and the discount factor to calculate the discounted reward of each state, and calculate the loss function of each sample based on the current policy output and the actual executed action. The total loss under the current perturbation task is obtained by summing up the losses of all samples, and the policy network parameters are optimized through backpropagation to make them more suitable for the current task environment.

5. The multimodal network bandwidth adaptive allocation method according to claim 4, characterized in that, The calculation of the discount reward for each state is specifically as follows: The quintuples from the multi-step sampling can be pieced together to form a complete interaction record from start to finish; for each step of the interaction, the cumulative reward of the discount needs to be calculated. Used to evaluate the trade-off between current rewards and future rewards: ； in The discount factor has a value range of (0, 1). The larger the value, the more the agent values ​​future rewards; conversely, the more it values ​​current rewards. Discounted cumulative rewards transform current reward signals into a global evaluation of future gains, enabling meta-reinforcement learning agents to adapt to complex environments and weigh short-term and long-term goals.

6. The multimodal network bandwidth adaptive allocation method according to claim 4, characterized in that, The total loss under the current perturbation task is obtained by accumulating all the mixed sample losses, and the policy network parameters are optimized through backpropagation. Specifically, for each perturbation task, historical data is first sampled from the experience pool when updating the parameters in the inner loop. Then, the current perturbation task data is mixed in. Obtain a mixed dataset ,in =K+n, where K represents the amount of newly sampled data in this task, i.e., the inner loop sampling length, and n represents the amount of historical experience pool samples, i.e., the number of historical trajectories used for mixed sampling. Represents the global network characteristic state. This represents the label for the disturbance task. Represents the reward value; calculates the mixed loss, i.e., for the mixed set The policy gradient loss of each sample is averaged to obtain the inner loop average loss of this perturbation task, and its expression is as follows: ;in These are the outer loop parameters, i.e., the meta-parameters; Discount rewards are applied to each entry in the mixed data; historical data is normalized for rewards. Represents the current global network feature state. Represents the state The action selected by the next strategy This represents the reward after performing the action. This represents the next state after the action is performed; finally, the policy network parameters are updated in the inner loop. ,in This is the learning rate for the inner loop.

7. The multimodal network bandwidth adaptive allocation method according to claim 1, characterized in that, In step S5, the globally shared meta-parameters are updated using the gradient descent method MAML. The tasks completed in multiple inner loops are archived, that is, the policy network parameters obtained after fine-tuning in the inner loop of each perturbation task are compared with the initial outer loop parameters. The fine-tuned parameters of all tasks are then differiated from the meta-parameters to calculate the parameter update direction. A weighted average is used, and the meta-policy parameters are uniformly adjusted using the outer loop learning rate. The updated meta-parameters are used as the starting point for a new round of inner loops. Since the outer loop update is based on the gradient descent method (MAML), within the MAML framework, gradients are calculated for each task completed in multiple inner loops, accumulated in the outer loop, and then updated all at once. Specifically: for one round of the outer loop, i rounds of inner loops have already been performed, N tasks have been sampled, and the fine-tuned parameters from each inner loop are obtained. Then the outer loop parameters are aggregated, as shown in the following expression: ； Where β is the outer loop learning rate.

8. A system for an adaptive bandwidth allocation method for multimodal networks based on meta-reinforcement learning according to any one of claims 1-7, characterized in that, Includes the following modules: (1) State collector: Actively and periodically acquires network feature data of each network mode and its subordinate flow, and performs feature normalization and preprocessing; (2) Dynamic event injection module: used to simulate and inject disturbance tasks such as traffic bursts, link quality changes, and new network mode access, and perform adaptive scheduling; (3) Intelligent agent module: Based on the MAML meta-reinforcement learning algorithm, it integrates multiple network modal features to realize bandwidth allocation strategy reasoning, rapid adaptation of the inner loop and aggregation of meta parameters of the outer loop; (4) Quota execution module: Translates the bandwidth allocation results of the agent's decision into control commands that can be recognized by the underlying network devices; (5) Performance monitoring module: end-to-end real-time monitoring of key indicators such as resource utilization, latency satisfaction rate and bandwidth allocation jitter of each network mode, and generating reward feedback for intelligent agent optimization decision-making; (6) Inner loop update module: Based on the rewards and status in the current perturbation task buffer, quickly and adaptively fine-tune the policy network parameters; (7) Outer loop update module: collects the parameter results after fine-tuning of the inner loop of multiple perturbation tasks and updates the meta-strategy parameters in a unified manner; (8) Repeat module: Used to repeatedly call the above modules in sequence until the bandwidth allocation of each network mode meets the communication quality and there is no overload at the output end.

9. The system according to claim 8, characterized in that, It also includes performance self-healing and automatic adjustment for fault detection, specifically including the following modules: Performance monitoring module: used to collect the operating status of various service modes and links in real time, and monitor core indicators, including bandwidth utilization, latency and packet loss; Adaptive optimization decision module: Used to diagnose the detection results of the performance monitoring module and dynamically generate adjustment strategies; Strategy execution module: Used to issue adjustment instructions based on the adjustment strategy generated by the adaptive optimization decision module, and continuously monitor the effect.

Images