A network decision method and system based on reinforcement learning

Abstract

This application proposes a network decision-making method and system based on reinforcement learning. The method includes: first, constructing a general interactive interface to achieve cross-modal fusion of network numerical states and natural language instructions through a hybrid embedding module, and outputting deterministic actions through a network action head; then, generating an offline experience pool using various heuristic algorithms under diverse network configurations; based on this experience pool, employing parameter-efficient supervised fine-tuning techniques to enable a large language model to quickly master basic decision-making paradigms; next, using the NetGRPO online reinforcement learning algorithm, leveraging tree-structured trajectory generation and grouping relative advantage optimization, achieving stable and efficient policy exploration and autonomous optimization; finally, deploying the fully trained model in a lightweight manner to form a low-latency, highly reliable real-time decision-making closed loop. This method significantly improves the performance ceiling, training stability, cross-task adaptability, and engineering feasibility of network decision-making systems.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and network communication technology, and in particular to a network decision-making method and system based on reinforcement learning. Background Technology

[0002] As networks continue to expand and services become increasingly complex, network decision-making tasks (such as adaptive rate selection, load balancing, and dynamic routing) place higher demands on real-time performance, accuracy, and adaptability. Related technologies primarily include rule-based heuristic algorithms and deep reinforcement learning-based methods.

[0003] However, heuristic algorithms rely on manually designed rules. While simple and efficient, they lack flexibility and generalization ability, making them difficult to adapt to complex and ever-changing network environments. Deep reinforcement learning methods autonomously learn decision-making strategies through interaction with the environment, possessing a certain degree of adaptability, but still suffer from the following problems: First, training is unstable: in complex network environments, reinforcement learning training is prone to slow convergence, large fluctuations, and even getting stuck in local optima; second, generalization ability is limited: models are often trained for specific tasks or environments, making it difficult to transfer to new network scenarios; third, decision interpretability is poor: traditional neural network decision-making processes lack semantic understanding and are difficult to integrate with network operation and maintenance knowledge.

[0004] Therefore, how to execute complex and dynamic network decision-making tasks more accurately, efficiently, and stably has become an urgent problem to be solved. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the first objective of this application is to propose a network decision-making method based on reinforcement learning, which achieves efficient and stable real-time network decision-making by designing a general interactive interface and a two-stage training algorithm.

[0007] The second objective of this application is to propose a network decision-making system based on reinforcement learning.

[0008] The third objective of this application is to propose an electronic device.

[0009] The fourth objective of this application is to provide a computer-readable storage medium.

[0010] To achieve the above objectives, the first aspect of this application is to propose a network decision-making method based on reinforcement learning, comprising the following steps: A general interaction interface is constructed between a large language model and the network environment. Based on the general interaction interface, the large language model can perceive the state of the network environment and output deterministic decision actions. The general interaction interface includes a hybrid embedding module and a network action head. Run existing heuristic algorithms and collect state-action trajectory data generated during the interaction between the heuristic algorithms and the environment to generate diverse offline experience pools; The offline experience pool is used to perform supervised fine-tuning of the large language model that has been connected to the general interaction interface, including using low-rank adaptation technology to optimize model parameters, so that the large language model can acquire basic decision-making capabilities and retain pre-trained knowledge. The fine-tuned large language model is subjected to online reinforcement learning using the NetGRPO algorithm to optimize the model's decision-making strategy. The NetGRPO algorithm includes tree-structured trajectory generation, action value estimation based on path aggregation, and grouping strategy optimization. The optimized large language model is deployed in a real network system, and online real-time decision-making is performed based on the trained model weights.

[0011] Optionally, the step of enabling the large language model to perceive the network environment state and output deterministic decision actions based on the general interaction interface includes: preprocessing the multimodal input through the hybrid embedding module, aligning and fusing the preprocessed multimodal input to generate a text embedding sequence and a numerical state embedding matrix; concatenating the text embedding sequence and the numerical state embedding matrix along the sequence dimension to generate an input embedding sequence, and inputting the input embedding sequence into the large language model; replacing the original language modeling head of the pre-trained large language model with the network action head, performing linear transformation through the network action head to calculate the logical value vector of each action in the action space; normalizing the logical value vector through the Softmax function to obtain the probability distribution of each effective action, and using a greedy strategy to select the action with the highest probability from among the effective actions as the decision output.

[0012] Optionally, the step of collecting state-action trajectory data generated during the interaction between the heuristic algorithm and the environment to generate a diverse offline experience pool includes: generating a large number of heterogeneous network environment configurations through a configurable environment simulator, wherein the large number of heterogeneous network environment configurations correspond to the complete actual network state space; running multiple heuristic algorithms through a multi-algorithm data collector to collect the decision behaviors of the multiple heuristic algorithms under various network environment configurations, generating complementary original expert trajectories; and standardizing, structuring, and quality balancing the original expert trajectories through an experience pool manager to output a dataset for supervised fine-tuning.

[0013] Optionally, the supervised fine-tuning of the large language model connected to the general interaction interface through the offline experience pool includes: initializing the large language model connected to the general interaction interface; adding a trainable low-rank factorization matrix to the attention mechanism module of the large language model and freezing the original relevant parameters of the large language model; setting the parameters of the network action head to a trainable state; randomly sampling trajectory data from the offline experience pool and constructing each trajectory data as a supervised fine-tuning sample; generating a supervised learning objective through a mimicry learning objective optimizer, wherein the supervised learning objective is used to maximize the log-likelihood of the large language model to the expert actions in the offline experience pool, the mimicry learning objective optimizer sets the basic loss term of the same batch of data as the classification cross-entropy loss and introduces a KL divergence regularization term to constrain the fine-tuning process; and performing supervised fine-tuning of the large language model based on the supervised fine-tuning samples and the supervised learning objective.

[0014] Optionally, the online reinforcement learning of the fine-tuned large language model using the NetGRPO algorithm includes: employing a tree-structured trajectory generation mechanism to construct a search tree of a preset depth starting from a given current environmental state, and evaluating the cumulative rewards of different action sequences through environmental simulation; performing dynamic depth-aware pruning at each layer of the search tree, retaining several child nodes with scores higher than a score threshold for each parent node for subsequent expansion; calculating the action value function estimate of the initial action by aggregating the cumulative rewards of all paths starting from the same initial action; forming action groups based on the action value function estimates of all actions in the same state, calculating the relative advantage within the action group, and updating policy parameters based on the relative advantage within the group to achieve grouping policy optimization.

[0015] Optionally, the step of deploying the optimized large language model in the actual network system and making online real-time decisions based on the trained model weights includes: using a lightweight model inference engine to transform the optimized large language model into an inference service engine in the network control plane; and using a decision-execution interface to transform the abstract decision actions output by the inference service engine into control commands executable by the network infrastructure.

[0016] To achieve the above objectives, a second aspect of this application also proposes a network decision-making system based on reinforcement learning, comprising the following modules: A construction module is used to build a general interaction interface between the large language model and the network environment. Based on the general interaction interface, the large language model can perceive the state of the network environment and output deterministic decision actions. The general interaction interface includes a hybrid embedding module and a network action head. The generation module is used to run existing heuristic algorithms, collect state-action trajectory data generated during the interaction between the heuristic algorithms and the environment, and generate diverse offline experience pools. The fine-tuning module is used to perform supervised fine-tuning of the large language model that has been connected to the general interaction interface through the offline experience pool, including optimizing the model parameters by using low-rank adaptation technology, so that the large language model can acquire basic decision-making capabilities and retain pre-trained knowledge. The reinforcement learning module is used to perform online reinforcement learning on the fine-tuned large language model using the NetGRPO algorithm to optimize the model's decision-making strategy. The NetGRPO algorithm includes tree-structured trajectory generation, action value estimation based on path aggregation, and grouping strategy optimization. The decision-making module is used to deploy the optimized large language model in the actual network system and make online real-time decisions based on the trained model weights.

[0017] To achieve the above objectives, a third aspect of this application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a network decision-making method based on reinforcement learning as described in any of the first aspects above.

[0018] To achieve the above objectives, the fourth aspect of this application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the network decision-making method based on reinforcement learning as described in any one of the first aspects.

[0019] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects: This application proposes a complete decision-making framework that integrates large language models and online reinforcement learning, constructing an end-to-end system that includes a general interface, two-stage training, and a dedicated optimization algorithm. This enables the model to actively explore and discover better strategies through real-time interaction with the environment, thereby achieving a leap from "imitating the known" to "autonomous optimization," significantly improving decision-making performance. The general interactive interface constructed in this application fundamentally solves the modal gap and decision controllability problems between large language models and the network environment. This interface not only achieves natural alignment and fusion of text instructions and numerical states, but also strictly constrains the model's open generation capabilities within a preset safe action space through defined action heads, ensuring the effectiveness and reliability of decision-making and laying the foundation for the safe application of the model in network control. Furthermore, this application employs the NetGRPO algorithm, effectively solving the core problem of instability in online training of large models in complex network environments. Accurate low-variance value assessment is achieved through tree-structured trajectory generation, and a policy update mechanism based on the relative advantage of actions within groups is employed. This encourages thorough exploration while ensuring extreme stability and efficiency in the training process, enabling large language models to handle complex online policy optimization. Furthermore, this application employs an offline experience pool combined with a two-stage training paradigm, possessing both rapid startup capability and continuous optimization potential. This paradigm ensures training efficiency and stability while preserving the model's long-term ability to continuously explore and adapt to environmental changes. Moreover, this application enhances cross-task and cross-environment generalization capabilities, significantly reducing deployment and adaptation costs. By simply changing the prompt word template and adjusting the action head dimension, the same system can quickly adapt to various network decision-making tasks such as adaptive bitrate selection and load balancing, maintaining good performance in unseen network environments, significantly improving the versatility of network decision-making and the economics of deployment.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a network decision-making method based on reinforcement learning proposed in this application embodiment; Figure 2 This is a schematic diagram illustrating the implementation principle of a network decision-making method based on reinforcement learning proposed in an embodiment of this application. Figure 3 This is a schematic diagram illustrating the working process of a hybrid embedded module proposed in an embodiment of this application; Figure 4This is a schematic diagram illustrating the computation process of a NetGRPO algorithm proposed in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a network decision-making system based on reinforcement learning proposed in an embodiment of this application. Detailed Implementation

[0022] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0023] It should be noted that in practical applications, real-world network environments require the simultaneous execution of multiple different decision-making tasks, where rule-based heuristic algorithms and traditional reinforcement learning suffer from limitations in generalization and stability. In some related embodiments, large language models, with their powerful semantic understanding, reasoning capabilities, and knowledge generalization potential, have been introduced into network decision-making tasks. However, directly applying large language models to network control still faces the following challenges: First, input-output mismatch: network states are numerical vectors, while the input to large language models is natural language; second, uncontrollable decision-making: the results generated by large language models are uncertain, making it difficult to guarantee that the output conforms to network control specifications; third, lack of online optimization capabilities: existing methods are mostly based on offline data training, making continuous learning and optimization in the environment impossible.

[0024] To this end, this application proposes a novel network decision-making method and system based on reinforcement learning, which can integrate the semantic understanding capabilities of large language models with the online optimization capabilities of reinforcement learning.

[0025] The following description, with reference to the accompanying drawings, illustrates a network decision-making method and system based on reinforcement learning, as proposed in an embodiment of this application.

[0026] Figure 1 This is a flowchart illustrating a network decision-making method based on reinforcement learning proposed in an embodiment of this application. Figure 2 This is a schematic diagram illustrating the implementation principle of a network decision-making method based on reinforcement learning proposed in an embodiment of this application. Figure 1 As shown, the method includes the following steps: Step S101: Construct a general interaction interface between the large language model and the network environment. Based on the general interaction interface, the large language model can perceive the state of the network environment and output deterministic decision actions. The general interaction interface includes a hybrid embedding module and a network action head.

[0027] Specifically, this step constructs a universal interaction interface between the large language model and the network environment. A unified input-output interface is designed to enable the pre-trained large language model to perceive the network environment state and output deterministic decision actions. This universal interaction interface includes: a hybrid embedding module, used to encode the numerical state of the network environment and natural language task prompts into a unified vector representation. The natural language prompts are processed through the original word embedding layer of the large language model, while the numerical state is mapped to an embedding vector through an independent multilayer perceptron. The two are concatenated and input into the model; and a network action head, used to replace the original model's generation head. This head employs a linear classification layer to map the hidden states output by the model to a probability distribution in a finite action space, ensuring the determinism and effectiveness of the output actions.

[0028] In one embodiment of this application, enabling a large language model to perceive the network environment state and output deterministic decision actions based on a general interactive interface includes the following steps: Step S11: The multimodal input is preprocessed by the hybrid embedding module, and the preprocessed multimodal input is aligned and fused to generate a text embedding sequence and a numerical state embedding matrix.

[0029] Specifically, this embodiment constructs a general interactive interface, enabling the pre-trained large language model to perceive the network environment state and output deterministic decision actions. This process can be achieved using the following two modules: a hybrid embedding module and a network action head. The core function of the hybrid embedding module is to address the modal gap between the large language model (which natively processes discrete text symbols) and the network environment (which outputs continuous numerical states), achieving efficient alignment and fusion of cross-modal information.

[0030] The working principle of the hybrid embedded module is as follows: Figure 3 As shown, the specific workflow of this module includes: First, multimodal input preprocessing is performed. For natural language modalities, a structured text prompt is generated based on the current decision task (such as load balancing or adaptive bitrate selection), for example, "Based on the current server load and task queue, please select the target server number (0-7):". This prompt is processed by the native word segmenter of the large language model, converted into a word sequence, and then mapped into a high-dimensional text embedding vector sequence through a pre-trained word embedding layer. For numerical state modalities, synchronously collected real-time network metrics (such as CPU utilization, memory usage, and link latency) constitute a state vector. After outlier processing and sliding window normalization, its numerical range is made consistent with the distribution of the training data.

[0031] Then, modal alignment and fusion are performed: each normalized scalar state value is nonlinearly projected through an independent multilayer perceptron, mapping the one-dimensional scalar into a vector with the same dimension as the text embedding, and the embedding vectors corresponding to all state values ​​are concatenated to form a numerical state embedding matrix.

[0032] Step S12: Concatenate the text embedding sequence and the numerical state embedding matrix along the sequence dimension to generate the input embedding sequence, and input the input embedding sequence into the large language model.

[0033] Specifically, the hybrid embedding module concatenates the text embedding sequence with the numerical state embedding matrix along the sequence dimension to form a complete input embedding, which is then fed into the large language model. Furthermore, the model's internal self-attention mechanism automatically learns the semantic relationships between text instructions and numerical states, achieving joint instruction-state understanding and providing a unified context-aware representation for subsequent decision generation.

[0034] Step S13: Replace the original language modeling head of the pre-trained large language model with a network action head, and calculate the logical value vector of each action in the action space by performing linear transformation through the network action head.

[0035] The core function of the network action head module is to strictly constrain and transform the open-ended text generation capabilities of the large language model into deterministic action outputs that conform to network operation specifications, fundamentally eliminating invalid or harmful "illusionary" decisions. The module's workflow first requires structural replacement and adaptation, replacing the original language modeling head of the pre-trained large language model with a dedicated network action head. This action head consists of a linear classification layer, and its output dimension is the size of the discrete action space predefined for the current network decision task (such as the number of selectable servers and bitrate levels).

[0036] Then, deterministic action generation is performed. During inference, the model takes the hidden state corresponding to the [CLS] token at the beginning of the sequence in the comprehensive context representation output by the hybrid embedding module, performs a linear transformation through the action head, and obtains the action logic value vector.

[0037] Step S14: Normalize the logical value vector using the Softmax function to obtain the probability distribution of each effective action, and use a greedy strategy to select the action with the highest probability from each effective action as the decision output.

[0038] Specifically, the network action head then applies the Softmax function to normalize the action logic value vector generated in step S13, obtaining the probability distribution of all valid actions. Finally, a greedy strategy is adopted to select the action with the highest probability as the decision output. This process ensures that the output is strictly limited to a preset set of safe actions, and that the output result has repeatable determinism under the same input.

[0039] The following example illustrates the process of integrating multimodal inputs and generating deterministic outputs using a general interactive interface.

[0040] In this example, a structured text prompt is first constructed for a specific network decision-making task. This prompt explicitly describes the task objective, the meaning of the input state, and the allowed action space. For example, for the adaptive bitrate selection task, the prompt is: "Please select the optimal video bitrate level (level 0 to 5) based on the following network states: previous bitrate, buffer level, throughput, download latency, subsequent fragment size, and remaining fragment count." This prompt is segmented using the native word segmenter of the large language model and converted into a sequence of word embedding vectors via a pre-trained word embedding layer. Simultaneously, real-time numerical metrics collected from the network environment (such as bandwidth, latency, and queue length) constitute the state vector and are converted into a sequence of numerical vectors through normalization and a multilayer perceptron. Finally, the word embedding vector sequence and the numerical vector sequence are concatenated along the sequence dimension to form a complete input embedding sequence. This concatenated sequence is then input into the Transformer layer of the large language model. The model's self-attention mechanism automatically learns the semantic relationships between the text instructions and numerical states during this process, achieving cross-modal information alignment and fusion, and generating a comprehensive representation containing task context and real-time environment information.

[0041] To eliminate the "illusion" problem that may arise in open generation of large language models and to ensure that the output strictly conforms to network control specifications, the interface uses a network action head to replace the original model's vocabulary probability head. This action head consists of a linear layer. The model takes the position vector corresponding to the [CLS] token from the output representation vector and calculates the original logical value of each action in the action space through the network action head. Subsequently, the probability distribution of all valid actions is obtained through the Softmax function. The system selects the action with the highest probability as the final output. This design ensures that the output is strictly limited to the preset set of valid actions, and that the decision-making process is deterministic and repeatable.

[0042] Therefore, based on the design of the interaction interface in this example and the semantic decoupling from the specific network task, when applied to different tasks, only the natural language prompt word template needs to be changed, and the output dimension of the network action head needs to be adjusted accordingly to match the action space of the new task, without changing the core architecture of the interface. This achieves the goal of a single interface adapting to multiple tasks.

[0043] Step S102: Run the existing heuristic algorithm and collect the state-action trajectory data generated during the interaction between the heuristic algorithm and the environment to generate a diverse offline experience pool.

[0044] Specifically, this step uses existing heuristic algorithms to generate an offline experience pool. One or more classic heuristic decision-making algorithms are run in diverse network environments, collecting state-action trajectory data generated during their interaction with the environment to construct the initial experience pool. This experience pool is used in the subsequent supervised fine-tuning phase to provide basic task knowledge and decision-making paradigms.

[0045] In one embodiment of this application, state-action trajectory data generated during the interaction between heuristic algorithms and the environment are collected to generate a diverse offline experience pool. This includes: generating a large number of heterogeneous network environment configurations through a configurable environment simulator, wherein the large number of heterogeneous network environment configurations correspond to the complete actual network state space; running multiple heuristic algorithms through a multi-algorithm data collector to collect the decision behaviors of multiple heuristic algorithms under various network environment configurations, generating complementary original expert trajectories; and standardizing, structuring, and quality balancing the original expert trajectories through an experience pool manager to output a dataset for supervised fine-tuning.

[0046] Specifically, in this embodiment, the process of generating diverse offline experience pools using existing heuristic algorithms can employ the following three functional modules: a multi-algorithm data collector, a configurable environment simulator, and an experience library manager.

[0047] The core function of the multi-algorithm data collector is to automatically run multiple classic heuristic algorithms with different design philosophies, collect their decision-making behaviors under various network conditions, and form complementary expert trajectories. Its specific workflow is as follows: First, the algorithm library is configured and initialized. The module has a built-in extensible heuristic algorithm library that integrates multiple representative algorithms for the same network task. For example, in the adaptive bitrate selection task, it simultaneously includes buffer-based algorithms, throughput prediction-based model predictive control algorithms, and fixed-rule conservative algorithms. Each algorithm is encapsulated as an independent decision agent. Then, interaction and trajectory recording are performed. In a given environment instance, multiple algorithm agents interact independently with the network environment after initialization. At each decision step, each agent observes the current environment state according to its internal logic, outputs an action, and receives immediate rewards and the next state from the environment. The system synchronously records complete interaction quadruples, forming decision trajectories, which ultimately constitute an experience pool.

[0048] The core function of the configurable environment simulator is to systematically generate a large number of heterogeneous network environment configurations, ensuring that the collected experience data can be widely distributed across all possible real-world scenarios, thereby greatly improving the generalization ability of subsequent training models. Its specific workflow begins with parametric configuration generation. Through configuration templates, a series of key network parameters and their value ranges or distributions are defined. Then, based on these templates, multiple specific and distinct environment configuration instances are generated. For example, configuration 1 might represent a "high-bandwidth, high-latency wide area network," configuration 2 might represent a "low-bandwidth, low-latency local area network," and configuration 3 might simulate a "mobile network with drastically fluctuating bandwidth." Finally, an interactive simulation environment is generated for each configuration instance. This environment can dynamically generate network state sequences that conform to the statistical characteristics of the configuration during interaction, providing the data collector with a realistic, diverse, and controllable interactive platform.

[0049] The Experience Base Manager is responsible for standardizing, structuring, and quality-balancing the collected raw trajectories, outputting a high-quality dataset that can be directly used for supervised fine-tuning. Its workflow is as follows: After receiving the raw trajectories from the collector, the manager's preprocessing pipeline performs the following operations: First, anomaly filtering: removing abnormal records containing invalid values ​​(such as NaN). Second, normalization: normalizing each feature dimension in the state vector based on statistical information from all data (such as minimum, maximum, mean, and variance). Third, sequence slicing: cutting long trajectories into fixed-length segments to accommodate model training. The cleaned data is stored in an efficient columnar storage format. Each record contains core fields such as state, action, reward, and next state, and is associated with its algorithm ID and environment configuration ID. The system also builds a multi-level index to support efficient data retrieval and sampling by algorithm, configuration, or state feature interval. Finally, the manager performs statistical analysis on the stored data to identify imbalances in data distribution (e.g., insufficient data from a particular algorithm or configuration). For data categories with insufficient quantity, the system will send instructions to the data collector and simulator to generate supplementary data. Ultimately, the manager outputs a large, evenly distributed, and quality-controlled offline experience pool, laying a solid foundation for subsequent supervised fine-tuning.

[0050] The following example illustrates the process of integrating multimodal inputs and generating deterministic outputs using a general interactive interface.

[0051] In this example, the offline experience base generation comprehensively utilizes multiple heuristic algorithms and various environment configurations, as detailed below: To construct a comprehensive, diverse, and high-quality initial experience pool and avoid decision bias from a single algorithm, multiple classic heuristic network decision-making algorithms are run. These algorithms are based on different design principles and can generate complementary decision strategies under the same environmental conditions. For example, in the adaptive bitrate selection task, a buffer-based algorithm (BBA), a throughput-based model predictive control (MPC) algorithm, and a fixed threshold-based selection algorithm can be run simultaneously; in the load balancing task, algorithms such as round-robin scheduling, minimum connections, and shortest remaining time can be run simultaneously. Each algorithm runs independently in the environment, generating its own series of "state-action-reward" trajectory tuples.

[0052] To ensure the experience pool covers a broad and representative network state space, an environment configuration generator systematically constructs various different environment configuration scenarios. Each configuration is defined by a set of key network parameters, including but not limited to: link bandwidth, network latency and jitter, background traffic patterns and interference intensity, and network topology. By running the heuristic algorithm under these significantly different configurations, it is ensured that the collected state data is widely distributed across the entire possible state space, thereby improving the generalization ability of subsequent model training.

[0053] Under each combination of algorithm and environment configuration, a complete data collection process is executed. The system records the raw environmental observations at each decision moment, the action selected by the algorithm, the immediate reward from the environment, and the state at the next moment. All collected trajectory data is cleaned, anomaly removed, and normalized before being stored in the experience pool in a unified structured format.

[0054] To overcome the problem of insufficient data volume or low data quality under certain configurations or algorithms, the system implements an experience pool balancing strategy. This strategy statistically analyzes data from different combinations (algorithms, configurations), and if the data volume of a certain type is significantly lower than the average level, the data collection time for that combination is extended accordingly.

[0055] Step S103 involves supervising the large language model that has been connected to the general interaction interface through an offline experience pool. This includes optimizing the model parameters using low-rank adaptation techniques to enable the large language model to acquire basic decision-making capabilities and retain pre-trained knowledge.

[0056] Specifically, this step involves supervised fine-tuning of the large language model based on the offline experience pool built in the previous step. During supervised fine-tuning, low-rank adaptation techniques are used to efficiently fine-tune the model's parameters, enabling it to mimic the decision-making behavior in the experience pool, quickly acquire preliminary network decision-making capabilities, and maintain the stability of pre-trained knowledge.

[0057] In one embodiment of this application, supervised fine-tuning of a large language model that has been connected to a general interaction interface is performed through an offline experience pool, including the following steps: Step S21: Initialize the large language model that has been connected to the general interaction interface, add a trainable low-rank decomposition matrix to the attention mechanism module of the large language model and freeze the original relevant parameters of the large language model, and set the parameters of the network action head to a trainable state.

[0058] Specifically, the supervised fine-tuning process of the large language model based on the offline experience pool in this embodiment can be achieved using the following two modules: a parameter-efficient fine-tuning trainer and an imitation learning objective optimizer. The core task of the parameter-efficient fine-tuning trainer is to utilize the offline experience pool constructed in the previous step to perform supervised learning on the large language model that has integrated general interfaces, enabling its decision-making ability to quickly converge to a high-performance initial baseline, while preserving its valuable pre-trained general knowledge and reasoning capabilities to the greatest extent possible.

[0059] The specific workflow of this module includes: First, initializing the complete model built in step S101, which integrates the hybrid embedding module and the network action head. To achieve efficient parameter training, a low-rank approximation technique is employed. Specifically, a trainable low-rank decomposition matrix is ​​added as a bypass to the attention projection matrix of the Transformer layer of the large language model, while the remaining parameters of the model (e.g., including the embedding layer, feedforward network layer, and the newly added hybrid embedding MLP) are frozen. Simultaneously, the parameters of the network action head are set to be fully trainable. This design ensures task adaptation can be achieved with the introduction of a very small number of new parameters (typically less than 1% of the original model).

[0060] Step S22: Randomly sample trajectory data from the offline experience pool and construct each trajectory data as a supervised fine-tuning sample.

[0061] Specifically, the trainer randomly samples trajectory data in batches from the experience pool. For each sample, the parameter-efficient fine-tuning trainer forward-processes the state vector and fixed task cues through a hybrid embedding module to generate the model input. The corresponding expert actions are then used as supervision labels.

[0062] Step S23: A supervised learning objective is generated by imitating the learning objective optimizer. The supervised learning objective is used to maximize the log-likelihood of the large language model to the expert actions in the offline experience pool. The imitating learning objective optimizer sets the basic loss term of the same batch of data as the classification cross-entropy loss and introduces the KL divergence regularization term to constrain the fine-tuning process.

[0063] Specifically, the core responsibility of the imitation learning objective optimizer is to define and compute a sophisticated supervised learning objective that not only drives the model to accurately mimic expert behavior but also ensures the stability of the learning process and knowledge retention. The optimizer's core objective is to maximize the model's log-likelihood of expert actions in the experience pool. For a batch of data, its basic loss term is the standard classification cross-entropy loss, which directly encourages the model to reproduce the decision-making patterns of various heuristic algorithms under diverse conditions.

[0064] Furthermore, to prevent the model from "catastrophic forgetting" on specialized task data and losing its original language comprehension and logical reasoning abilities (which are crucial for handling unseen states and making complex decisions), the optimizer introduces KL divergence regularization. It imposes constraints by calculating the KL divergence of the output probability distribution in the action space between the fine-tuned model and the original frozen pre-trained model (as a reference model) given the same input. This regularization term acts as an "anchor," pulling the learning trajectory of the fine-tuned model closer to the pre-trained knowledge space, ensuring that the evolution of capabilities is "evolution" rather than "mutation." The optimizer ultimately performs a weighted sum of the imitation objective and the regularization objective to form the total loss, and uses the backpropagation algorithm to calculate the gradient with respect to the trainable parameters.

[0065] Step S24: Perform supervised fine-tuning of the large language model based on supervised fine-tuning samples and supervised learning objectives.

[0066] Specifically, the model is fine-tuned in a supervised manner using the supervised fine-tuning samples and supervised learning objectives generated in the above process. The supervised fine-tuning model produced by this optimization method can change the capability paradigm from open text generation to deterministic decision-making in a limited action space, and has the basic understanding and responsiveness to network decision-making tasks. This provides a stable, reliable and high-performance initial strategy for the subsequent high-risk, high-reward online reinforcement learning stage.

[0067] The following example illustrates how supervised fine-tuning can be used to shift the paradigm of large models from generative language to decision-making.

[0068] In this example, the offline experience pool containing diverse state-action pairs, constructed in step S102, is first invoked. For each trajectory in the pool, it is constructed as a supervised fine-tuning sample. Its input includes a state value vector and a fixed task description prompt (e.g., "Please select the optimal action based on the following network states:"). A unified input representation that the model can understand is generated using the hybrid embedding module in the general interaction interface constructed in step S101. Its label is set to the action in the experience pool. Before fine-tuning, the network action head (a linear classification layer) designed in S101 is integrated on top of the large language model to be fine-tuned. The output dimension of this layer is strictly consistent with the size of the task's action space. To efficiently adapt to the decision task while preserving the general knowledge of the large language model, low-rank adaptation (LoRA) is employed. Specifically, a trainable low-rank decomposition matrix is ​​added to the model's attention mechanism module, while most of the original model parameters are frozen. The parameters of the network action head are set to be fully trainable.

[0069] The training objective is to maximize the probability that the model predicts expert actions from the experience pool under given conditions. The standard cross-entropy loss function is used as the primary optimization objective. Simultaneously, to prevent the model from deviating excessively from its valuable pre-trained knowledge base during fine-tuning, a lightweight KL divergence regularization term is introduced to constrain the output distribution of the fine-tuned model from deviating too much from the "behavior" of the original pre-trained model under the same input. Gradient descent is used to update the trainable parameters, and the model is retrained multiple times until the loss converges.

[0070] Thus, through supervised fine-tuning, the model's capabilities have undergone a fundamental transformation, shifting from understanding open text to understanding a specific multimodal input pattern of "structured cue words + numerical environmental states." Its attention mechanism has changed from capturing semantic relationships between texts to learning the decision logic mapping between network state features and task objectives. Its output has changed from generating arbitrary-length, open-vocabulary text sequences to outputting a definite, executable decision label within a finite, pre-defined action space.

[0071] Step S104: Online reinforcement learning is performed on the fine-tuned large language model using the NetGRPO algorithm to optimize the model's decision-making strategy. The NetGRPO algorithm includes tree-structured trajectory generation, action value estimation based on path aggregation, and grouping strategy optimization.

[0072] Specifically, this step uses the NetGRPO algorithm for online reinforcement learning. Building upon the supervised fine-tuning performed in the previous step, the model is placed in a real or simulated network environment for online interaction. The NetGRPO algorithm is used for online environment interaction and policy optimization. The online reinforcement learning process includes using tree-structured trajectory generation and path aggregation Q-value estimation to obtain more accurate action advantage estimates, followed by grouped policy optimization to achieve more stable policy updates, breaking through the performance ceiling of imitation learning.

[0073] The NetGRPO algorithm used in this application may include the following steps: First, tree-structured trajectory generation: Starting from the current state, construct a search tree with a maximum depth of D, and evaluate the cumulative reward of different action sequences through environmental simulation. Second, dynamic depth-aware pruning: Adjust the balance between exploration and utilization based on depth, retaining only the highest-scoring child nodes at each layer to control computational complexity. Third, action-value function (Q-value) estimation based on path aggregation: Calculate a robust Q-value estimate for an action by aggregating the cumulative rewards of all paths starting from the same initial action. Fourth, grouping strategy optimization: Construct action groups based on the Q-value estimates of all possible actions in the same state, calculate the relative advantage within each group, and update the policy parameters accordingly to ensure training stability.

[0074] In one embodiment of this application, online reinforcement learning is performed on a fine-tuned large language model using the NetGRPO algorithm, including the following steps: Step S31: Using the tree structure trajectory generation mechanism, starting from the given current environment state, construct a search tree of a preset depth, and evaluate the cumulative reward of different action sequences through environment simulation.

[0075] Specifically, the online reinforcement learning process using the NetGRPO algorithm in this embodiment can employ the following two modules: a tree-structured trajectory estimator and planner, and a grouped relative policy optimizer. The execution process of the NetGRPO algorithm is as follows: Figure 4 As shown. Among them, the core function of the tree-structured trajectory evaluator and planner is to overcome the problems of large variance and inaccurate action value estimation in traditional online reinforcement learning single-path Monte Carlo evaluation. Through multi-step look-ahead tree search, it provides a robust, low-variance long-term value estimate for each possible action under the current policy.

[0076] The specific workflow of this module includes: First, given the current environment state as the root node, the module, guided by the current policy, enumerates all possible actions, simulates the execution of each action using the environment, and generates corresponding child nodes. Then, starting from the first layer, it iteratively expands the tree using a breadth-first search strategy until a preset maximum depth is reached. During each layer expansion, the tree structure trajectory evaluator and planner collects the states of all current leaf nodes, batches them through the policy network to obtain the action probability distribution, and simulates execution to generate the next layer of child nodes. The cumulative discounted reward of each new node is recursively calculated from the rewards of all its descendant nodes, the reward of the current step, and the discount factor.

[0077] Step S32: Perform dynamic depth-aware pruning at each level of the search tree, and retain several child nodes with scores higher than the score threshold for each parent node for subsequent expansion.

[0078] Specifically, to avoid state space combinatorial explosion, the tree-structured trajectory evaluator and planner also performs intelligent pruning at each level. For each parent node to be expanded, it calculates a comprehensive score for all its candidate child nodes, and each parent node only retains the top few child nodes with the highest scores (i.e., scores above a preset score threshold) for subsequent expansion. Thus, this mechanism achieves in-depth exploration of high-potential decision branches with controllable computational overhead.

[0079] Step S33: Calculate the action value function estimate of the initial action by aggregating the cumulative rewards of all paths originating from the same initial action.

[0080] Specifically, the core function of the grouped relative policy optimizer is to use the refined value assessment provided by the planner to stably update the policy parameters in a comparative learning manner, thus completely solving the problems of training instability and slow convergence caused by high variance reward signals in traditional policy gradient methods.

[0081] The specific workflow of this module includes: First, for each initial action at the root node, the optimizer aggregates the final cumulative rewards of all complete paths originating from that root node in the search tree, and calculates a robust Q-value estimate for that action by averaging the results. This method significantly smooths out the random noise of a single trajectory through multi-path averaging, generating a high-quality value signal.

[0082] Step S34: Based on the estimated action value function values ​​of all actions in the same state, form an action group, calculate the relative advantage within the action group, and update the policy parameters based on the relative advantage within the group to achieve grouping policy optimization.

[0083] Specifically, the grouped relative policy optimizer defines all possible initial actions and their estimated Q-values ​​in the same state as an action group. Within each group, the standardized relative advantage of each action is calculated. This advantage value represents the relative merit of an action within the current group, rather than its absolute value, which makes the learned signal more stable.

[0084] Furthermore, the grouped relative policy optimizer utilizes the calculated intra-group relative advantage to update policy parameters through an improved proximal policy optimization objective. Its loss function combines a pruning alternative objective with a KL divergence regularization term. This policy parameter update process is based on the relative advantage of an action within the group, rather than isolated absolute advantages. This guides policy learning to make better choices among current options, achieving a more natural and stable learning paradigm.

[0085] The following example illustrates how NetGRPO can be used for online reinforcement learning to improve the model's decision-making capabilities.

[0086] In this example, the supervised fine-tuning model is deployed within a network environment interaction loop. The system copies the model weights as the current policy parameters and initializes the online policy model accordingly. In each training round, the model observes the current environment state through the generic interaction interface set in step S101 and outputs the action probability distribution, thereby sampling actions for execution. The environment provides immediate rewards and transitions to the new state. The data generated by the interaction is collected in the online experience buffer pool for policy updates using the NetGRPO algorithm. The NetGRPO algorithm achieves stable and efficient policy optimization through four core mechanisms.

[0087] To more accurately assess the long-term value of individual actions, the algorithm constructs a search tree starting from the current state as the root node. At the root node, all possible actions are enumerated, and a child node is generated for each action using the current policy and environment model. Each child node stores the simulated state after executing the action, the immediate reward obtained, and the prior probability of the policy. The tree is then expanded breadth-first to a predetermined depth. At each non-root node, all possible actions are simulated using the same policy and environment model, generating the next level of child nodes, and the discounted reward is cumulatively calculated. To avoid combinatorial explosion, the algorithm performs intelligent pruning during node expansion. A comprehensive score is calculated for each potential child node of each node to be expanded, and only the top few child nodes with the highest scores are retained for subsequent expansion. For each initial action at the root node, the algorithm aggregates the final cumulative reward of all complete paths in the search tree starting from that action. The Q-value estimate for this action is calculated as the average of the rewards of these paths.

[0088] Therefore, this example significantly reduces the variance introduced by a single Monte Carlo method through multi-path averaging, providing a more stable and accurate payoff signal for policy updates. All initial actions and their estimated Q-values ​​are grouped into an action group, and the standardized advantage of each action within the group is calculated. The objective function for updating policy parameters is calculated based on the standardized advantage, combining robustness from pruning near-end policy optimization with KL divergence regularization. By updating the relative policy probabilities of a group of actions, rather than relying on the absolute payoff of a single action, the stability of policy updates is greatly improved.

[0089] Step S105: Deploy the optimized large language model in the actual network system and make online real-time decisions based on the trained model weights.

[0090] Specifically, this step involves online real-time decision-making based on the trained model weights. The large language model, optimized through steps S103 and S104, is then deployed in the actual network system. During the inference phase, the model receives real-time network status and task prompts, outputs the action probability distribution through forward computation, and selects the optimal action to execute, thus achieving an adaptive, low-latency network decision-making closed loop.

[0091] In one embodiment of this application, the optimized large language model is deployed in an actual network system, and online real-time decision-making is performed based on the trained model weights. This includes: using a lightweight model inference engine to transform the optimized large language model into an inference service engine in the network control plane; and using a decision-execution interface to transform the abstract decision actions output by the inference service engine into control commands executable by the network infrastructure.

[0092] Specifically, the online real-time decision-making process based on the trained model weights in this embodiment can employ the following two modules: a lightweight model inference engine and a decision-execution interface. The core function of the lightweight model inference engine is to transform the large language model decision system with massive parameters, trained in steps S103 and S104, into a service capable of millisecond-level, high-throughput real-time inference within a resource-constrained network control plane.

[0093] The specific workflow of this module includes: First, the lightweight model inference engine receives the final training weights, converts the dynamic computation graph into a static graph, and removes all components used only for training (such as Dropout layers and gradient calculation nodes). Then, a series of inference optimization techniques are applied to generate highly optimized executable code. To ensure the repeatability and reliability of decisions, the engine fixes the random seed during inference and directly selects the action with the highest probability.

[0094] Furthermore, the core function of the decision-execution interface is to securely and reliably translate the abstract decision actions output by the inference engine into control commands executable by specific network infrastructure. The specific execution process of this module includes: First, the component maintains an action-instruction mapping table, mapping the discrete action IDs output by the model to specific operation instructions for the corresponding network devices or controllers. For example, in a load balancing task, action "3" is translated into an SDN flow table entry for "schedule the new flow to server group 3"; in congestion control, the action "reduce window" is translated into a specific TCP congestion control parameter modification command. The translation process ensures that the instruction syntax, protocol, and target device are fully compatible. Subsequently, before issuing the instruction, the decision-execution interface performs security and feasibility verification to ensure that the instruction parameters are within the safe range allowed by the device and do not conflict with the current overall network policy.

[0095] The following example illustrates the process of deploying trained model weights online in real time.

[0096] In this example, after completing the online reinforcement learning training in step S104, the final optimized policy model weights are obtained. These model weights (including the large language model backbone, LoRA adapter, mapping network parameters of the hybrid embedding module, and weights of the network action head, etc.) are then solidified and serialized, saved as a model file that can be directly used for forward inference. This file is deployed to a decision server or edge computing node in a real-world network environment using a dedicated loader, forming an online decision engine. When the network system needs to make a decision about the current state, an automated real-time inference sequence is triggered.

[0097] First, the system collects raw state indicators of the current network environment (such as bandwidth utilization, latency, packet loss rate, queue length, task metadata, etc.) in real time through monitoring probes or network controller interfaces. The collected raw data is cleaned and normalized according to the same specifications defined in the training phase, transforming it into a fixed-dimensional state vector consistent with the training data distribution. The normalized state vector and predefined task prompt text are input into the hybrid embedding module of the general interaction interface. This module generates a unified multimodal embedding sequence on the fly, which serves as the input to the model. The solidified model processes the input embedding sequence using a forward propagation method. During the inference phase, all training-related components and gradient calculations are disabled to minimize computational latency. The network action head receives the hidden state representation, calculates the logical value of each valid action, and converts it into an action probability distribution using a Softmax function. The system selects the action with the highest probability as the final decision. The generated decision action is converted into specific control instructions that can be executed by network devices or controllers (such as API calls, configuration commands, and forwarding rule updates), and sent to the execution unit through a secure network channel.

[0098] In summary, the reinforcement learning-based network decision-making method of this application first constructs a general interactive interface to bridge the large language model and the network environment, and uses various heuristic algorithms to generate a widely covered offline experience pool; then, through supervised fine-tuning, the model quickly masters basic decision-making capabilities, and the NetGRPO online reinforcement learning algorithm is used to guide the model to break through the performance limit and achieve autonomous policy optimization in stable exploration; finally, the fully trained model is lightweightly deployed in the actual network system to form a low-latency, highly reliable real-time perception-decision-execution closed loop, thereby significantly improving the intelligence level and dynamic adaptability of network autonomous operation and maintenance.

[0099] To achieve the above embodiments, this application also proposes a network decision-making system based on reinforcement learning. Figure 5 This is a schematic diagram of the structure of a network decision-making system based on reinforcement learning proposed in an embodiment of this application, as shown below. Figure 5 As shown, the system includes: Module 100 is used to build a general interaction interface between the large language model and the network environment. Based on the general interaction interface, the large language model can perceive the state of the network environment and output deterministic decision actions. The general interaction interface includes a hybrid embedding module and a network action head.

[0100] The generation module 200 is used to run existing heuristic algorithms, collect state-action trajectory data generated during the interaction between the heuristic algorithms and the environment, and generate a diverse offline experience pool.

[0101] The fine-tuning module 300 is used to perform supervised fine-tuning of a large language model that has been connected to the common interaction interface through an offline experience pool. This includes optimizing model parameters using low-rank adaptation techniques to enable the large language model to acquire basic decision-making capabilities and retain pre-trained knowledge.

[0102] The reinforcement learning module 400 is used to perform online reinforcement learning on the fine-tuned large language model using the NetGRPO algorithm to optimize the model's decision-making strategy. The NetGRPO algorithm includes tree-structured trajectory generation, action value estimation based on path aggregation, and grouping strategy optimization.

[0103] The decision module 500 is used to deploy the optimized large language model in the actual network system and make online real-time decisions based on the trained model weights.

[0104] In one embodiment of this application, the construction module 100 is specifically used for: preprocessing the multimodal input through the hybrid embedding module, aligning and fusing the preprocessed multimodal input to generate a text embedding sequence and a numerical state embedding matrix; concatenating the text embedding sequence and the numerical state embedding matrix in the sequence dimension to generate an input embedding sequence, and inputting the input embedding sequence into the large language model; replacing the original language modeling head of the pre-trained large language model with a network action head, performing linear transformation through the network action head to calculate the logical value vector of each action in the action space; normalizing the logical value vector through the Softmax function to obtain the probability distribution of each effective action, and using a greedy strategy to select the action with the highest probability from each effective action as the decision output.

[0105] It should be noted that the foregoing explanation of the embodiments of the network decision-making method based on reinforcement learning also applies to the system of this embodiment, and will not be repeated here.

[0106] In summary, the reinforcement learning-based network decision-making system of this application significantly improves the performance ceiling, training stability, cross-task adaptability, and engineering feasibility of network decision-making systems.

[0107] To implement the above embodiments, this application also proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the reinforcement learning-based network decision-making method as described in any of the first aspect embodiments above.

[0108] To implement the above embodiments, this application also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the network decision-making method based on reinforcement learning as described in any one of the first aspects of the embodiments above.

[0109] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0110] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0111] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0112] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0113] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0114] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0116] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A network decision-making method based on reinforcement learning, characterized in that, Includes the following steps: A general interaction interface is constructed between a large language model and the network environment. Based on the general interaction interface, the large language model can perceive the state of the network environment and output deterministic decision actions. The general interaction interface includes a hybrid embedding module and a network action head. Run existing heuristic algorithms and collect state-action trajectory data generated during the interaction between the heuristic algorithms and the environment to generate diverse offline experience pools; The offline experience pool is used to perform supervised fine-tuning of the large language model that has been connected to the general interaction interface, including using low-rank adaptation technology to optimize model parameters, so that the large language model can acquire basic decision-making capabilities and retain pre-trained knowledge. The fine-tuned large language model is subjected to online reinforcement learning using the NetGRPO algorithm to optimize the model's decision-making strategy. The NetGRPO algorithm includes tree-structured trajectory generation, action value estimation based on path aggregation, and grouping strategy optimization. The optimized large language model is deployed in a real network system, and online real-time decision-making is performed based on the trained model weights.

2. The method according to claim 1, characterized in that, The process of enabling a large language model to perceive the network environment state and output deterministic decision actions based on the general interactive interface includes: The hybrid embedding module preprocesses the multimodal input, aligns and fuses the preprocessed multimodal input, and generates a text embedding sequence and a numerical state embedding matrix. The text embedding sequence and the numerical state embedding matrix are concatenated along the sequence dimension to generate an input embedding sequence, which is then input into the large language model. The original language modeling head of the pre-trained large language model is replaced with the network action head. The network action head is then used to perform linear transformations to calculate the logical value vector of each action in the action space. The logical value vector is normalized using the Softmax function to obtain the probability distribution of each effective action, and a greedy strategy is adopted to select the action with the highest probability from among the effective actions as the decision output.

3. The method according to claim 1, characterized in that, The process of collecting state-action trajectory data generated during the interaction between the heuristic algorithm and the environment to generate diverse offline experience pools includes: A large number of heterogeneous network environment configurations are generated by a configurable environment simulator, wherein the large number of heterogeneous network environment configurations correspond to the complete actual network state space; By running multiple heuristic algorithms through a multi-algorithm data collector, the decision-making behavior of the multiple heuristic algorithms under various network environment configurations is collected, and complementary original expert trajectories are generated. The original expert trajectories are standardized, structured, and quality balanced using the experience base manager, outputting a dataset for supervised fine-tuning.

4. The method according to claim 1, characterized in that, The supervised fine-tuning of the large language model that has been connected to the general interaction interface through the offline experience pool includes: The large language model that has been connected to the general interaction interface is initialized. A trainable low-rank decomposition matrix is ​​added to the attention mechanism module of the large language model and the original relevant parameters of the large language model are frozen. The parameters of the network action head are set to a trainable state. Trajectory data is randomly sampled from the offline experience pool, and each trajectory data is constructed as a supervised fine-tuning sample; A supervised learning objective is generated by imitating a learning objective optimizer, wherein the supervised learning objective is used to maximize the log likelihood of the large language model to the expert actions in the offline experience pool. The imitation learning objective optimizer sets the basic loss term of the same batch of data as the classification cross-entropy loss and introduces a KL divergence regularization term to constrain the fine-tuning process. Supervised fine-tuning of the large language model is performed based on the supervised fine-tuning samples and the supervised learning objective.

5. The method according to claim 1, characterized in that, The online reinforcement learning of the fine-tuned large language model using the NetGRPO algorithm includes: Using a tree-structured trajectory generation mechanism, starting from the given current environment state, a search tree of a preset depth is constructed, and the cumulative reward of different action sequences is evaluated through environment simulation. Dynamic depth-aware pruning is performed at each layer of the search tree, and for each parent node, several child nodes with scores higher than the score threshold are retained for subsequent expansion; The action value function estimate of the initial action is calculated by aggregating the cumulative rewards of all paths originating from the same initial action; Action groups are formed based on the estimated values ​​of the action value function of all actions under the same state. The relative advantage within the action group is calculated, and the policy parameters are updated based on the relative advantage within the group to achieve grouping policy optimization.

6. The method according to claim 1, characterized in that, The process of deploying the optimized large language model in a real-world network system and making online real-time decisions based on the trained model weights includes: The optimized large language model is transformed into an inference service engine in the network control plane using a lightweight model inference engine. The abstract decision actions output by the inference service engine are transformed into control commands that can be executed by the network infrastructure through the decision-execution interface.

7. A network decision-making system based on reinforcement learning, characterized in that, Includes the following modules: A construction module is used to build a general interaction interface between the large language model and the network environment. Based on the general interaction interface, the large language model can perceive the state of the network environment and output deterministic decision actions. The general interaction interface includes a hybrid embedding module and a network action head. The generation module is used to run existing heuristic algorithms, collect state-action trajectory data generated during the interaction between the heuristic algorithms and the environment, and generate diverse offline experience pools. The fine-tuning module is used to perform supervised fine-tuning of the large language model that has been connected to the general interaction interface through the offline experience pool, including optimizing the model parameters by using low-rank adaptation technology, so that the large language model can acquire basic decision-making capabilities and retain pre-trained knowledge. The reinforcement learning module is used to perform online reinforcement learning on the fine-tuned large language model using the NetGRPO algorithm to optimize the model's decision-making strategy. The NetGRPO algorithm includes tree-structured trajectory generation, action value estimation based on path aggregation, and grouping strategy optimization. The decision-making module is used to deploy the optimized large language model in the actual network system and make online real-time decisions based on the trained model weights.

8. The system according to claim 7, characterized in that, The building module is specifically used for: The hybrid embedding module preprocesses the multimodal input, aligns and fuses the preprocessed multimodal input, and generates a text embedding sequence and a numerical state embedding matrix. The text embedding sequence and the numerical state embedding matrix are concatenated along the sequence dimension to generate an input embedding sequence, which is then input into the large language model. The original language modeling head of the pre-trained large language model is replaced with the network action head. The network action head is then used to perform linear transformations to calculate the logical value vector of each action in the action space. The logical value vector is normalized using the Softmax function to obtain the probability distribution of each effective action, and a greedy strategy is adopted to select the action with the highest probability from among the effective actions as the decision output.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the reinforcement learning-based network decision-making method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the network decision-making method based on reinforcement learning as described in any one of claims 1-6.

Images