Wireless network intelligent agent dynamic reward generation method and system

Abstract

The application provides a wireless network intelligent agent dynamic reward generation method and system, belonging to the technical field of wireless network optimization, comprising: obtaining natural language optimization targets and network node performance index data; inputting the same to a large language model to obtain target semantic embedding vectors and performance index weight vectors, and calculating local rewards of target nodes according to the same; constructing a network topology graph based on network nodes and associated relationships, inputting the topology graph and the semantic embedding vectors into a graph neural network to obtain regional rewards; and fusing the local rewards and the regional rewards to generate a comprehensive reward signal. The application dynamically generates index weights through a large language model, captures interference and synergy effects through a graph network that perceives topology, makes the comprehensive reward signal take into account both local performance improvement and global network stability, and effectively improves the evaluation accuracy of network optimization and the global resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of wireless network optimization technology, and in particular to a method and system for generating dynamic rewards for wireless network agents. Background Technology

[0002] In mobile communication networks, in order to meet the ever-increasing user experience and complex service requirements, it is usually necessary to dynamically adjust and optimize the network parameters of base stations or cells to maintain the efficient and stable operation of the network.

[0003] In the intelligent optimization of wireless networks, reinforcement learning agents are often used to adjust network parameters. In existing technologies, reinforcement learning agents typically evaluate the effect of parameter adjustments and update strategies based on a preset reward function. This reward function generally uses manually fixed weights to linearly combine the performance indicators of individual network nodes, such as throughput and latency, to generate reward signals to guide the agent's learning.

[0004] Existing technologies employ a single-node performance evaluation method with manually set fixed weights. This results in the reward function being unable to dynamically adjust the evaluation focus according to different natural language business objectives. Furthermore, it only evaluates single-point indicators while ignoring the interference and collaborative effects of parameter adjustments on adjacent network nodes. Consequently, the reward signal cannot accurately balance local performance requirements with global network stability, leading to low evaluation accuracy and low global resource utilization efficiency in network optimization. Summary of the Invention

[0005] This invention provides a method and system for generating dynamic rewards for intelligent agents in wireless networks, which addresses the shortcomings of existing technologies and enables the comprehensive reward signal to simultaneously improve local performance and global network stability, effectively enhancing the evaluation accuracy of network optimization and the efficiency of global resource utilization.

[0006] This invention provides a method for generating dynamic rewards for intelligent agents in wireless networks, comprising the following steps: Obtain the natural language optimization target and the performance index data of multiple network nodes, including the target node to be optimized; The natural language optimization objective is input into a large language model to obtain the target semantic embedding vector and performance index weight vector output by the large language model. The local reward of the target node is obtained based on the performance index weight vector and the performance index data of the target node. The network topology graph and the target semantic embedding vector are input into a graph neural network to obtain the region reward output by the graph neural network; the network topology graph is constructed based on the multiple network nodes and the relationships between the network nodes. The local reward and the regional reward are merged to generate a comprehensive reward signal.

[0007] According to the present invention, a method for generating dynamic rewards for wireless network agents includes inputting a network topology graph and the target semantic embedding vector into a graph neural network to obtain the region reward output by the graph neural network, comprising: The target semantic embedding vector can be concatenated as a global feature to the node features of each network node in the network topology graph, or the target semantic embedding vector can be added as a global additional node to the network topology graph. The hidden representation of each network node in the network topology graph is updated by weighting and summarizing the node features of each network node and its neighboring nodes using the graph neural network based on edge weights. The regional reward is generated through aggregation operations based on the updated hidden representation of the target node and its neighboring nodes within a preset hop range.

[0008] According to the present invention, a method for generating dynamic rewards for wireless network agents is provided, wherein the edges in the network topology graph include a first type of edge representing geographical adjacency and a second type of edge representing frequency interference. The step of updating the hidden representation of each network node in the network topology graph by weighting and summarizing the node features of each network node and its neighboring nodes using the graph neural network based on edge weights includes: During message transmission, feature information of the network node's neighboring nodes is extracted; Based on the edge weight coefficients of the corresponding edges in the network topology graph, the feature information of the neighboring nodes is weighted and aggregated to obtain aggregated feature information; The aggregated feature information is fused with the current hidden representation of the network node, and a nonlinear transformation is performed on the fusion result to output the updated hidden representation.

[0009] According to the present invention, a method for generating dynamic rewards for a wireless network agent includes obtaining the local reward of the target node based on the performance index weight vector and the performance index data of the target node, comprising: After performing parameter adjustment actions on the target node, the indicator reward score is obtained based on the performance indicator weight vector and the change in the performance indicator data of the target node before and after the parameter adjustment actions are performed. Based on the performance index data of the target node at the first time point before the parameter adjustment action is executed, the first time point after the execution, and the second time point after the execution, calculate the trend reward score. The local reward is generated by weighting and combining the indicator reward score with the trend reward score.

[0010] According to the present invention, a method for generating dynamic rewards for a wireless network agent includes calculating a trend reward score based on performance index data of the target node before, at a first time point after, and at a second time point after the parameter adjustment action. Calculate the difference between the performance index data at the first time point after execution and the performance index data before execution to obtain the instantaneous performance change. The difference between the performance index data at the second time point after execution and the performance index data at the first time point after execution is calculated to obtain the continuous trend change amount; The trend reward score is obtained by weighted summing of the instantaneous performance change and the continuous trend change.

[0011] According to a method for generating dynamic rewards for a wireless network agent provided by the present invention, before obtaining the local reward of the target node based on the performance index weight vector and the performance index data of the target node, the method further includes: Obtain historical network performance metrics data within a preset time window; The distribution characteristics of each of the historical network performance index data are analyzed using unsupervised clustering algorithms or probability distribution fitting algorithms. Extract the boundary points between cluster center values ​​or the inflection points of the mixed probability distribution, and determine the values ​​corresponding to the boundary points or inflection points as adaptive thresholds; the adaptive thresholds are used to measure the improvement level of the performance index data when calculating the local reward.

[0012] According to the present invention, a method for generating dynamic rewards for a wireless network agent includes fusing the local reward with the regional reward to generate a comprehensive reward signal, comprising: The local reward is normalized and mapped to convert it into a preset numerical range. The regional reward is normalized and mapped to convert it into a preset numerical range. The normalized local reward and the normalized regional reward are linearly weighted according to the preset fusion weight coefficient to generate the comprehensive reward signal.

[0013] According to the present invention, a method for generating dynamic rewards for a wireless network agent, after fusing the local reward and the regional reward to generate a comprehensive reward signal, further includes: The comprehensive reward signal is input into the reinforcement learning model to update the action policy of the reinforcement learning agent; The comprehensive reward signal is compared with a preset decision threshold to generate a status signal; wherein, when the comprehensive reward signal is greater than the decision threshold, a pass status signal is generated; when the comprehensive reward signal is not greater than the decision threshold, a fail status signal is generated. The state signal is output as a discrete evaluation signal independent of the comprehensive reward signal.

[0014] The present invention also provides a dynamic reward generation system for wireless network intelligent agents, comprising the following modules: The first processing module is used to obtain the natural language optimization target and the performance index data of multiple network nodes, wherein the multiple network nodes include the target node to be optimized. The second processing module is used to input the natural language optimization target into the large language model and obtain the target semantic embedding vector and performance index weight vector output by the large language model. The third processing module is used to obtain the local reward of the target node based on the performance index weight vector and the performance index data of the target node. The fourth processing module is used to input the network topology graph and the target semantic embedding vector into the graph neural network to obtain the region reward output by the graph neural network; the network topology graph is constructed based on the multiple network nodes and the relationships between the network nodes; The fifth processing module is used to merge the local reward and the regional reward to generate a comprehensive reward signal.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic reward generation method for wireless network intelligent agents as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic reward generation method for wireless network agents as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the dynamic reward generation method for wireless network intelligent agents as described above.

[0018] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By acquiring natural language optimization objectives and performance index data from multiple network nodes, a dual foundation of semantic information and real-time network data is provided for reward generation. The natural language optimization objectives are input into a large language model, yielding the target semantic embedding vector and performance index weight vector output by the model. This allows the performance index weights to be dynamically updated as the optimization objectives change, avoiding the parameter tuning lag problem caused by manually fixing weights. The local reward for the target node is obtained based on the performance index weight vector and the target node's performance index data, enabling dynamic adaptation of the reward function to different business scenarios. A network topology graph is constructed based on multiple network nodes and their interrelationships. This graph, along with the target semantic embedding vector, is input into a graph neural network to obtain the regional reward output by the network. This regional reward captures interference propagation and collaborative gain effects across network nodes, avoiding evaluation bias that focuses only on single-point indicators while ignoring regional linkages. Furthermore, the injection of the target semantic embedding vector ensures consistency between regional and local rewards in terms of target orientation. By fusing local rewards with regional rewards to generate a comprehensive reward signal, the comprehensive reward signal can simultaneously take into account local performance improvement and global network stability, effectively improving the evaluation accuracy of network optimization and the efficiency of global resource utilization. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the closed-loop process of wireless network reinforcement learning provided by the present invention.

[0021] Figure 2 This is a flowchart illustrating the dynamic reward generation method for wireless network intelligent agents provided by the present invention.

[0022] Figure 3 This is a schematic diagram comparing the performance optimization effects of the target node provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the network topology for regional performance modeling based on graph neural networks provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the structure of the wireless network intelligent agent dynamic reward generation system provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships according to the accompanying drawings, are only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0028] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] The following is combined with Figures 1 to 6 This invention describes the method, system, electronic device, storage medium, and computer program product for generating dynamic rewards for wireless network intelligent agents.

[0030] The dynamic reward generation method for wireless network intelligent agents described in this invention is executed by a processing device with computing capabilities in real-world scenarios. In one embodiment, the executing entity is a network optimization server deployed in a wireless network management system. The network optimization server communicates with the network management system via a data interface. The network optimization server obtains operational data from each base station and cell from the network management system and integrates a large language model inference engine and a graph neural network inference engine internally to complete all computational tasks for reward generation. In another embodiment, the executing entity is an edge computing node deployed at the network edge. Edge computing nodes possess local inference capabilities, making them suitable for latency-sensitive optimization scenarios. In yet another embodiment, the executing entity is a distributed processing cluster in a cloud-based network optimization platform. The distributed processing cluster undertakes the reward generation tasks for a large number of network nodes through parallel computing.

[0031] The following embodiments use a network optimization server as the execution subject to provide a detailed description of the technical solution of the present invention.

[0032] Reference Figure 1 , Figure 1 This is a schematic diagram of the closed-loop process of wireless network reinforcement learning provided by the present invention. Figure 1 As shown, the dynamic reward generation method for wireless network agents of the present invention is located in the effect evaluation stage of the reinforcement learning closed-loop process. The reinforcement learning closed-loop process includes six stages in sequence: problem identification, root cause localization, large language model solution generation, solution execution, effect evaluation, and optimization completion. The output results of the effect evaluation stage are fed back to the large language model solution generation stage, thus forming a closed loop of continuous iterative optimization. The dynamic reward generation method for wireless network agents of the present invention is executed in the effect evaluation stage, providing a comprehensive reward signal for the reinforcement learning agent.

[0033] Reference Figure 2 , Figure 2 This is a flowchart illustrating the dynamic reward generation method for wireless network intelligent agents provided by the present invention, as shown below. Figure 2 As shown, the method for generating dynamic rewards for wireless network agents includes steps 110 to 150: Step 110: Obtain the natural language optimization target and the performance index data of multiple network nodes, including the target node to be optimized.

[0034] Specifically, natural language optimization goals are network optimization requirements described in natural language text. In wireless network operation and maintenance scenarios, the sources of natural language optimization goals include operation and maintenance work order descriptions, root cause analysis results, and the results of the previous round of solution execution. For example, one natural language optimization goal might be "weak coverage problem, excessive inter-site spacing, adjust the maximum transmit power from -10 to -5"; another might be "increase throughput and appropriately reduce latency." Natural language optimization goals implicitly convey the operator's focus and constraints regarding the network optimization effect.

[0035] Multiple network nodes correspond to base stations or cells in a wireless network. Each network node has corresponding performance metrics data, which encompass multi-dimensional measures reflecting the network's operational status. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram comparing the performance optimization effects of the target node provided by the present invention. For example... Figure 3 As shown, in wireless network optimization scenarios, performance metrics include, but are not limited to, throughput, latency, wireless drop rate, Flow drop rate, RRC connection reconstruction ratio, handover success rate, wireless connection success rate, QoS Flow establishment success rate, and EPSFallback success rate. Performance metrics are categorized into two types based on their direction: the first type represents metrics where lower values ​​indicate better performance, including wireless drop rate, Flow drop rate, and RRC connection reconstruction ratio; the second type represents metrics where higher values ​​indicate better performance, including handover success rate, wireless connection success rate, QoS Flow establishment success rate, and EPSFallback success rate.

[0036] The network optimization server collects performance index data of multiple network nodes from the network management system through a data interface, and simultaneously receives optimization targets in natural language input in text form.

[0037] Step 120: Input the natural language optimization target into the large language model and obtain the target semantic embedding vector and performance index weight vector output by the large language model.

[0038] Specifically, the language model is a large-scale pre-trained language model based on the Transformer architecture. During the training phase, the large language model learns semantic understanding and reasoning capabilities from massive text corpora. After domain adaptation training in the field of communication networks, the large language model has the ability to understand network optimization terminology and business logic.

[0039] The large language model receives input containing three parts: work order description text, root cause analysis results text, and solution execution results text. The large language model performs semantic parsing and intent understanding on the input natural language optimization objective, extracting the implicit optimization focus and constraints within the natural language optimization objective.

[0040] The output of a large language model includes two items: The first term is the target semantic embedding vector. The target semantic embedding vector is a fixed-dimensional vector representation generated by the large language model after encoding the natural language optimization objective. It condenses complex human optimization requirements into a machine-processable numerical vector, representing the meaning of the natural language optimization objective in the semantic space. In subsequent steps, the target semantic embedding vector is injected into the graph neural network to adjust the network's evaluation bias for region performance.

[0041] The second item is the performance metric weight vector. This vector contains the weight coefficients of each performance metric in the reward function. For example, when the natural language optimization objective emphasizes "improving throughput and moderately reducing latency," the large language model interprets throughput as relatively more important, outputting a performance metric weight vector of "throughput: 0.5, latency: 0.2, wireless drop rate: 0.15, ...". When the natural language optimization objective mentions "ensuring video streaming service quality," the large language model identifies that latency and jitter metrics need strict control, thus increasing the weight of latency and jitter in the performance metric weight vector.

[0042] In the above process, the large language model acts as a translator, transforming the user's optimization objective in natural language form into numerical parameters that the machine can directly use, namely, the target semantic embedding vector and the performance index weight vector. This approach differs from traditional methods that rely on manually set fixed weights; the performance index weight vector is dynamically updated as the natural language optimization objective changes.

[0043] Step 130: Obtain the local reward of the target node based on the performance index weight vector and the performance index data of the target node.

[0044] Specifically, local rewards reflect the performance improvements of the target node itself.

[0045] In the specific calculation process, the network optimization server extracts the weight coefficients corresponding to each performance indicator from the performance indicator weight vector, combines them with the performance indicator data of the target node, and obtains a reward value reflecting the performance level of the target node through weighted calculation. When the weight coefficient of a certain performance indicator in the performance indicator weight vector is large, that performance indicator occupies a larger proportion in the local reward calculation, thus making the local reward more focused on reflecting the improvement effect of that performance indicator.

[0046] The calculation of local rewards is limited to the performance metrics data of the target node itself, and does not involve the performance status of the target node's neighboring network nodes.

[0047] Step 140: Input the network topology graph and the target semantic embedding vector into the graph neural network to obtain the region reward output by the graph neural network; the network topology graph is constructed based on multiple network nodes and the relationships between them.

[0048] Specifically, a network topology graph is a graphical representation of the wireless network structure. Each node in the graph corresponds to a network node, and each edge represents the relationship between two network nodes. These relationships reflect the mutual influence between base stations or cells in the wireless network. Specific types of relationships include geographical adjacency, frequency interference, and load-related relationships. The initial node characteristics of each network node in the topology graph are determined based on its performance metrics; that is, each node's initial characteristics include state information such as throughput, latency, and recent performance changes.

[0049] The network optimization server inputs the network topology graph and the target semantic embedding vector into the graph neural network to obtain the region reward output by the graph neural network. Graph neural networks are a type of deep learning model specifically designed to process graph-structured data. Through an iterative message-passing mechanism, graph neural networks propagate and aggregate information between adjacent nodes in the graph structure, thereby learning the representation of each node in the context of the graph structure.

[0050] In this invention, a graph neural network encodes the network topology graph, so that the representation of each network node not only includes the performance state of the network node itself, but also incorporates the performance information of its neighboring network nodes. (Refer to...) Figure 4 , Figure 4 This is a schematic diagram of the network topology for regional performance modeling based on graph neural networks, provided by this invention. The network optimization server constructs the network topology map based on the distribution of physical cells in the wireless network. Figure 4 In the network topology diagram shown, the central node represents the target node to be optimized, i.e., the problem cell with a performance bottleneck. The target node is associated with the regional performance score output by the graph neural network, such as... Figure 4 The center value displayed by the center node is 0.48.

[0051] Other nodes connected to the target node represent neighboring network nodes in the surrounding environment. The network optimization server classifies neighboring network nodes into high-impact and low-impact neighboring areas based on adjacency and interference levels. In the network topology graph, the target node interacts with one-hop neighboring network nodes through directly connected edges. These one-hop neighboring network nodes further establish connections with two-hop neighboring network nodes, thus forming a multi-hop topology.

[0052] The network optimization server inputs the constructed network topology graph into the graph neural network to perform feature extraction and aggregation operations. During message passing in the graph neural network, the network optimization server assigns edge weight coefficients differently based on the type of neighboring network nodes. The network optimization server assigns larger edge weight coefficients to high-impact neighbors and smaller edge weight coefficients to low-impact neighbors. The graph neural network uses a multi-layer message passing mechanism to progressively weight and aggregate the performance index data of high-impact and low-impact neighbors to the target node.

[0053] Finally, the network optimization server outputs the regional reward for the target node through a graph neural network. The regional reward precisely quantifies the comprehensive impact of the parameter adjustment actions performed by the target node on the entire local topology network.

[0054] The target semantic embedding vector participates in the computation process of the graph neural network as a global feature, thereby adjusting the graph neural network's judgment bias on regional performance. For example, when the natural language optimization objective emphasizes low latency, the target semantic embedding vector influences the graph neural network to pay more attention to latency-related neighbor state information, thus reflecting the importance of latency factors in regional rewards. By introducing the target semantic embedding vector, the same set of graph neural network parameters can dynamically adjust the evaluation criteria for different natural language optimization objectives.

[0055] The training methods for graph neural networks include the following two types: The first is supervised learning, which uses the performance changes of simulation environment or historical parameter change data as training labels to train the graph neural network to predict regional performance indicators or directly output regional reward values; the second is reinforcement learning collaborative training, which optimizes the graph neural network and the policy network of the reinforcement learning agent together, so that the regional reward output of the graph neural network is consistent with the policy optimization direction.

[0056] Regional rewards reflect the impact of a target node's parameter adjustments on the performance of its neighboring network nodes and even the overall local network. When the target node's parameter adjustments improve the performance metrics of both the target node itself and its neighboring network nodes, the regional reward output by the graph neural network increases; conversely, when the target node's parameter adjustments lead to a deterioration in the performance of its neighboring network nodes, the regional reward output by the graph neural network decreases.

[0057] Step 150: Merge local rewards and regional rewards to generate a comprehensive reward signal.

[0058] Specifically, the comprehensive reward signal is a single scalar value, which is fed back to the reinforcement learning agent to drive the agent to update its action policy.

[0059] During the fusion process, the network optimization server, following a pre-defined fusion strategy, combines local rewards reflecting the performance improvement of the target node itself with regional rewards reflecting changes in the overall performance of the region. The fusion strategy determines the degree to which the combined reward signal favors local performance over global performance.

[0060] Through steps 110 to 150 above, the present invention achieves the following technical effects: The large language model extracts semantic intent from the natural language optimization objective and dynamically generates performance index weight vectors, enabling the weight configuration of the reward function to be adjusted in real time according to the network scenario and optimization requirements, avoiding the parameter tuning lag problem caused by changes in the optimization objective when the weights are manually fixed in the traditional way; The graph neural network incorporates the performance status of neighboring network nodes into the regional reward evaluation by modeling the network topology, so that the reward signal can reflect the impact of parameter adjustment on the overall performance of the local network, avoiding the problem of only focusing on single-point indicators and ignoring the chain effect across network nodes in the traditional way; The target semantic embedding vector is injected into the graph neural network to ensure that the regional reward evaluation maintains the same evaluation tendency as the natural language optimization objective, realizing customized reward evaluation on demand; The fusion of local rewards and regional rewards enables the comprehensive reward signal to take into account both local optimization effects and global network stability, guiding the reinforcement learning agent to perform parameter optimization while ensuring the overall network performance.

[0061] Based on the foregoing embodiments, this embodiment further specifies that an adaptive threshold generation step is performed in advance before calculating the local reward.

[0062] Step 101: Obtain historical network performance index data within the preset time window.

[0063] Specifically, the length of the preset time window is set according to data accumulation needs. For example, the preset time window can be set to the most recent 7 days or the most recent 30 days. Historical network performance indicator data covers various performance indicators collected from multiple network nodes at multiple time sampling points within the preset time window, including multi-dimensional metrics such as throughput, latency, packet loss rate, and disconnection rate. The network optimization server periodically performs the historical network performance indicator data acquisition operation, with the frequency set to once daily or once weekly, thereby ensuring that the historical network performance indicator data is continuously updated as the network operating conditions change.

[0064] Step 102: Analyze the distribution characteristics of various historical network performance index data using unsupervised clustering algorithms or probability distribution fitting algorithms.

[0065] Specifically, when using unsupervised clustering algorithms, the network optimization server applies the K-means clustering algorithm to the historical data of each performance metric. The K-means clustering algorithm automatically divides the historical data of each performance metric into a preset number of clusters, typically three, corresponding to high-performance, medium-performance, and low-performance clusters, respectively. The K-means clustering algorithm iteratively calculates the center value of each cluster and assigns each historical data sample to the cluster containing the nearest cluster center.

[0066] When using a probability distribution fitting algorithm, the network optimization server fits a Gaussian mixture distribution model to the historical data of each performance indicator. The Gaussian mixture distribution model treats the historical data of each performance indicator as a mixture of multiple Gaussian components, and estimates the mean, variance, and mixture weight parameters of each Gaussian component using the expectation-maximization algorithm.

[0067] In addition, the network optimization server also uses quantile analysis as an auxiliary means of distribution characteristic analysis. In quantile analysis, the network optimization server calculates multiple quantile values ​​for various historical network performance indicators, for example, calculating the 90th percentile value and the 10th percentile value.

[0068] Step 103: Extract the boundary points between cluster center values ​​or the inflection points of the mixed probability distribution, and determine the values ​​corresponding to the boundary points or inflection points as adaptive thresholds; the adaptive thresholds are used to measure the improvement level of performance index data when calculating local rewards.

[0069] Specifically, when analyzing the results based on unsupervised clustering algorithms, the network optimization server calculates the midpoint between the high-performance cluster center value and the medium-performance cluster center value, and determines the value corresponding to the midpoint as the upper limit of the adaptive threshold for distinguishing between high performance and medium performance; the network optimization server calculates the midpoint between the medium-performance cluster center value and the low-performance cluster center value, and determines the value corresponding to the midpoint as the lower limit of the adaptive threshold for distinguishing between medium performance and low performance.

[0070] When analyzing the results based on the probability distribution fitting algorithm, the network optimization server calculates the intersection point of two adjacent Gaussian components in the Gaussian mixture distribution model. This intersection point represents the inflection point of the probability mixture distribution. The network optimization server then determines the value corresponding to the inflection point as the adaptive threshold.

[0071] When based on the results of quantile analysis, the network optimization server determines the 90th percentile value as the "excellent" adaptive threshold and the 10th percentile value as the "poor" adaptive threshold.

[0072] For example, if large-scale historical network performance data shows that network nodes in a certain city area generally have a throughput of over 50Mbps during peak hours and mostly below 10Mbps during off-peak hours, the network optimization server will automatically define throughput greater than 50Mbps as the "high performance" adaptive threshold, throughput less than 10Mbps as the "low performance" adaptive threshold, and throughput between 10Mbps and 50Mbps as the "medium performance" range.

[0073] The adaptive threshold plays a role in calculating local rewards. Specifically, it measures the degree of improvement in the target node's performance metrics. After a target node performs parameter adjustments, the network optimization server compares the change in its performance metrics with the adaptive threshold to determine whether the improvement is minor, significant, or satisfactory, thus assigning different reward scores to the local rewards. In areas with poor overall performance, a small improvement that crosses the adaptive threshold's level boundary receives a positive reward; conversely, in high-performance areas, an improvement of the same magnitude that does not cross the adaptive threshold's level boundary receives a lower reward.

[0074] like Figure 3 As shown, the "Upper Limit (AI)" and "Lower Limit (AI)" columns in the indicator details are the outputs of the adaptive threshold automatic generation mechanism. For example, the manually set upper limit for wireless drop rate is 2, while the upper limit output by the adaptive threshold automatic generation mechanism is 1.933; the manually set lower limit for handover success rate is 95, while the lower limit output by the adaptive threshold automatic generation mechanism is 95.01. The adaptive threshold is dynamically generated based on the distribution characteristics of historical network performance indicator data, and is closer to the actual benchmark of the current network operating status compared to manually set fixed thresholds.

[0075] By automatically generating adaptive thresholds from historical network performance data within a preset time window, this invention achieves a closed-loop mechanism for automatically calibrating evaluation standards as the network environment evolves. The adaptive thresholds eliminate the problem of inaccurate evaluation standards caused by expert experience bias or network changes when manually setting fixed thresholds, ensuring that network nodes in different regions and time periods receive evaluation standards that align with their actual operating benchmarks. The periodically updated adaptive thresholds ensure that the evaluation standards are adjusted upwards when overall network performance improves and downwards when network performance declines due to external factors, guaranteeing that reinforcement learning agents can receive discriminative reward signals under any network conditions.

[0076] Based on the aforementioned embodiments, this embodiment further defines the specific implementation method of local reward calculation, and decomposes the calculation of local reward into three steps: indicator reward score calculation, trend reward score calculation, and weighted combination.

[0077] Step 131: After performing parameter adjustment actions on the target node, obtain the indicator reward score based on the performance indicator weight vector and the change in performance indicator data of the target node before and after the parameter adjustment actions are performed.

[0078] The network optimization server collects performance index data of the target node before and after the parameter adjustment action is performed, and calculates the difference between the two to obtain the change in performance index data.

[0079] For performance metrics where lower values ​​indicate better performance, the network optimization server subtracts the value before execution from the value after execution to obtain the change in performance metric data, ensuring that the change in performance metric data is positive when performance improves. For performance metrics where higher values ​​indicate better performance, the network optimization server subtracts the value before execution from the value after execution to obtain the change in performance metric data, ensuring that the change in performance metric data is also positive when performance improves.

[0080] The network optimization server extracts the weight coefficients corresponding to each performance indicator from the performance indicator weight vector. It then multiplies the change in performance indicator data for each indicator by its corresponding weight coefficient and sums the results to obtain the indicator reward score. The indicator reward score reflects the overall improvement effect of parameter adjustments on the target node's various performance indicators.

[0081] like Figure 3 As shown, for example, the target node's wireless disconnection rate changed from 0.8 before optimization to 2.2 after optimization, which is higher than the upper limit of 2. Since a lower wireless disconnection rate is generally better, the performance metric data change is 0.8 - 2.2 = -1.4, indicating performance degradation. Similarly, the handover success rate changed from 95.6 before optimization to 93.7 after optimization, which is lower than the lower limit of 95. Since a higher handover success rate is generally better, the performance metric data change is 93.7 - 95.6 = -1.9, also indicating performance degradation. The network optimization server calculates the weighted sum of the changes in each performance metric data according to the corresponding weight coefficients in the performance metric weight vector, thus obtaining the metric reward score.

[0082] Step 132: Calculate the trend reward score based on the performance index data of the target node before, at the first time point after, and at the second time point after the parameter adjustment action.

[0083] Specifically, the design considerations for trend-based reward scores are as follows: the calculation of local rewards not only focuses on the performance value at a single moment, but also needs to consider whether the solution is effective after implementation and whether there is a trend of improvement or deterioration over a period of time after implementation. For example, if the throughput of a network node continues to increase and the latency continues to decrease after parameter adjustments, even if the absolute values ​​have not yet reached the normal range, it should still be positively evaluated in the reward assessment. This trend-based assessment ensures that the reward signal focuses on the direction of improvement brought about by the optimization action, rather than merely measuring the static level.

[0084] Trend reward score The trend reward score indicates that it reflects the changing trend of the target node's performance metrics data over time.

[0085] Step 133: Weight the indicator reward score and the trend reward score to generate a local reward.

[0086] Specifically, the formula for calculating local rewards is as follows: In the above formula, This represents the local reward of the target node v. This represents the reward score for the target node v. Let α represent the trend reward score of the target node v, and let α be the weighting hyperparameter of the trend reward. The weighting hyperparameter α controls the proportion of the trend reward score in the local reward, and the value of the weighting hyperparameter α is determined according to the needs of the actual network optimization scenario. When the weighting hyperparameter α is large, the local reward focuses more on reflecting the trend direction of performance changes; when the weighting hyperparameter α is small, the local reward focuses more on reflecting the performance improvement at the current moment.

[0087] Based on the aforementioned embodiments, this embodiment further defines the specific calculation process for the trend reward score.

[0088] The network optimization server defines three time observation points: T0 is the time point before the parameter adjustment action is executed, T1 is the first time point after the parameter adjustment action is executed, and T2 is the second time point after the parameter adjustment action is executed. The first time point T1 is the immediate observation time after the parameter adjustment action is completed. The second time point T2 is the observation time after a preset time interval has elapsed since the parameter adjustment action was completed. The default value for the preset time interval is 30 minutes, and the preset time interval can be adjusted according to actual operational needs.

[0089] The network optimization server calculates the difference between the performance metric data at time point T1 after execution and the performance metric data at time point T0 before execution, thus obtaining the instantaneous performance change. The formula for calculating the instantaneous performance change is as follows: In the above formula, This represents the performance metric data value of the target node at the first time point T1. This represents the performance metric data value of the target node at time point T0 before execution. This indicates the immediate change in performance. The immediate change in performance reflects the immediate effect of parameter adjustments after they are executed.

[0090] The network optimization server calculates the difference between the performance metric data at the second time point T2 after execution and the performance metric data at the first time point T1 after execution, and obtains the continuous trend change. The formula for calculating the continuous trend change is as follows: In the above formula, This represents the performance index value of the target node at the second time point T2. This represents the amount of change in a sustained trend. The amount of change in a sustained trend reflects the further evolution of performance metrics over a period of time after parameter adjustments are performed; that is, whether performance continues to improve, remains stable, or deteriorates.

[0091] The network optimization server performs a weighted sum of immediate performance changes and sustained trend changes to obtain a trend reward score. The formula for calculating the trend reward score is as follows: In the above formula, Real-time performance change The weighting coefficients, For continuous trend change Weighting coefficients. and weighting coefficients The value is determined based on the operation and maintenance strategy. When the weighting coefficient... Greater than the weight coefficient At that time, the trend reward score focuses more on reflecting the immediate effectiveness of parameter adjustment actions; when the weight coefficient Greater than the weight coefficient In this case, the trend reward score focuses more on reflecting the long-term and sustained effects of parameter adjustment actions.

[0092] For example, if the throughput of the target node is 50Mbps at time point T0, 55Mbps at the first time point T1, and 58Mbps at the second time point T2, then the instantaneous performance change is... Continuous trend change When the weighting coefficient And weighting coefficients At that time, the trend reward score A positive trend reward score indicates that the throughput of the target node shows a continuous upward trend after the parameter adjustment action is executed.

[0093] Based on the foregoing embodiments, this embodiment further specifies the specific implementation method of inputting the network topology graph and the target semantic embedding vector into the graph neural network and obtaining the region reward output by the graph neural network, including the following steps: Step 141: Concatenate the target semantic embedding vector as a global feature into the node features of each network node in the network topology graph, or connect the target semantic embedding vector as a global additional node to the network topology graph.

[0094] Specifically, the network optimization server injects the target semantic embedding vector into the network topology graph, and the injection methods include the following two: The first injection method is feature splicing.

[0095] The network optimization server uses the target semantic embedding vector as a global feature and concatenates it into the node features of each network node in the network topology graph. Specifically, for each network node in the network topology graph, the network optimization server performs a vector concatenation operation between the target semantic embedding vector and the initial node features of the corresponding network node to generate an enhanced node feature vector. The enhanced node feature vector contains both the performance index data of the corresponding network node and the semantic information of the natural language optimization target. In this way, all network nodes in the network topology graph obtain contextual information about the current optimization target, enabling the graph neural network to perform feature aggregation guided by the current optimization target during subsequent message passing.

[0096] The second injection method is the global append node method.

[0097] The network optimization server connects the target semantic embedding vector as a global append node to the network topology graph. Connections are established between the global append node and all network nodes in the topology graph. The node features of the global append node are the target semantic embedding vector. During message passing in the graph neural network, each network node receives target semantic information from the global append node, thus integrating the semantic guidance of the optimization target into the hidden representation update process of the network nodes.

[0098] Step 142: Use a graph neural network to perform a weighted aggregation of the node features of each network node and the node features of its neighboring nodes based on edge weights, and update the hidden representation of each network node in the network topology graph.

[0099] Specifically, graph neural networks consist of multiple message-passing layers. In each message-passing layer, each network node receives feature information from its one-hop neighbor network nodes and weights and summarizes the feature information of its neighbor network nodes according to the weights or attributes of the connecting edges, thereby updating its own hidden representation. The stacking of multiple message-passing layers ensures that after the first message-passing layer, the hidden representation of the network node includes information from its direct neighbors; after the second message-passing layer, the hidden representation of the network node further covers information from its two-hop neighbors. And so on, as the number of message-passing layers increases, the hidden representation of the network node gradually propagates and covers a wider range of network structural influences.

[0100] The specific network structure of a graph neural network can be implemented using either a graph convolutional network or a graph attention network. When using a graph convolutional network, the graph convolutional network performs feature propagation and aggregation between adjacent network nodes through graph convolution operations; when using a graph attention network, the graph attention network dynamically calculates the contribution weight of each neighboring network node to the current network node through an attention mechanism, thereby achieving differentiated feature aggregation.

[0101] Step 143: Generate regional rewards through aggregation operations based on the updated hidden representation of the target node and its neighboring nodes within a preset hop range.

[0102] Specifically, the preset hop count range is set to 2 hops. The network optimization server extracts the updated hidden representations of the target node and all neighboring network nodes within a 2-hop range of the target node, and performs aggregation operations on the extracted hidden representations. The aggregation operation can be implemented using weighted summation and attention mechanisms.

[0103] When using a weighted summation method, the formula for calculating regional rewards is as follows: In the above formula, This represents the regional reward for the target node v. This represents the hidden representation of neighboring network node u after being updated by a graph neural network. This represents the set of all neighboring network nodes of the target node v within a 2-hop range. This represents an aggregate function.

[0104] For scenarios involving multiple target nodes, the formula for calculating the total regional reward is as follows: In the above formula, V represents the set of all target nodes. This represents the sum of the regional rewards for all target nodes.

[0105] Based on the foregoing embodiments, this embodiment further defines the types of edges in the network topology graph and the specific calculation method for updating hidden representations during message passing in the graph neural network.

[0106] The edges in the network topology graph include the first type of edges that represent geographical adjacency and the second type of edges that represent frequency interference.

[0107] The first type of edge connects two geographically adjacent network nodes. When the base stations or cells corresponding to two network nodes meet a preset distance condition in terms of geographical location, the network optimization server creates a first-type edge between the corresponding two nodes in the network topology graph. The edge weight coefficient of the first-type edge is determined based on the geographical distance between the two network nodes; the closer the geographical distance between two network nodes, the larger the edge weight coefficient of the first-type edge.

[0108] Type II edges connect two network nodes that interfere with each other on frequency resources. When the base stations or cells corresponding to two network nodes use the same or adjacent frequency resources, and the signals between the two base stations or cells interfere with each other, the network optimization server creates a Type II edge between the corresponding two nodes in the network topology graph. The edge weight coefficient of Type II edges is determined according to the degree of interference between the two network nodes; the stronger the interference, the larger the edge weight coefficient between the two network nodes.

[0109] By distinguishing between the first type of edge and the second type of edge, the network topology graph simultaneously encodes the spatial proximity relationship and frequency interference relationship between network nodes, enabling the graph neural network to consider both types of influencing factors during message passing.

[0110] During message passing in a graph neural network, the network optimization server updates the hidden representations of each network node in the following three stages: Phase 1: Neighbor feature extraction.

[0111] The network optimization server extracts the feature information of all neighboring network nodes of the current network node. The feature information of the neighboring network nodes is a hidden representation of the message passing output of the previous layer. For the first layer message passing, the feature information of the neighboring network nodes is the initial node feature.

[0112] Second stage: weighted aggregation.

[0113] The network optimization server performs weighted aggregation of the feature information of all neighboring network nodes based on the edge weight coefficients of corresponding edges in the network topology graph, resulting in aggregated feature information. During the weighted aggregation process, the feature information of neighboring network nodes with larger edge weight coefficients carries a greater weight in the aggregated feature information. For two network nodes connected by both Type I and Type II edges, the network optimization server applies the edge weight coefficients of Type I and Type II edges to the feature information of the neighboring network nodes respectively, and then superimposes the two weighted results before incorporating them into the aggregated feature information.

[0114] The third stage: fusion and nonlinear transformation.

[0115] The network optimization server fuses the aggregated feature information with the current hidden representation of the current network node. The fusion methods include vector concatenation and vector addition. The network optimization server then performs a nonlinear transformation on the fusion result. This nonlinear transformation is implemented using an activation function, such as ReLU or Tanh. The output of the nonlinear transformation is the updated hidden representation of the current network node.

[0116] The above three stages are repeatedly executed in each layer of message passing in the graph neural network. After multiple layers of message passing, the hidden representations of each network node incorporate the feature information of neighboring network nodes within a multi-hop range.

[0117] Based on the aforementioned embodiments, this embodiment further defines the specific implementation method of merging local rewards and regional rewards into a comprehensive reward signal, including two steps: normalization mapping and linear weighting.

[0118] Step 151: Normalize the local reward to convert it into a preset numerical range.

[0119] Specifically, the normalization mapping is implemented using a sigmoid function. The sigmoid function maps the local reward from any range of real numbers to a numerical range between 0 and 1. The formula for calculating the normalized local reward is as follows: In the above formula, This represents the local reward of the target node v before normalization. Let represent the local reward of the target node v after normalization, and e be the natural constant.

[0120] Through the normalization mapping of the Sigmoid function, regardless of the differences in the dimensions and numerical ranges of various performance indicators, the normalized local reward falls within a uniform numerical range between 0 and 1. Normalization mapping ensures the stability of the local reward signal, preventing numerical instability due to differences in the dimensions of different performance indicators or extreme values.

[0121] In another implementation, the normalization mapping is implemented using the Min-Max normalization method. The network optimization server linearly maps the current local reward to a value range between 0 and 1 based on the maximum and minimum values ​​of the historical local reward.

[0122] Step 152: Normalize the regional rewards to convert them into a preset numerical range.

[0123] The normalization mapping method for regional rewards is consistent with that for local rewards, using either the Sigmoid function or Min-Max normalization. This normalization ensures that regional and local rewards have similar numerical scales, eliminating unfairness caused by differences in units or numerical levels.

[0124] Step 153: Linearly weight the normalized local reward and the normalized regional reward according to the preset fusion weight coefficient to generate a comprehensive reward signal.

[0125] Specifically, the formula for calculating the comprehensive reward signal is as follows: In the above formula, Indicates a comprehensive reward signal. This represents the normalized local reward. This represents the normalized regional reward, and β represents the fusion weight coefficient. The fusion weight coefficient β ranges from 0 to 1.

[0126] The value of the fusion weight coefficient β determines the degree to which the comprehensive reward signal leans towards local performance versus global performance. When the fusion weight coefficient β is close to 1, the comprehensive reward signal is more focused on reflecting the performance improvement of the target node itself, and the reinforcement learning agent tends to pursue the improvement of individual performance; when the fusion weight coefficient β is close to 0, the comprehensive reward signal is more focused on reflecting the changes in the overall performance of the region, and the reinforcement learning agent tends to pursue the coordinated optimization of the global network.

[0127] There are three ways to set the fusion weight coefficient β: The first is the static setting method, where a fixed value of the fusion weight coefficient β is predetermined based on the operator's strategy. The second is the phased adjustment method, where the fusion weight coefficient β is set to a larger value in the early stage of reinforcement learning training so that the reinforcement learning agent can first learn actions to improve basic performance, and then the fusion weight coefficient β is gradually reduced in the later stage of training to guide the reinforcement learning agent to focus on global coordination. The third is the scenario-adaptive method, where the fusion weight coefficient β is temporarily lowered during major events to emphasize overall collaboration, and the fusion weight coefficient β is increased during daily operations to emphasize output efficiency.

[0128] By performing normalization mapping on local and regional rewards separately before linear weighted fusion, this invention eliminates the unfair impact of differences in the dimensions and numerical levels of different performance indicators on the comprehensive reward signal, ensuring that no single performance indicator or single reward component dominates the comprehensive reward signal due to excessively large absolute values. The normalized local and regional rewards fall within the same numerical range, enabling them to be weighted and fused on a unified scale, thus guaranteeing the fairness and consistency of the comprehensive reward signal. The fusion weight coefficient β provides a flexible means of adjusting between local and global performance. Operators can instantly influence the decision-making tendency of the reinforcement learning agent by adjusting the fusion weight coefficient β without retraining the reinforcement learning model.

[0129] Based on the aforementioned embodiments, this embodiment further defines a dual output mechanism after the comprehensive reward signal is generated.

[0130] Step 161: Input the comprehensive reward signal into the reinforcement learning model to update the action policy of the reinforcement learning agent.

[0131] Specifically, the comprehensive reward signal is a continuous real-valued scalar that directly influences the policy parameter updates of the reinforcement learning agent. The agent adjusts its action policy by accumulating the expected reward of the comprehensive reward signal, tending to choose actions with higher expected comprehensive reward values. The comprehensive reward signal plays the role of dense reward in the reinforcement learning model, providing continuous gradient information on the quality of parameter adjustment actions; that is, it not only indicates whether the parameter adjustment action is successful but also reflects how good or bad the action is. A dense comprehensive reward signal is beneficial for improving the stability and convergence speed of reinforcement learning training.

[0132] Step 162: Compare the comprehensive reward signal with the preset decision threshold to generate a status signal; wherein, when the comprehensive reward signal is greater than the decision threshold, a pass status signal is generated; when the comprehensive reward signal is not greater than the decision threshold, a fail status signal is generated.

[0133] Specifically, when the overall reward signal exceeds a preset decision threshold, the network optimization server generates a pass status signal. The pass status signal indicates that the overall performance improvement brought about by the current parameter adjustment action has met operational requirements, and the current parameter adjustment action plan has been determined to be acceptable and applied to the actual network.

[0134] When the overall reward signal is not greater than the preset decision threshold, the network optimization server generates a failure status signal. The failure status signal indicates that the overall performance improvement of the current parameter adjustment action has not met the operational requirements, and even if the overall reward signal is improved, the current parameter adjustment action plan will still not be adopted and deployed.

[0135] For example, the preset decision threshold is set to 0.8. When the comprehensive reward signal is 0.85, the network optimization server generates a pass status signal, indicating that the solution meets the requirements; when the comprehensive reward signal is 0.65, although the comprehensive reward signal is positive and indicates some improvement, the network optimization server still generates a fail status signal, indicating that the improvement is insufficient to pass the operational review.

[0136] The preset decision thresholds can be determined manually or automatically. In the manual setting method, operations personnel directly set the preset decision thresholds based on operational standards. In the automatic setting method, the preset decision thresholds are combined with an adaptive threshold generation mechanism, using historical data statistical analysis to determine reasonable pass criteria, thus making the generation of status signals equally adaptive.

[0137] To illustrate the synergistic effect of dual outputs, consider a complete scenario: Suppose the goal of natural language optimization is to reduce the disconnection rate to below 1%. During training, the reinforcement learning agent performs a parameter adjustment action that reduces the disconnection rate from 5% to 4%. The overall reward signal receives a moderately positive value due to the relative improvement of 20%. The reinforcement learning agent receives positive feedback from the overall reward signal and continues to explore better parameter adjustment actions. Simultaneously, since the disconnection rate of 4% is far from the 1% operational standard, the overall reward signal does not exceed the preset decision threshold, and the network optimization server generates a fail status signal; the parameter adjustment action is not actually deployed. The reinforcement learning agent continues to iterate until it finds a parameter adjustment action that reduces the disconnection rate to below 1%. At this point, the overall reward signal exceeds the preset decision threshold, the network optimization server generates a pass status signal, and the parameter adjustment action is actually deployed. In this process, the reinforcement learning agent does not passively wait for the disconnection rate to reach the target without feedback; instead, it receives encouragement from the overall reward signal each time the disconnection rate is reduced, continuously moving closer to the goal.

[0138] Step 163: Output the state signal as a discrete evaluation signal independent of the comprehensive reward signal.

[0139] Specifically, status signals are discrete, categorized indicator signals. These signals are directly provided to the network control system or maintenance personnel for reference, enabling immediate intervention or adjustment. Status signals can be presented in two forms: a three-level indicator light format ("normal / warning / alarm") and a binary judgment format ("pass / fail").

[0140] The state signal and the overall reward signal are independent of each other: the overall reward signal is input into the reinforcement learning model in continuous numerical form to drive long-term policy optimization, while the state signal is output to the network management interface in discrete form to meet the needs of real-time monitoring and rapid response. The same reward evaluation mechanism supports both long-term policy optimization and immediate deployment decision-making.

[0141] The network optimization server supports two operating modes: the first is the training mode, in which the network optimization server only inputs the comprehensive reward signal into the reinforcement learning model to drive policy updates, and the state signal is only recorded in the log for analysis and does not participate in the actual deployment decision; the second is the deployment mode, in which the comprehensive reward signal and the state signal are used in parallel. The comprehensive reward signal continues to drive the online learning of the reinforcement learning model, while the state signal controls whether the parameter adjustment action is actually applied to the network.

[0142] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of the wireless network intelligent agent dynamic reward generation system provided by the present invention. The system includes: The first processing module is used to obtain the natural language optimization target and the performance index data of multiple network nodes, including the target node to be optimized. The second processing module is used to input the natural language optimization target into the large language model and obtain the target semantic embedding vector and performance index weight vector output by the large language model. The third processing module is used to obtain the local reward of the target node based on the performance index weight vector and the performance index data of the target node. The fourth processing module is used to input the network topology graph and the target semantic embedding vector into the graph neural network to obtain the region reward output by the graph neural network; the network topology graph is constructed based on multiple network nodes and the relationships between them. The fifth processing module is used to merge local rewards and regional rewards to generate a comprehensive reward signal.

[0143] It should be noted that the wireless network agent dynamic reward generation system provided by the present invention can execute the wireless network agent dynamic reward generation method of any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0144] Figure 6This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the wireless network intelligent agent dynamic reward generation method provided in the above embodiments.

[0145] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the wireless network intelligent agent dynamic reward generation method provided in the above embodiments.

[0147] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the wireless network agent dynamic reward generation method provided in the above embodiments.

[0148] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating dynamic rewards for intelligent agents in a wireless network, characterized in that, include: Obtain the natural language optimization target and the performance index data of multiple network nodes, including the target node to be optimized; The natural language optimization objective is input into a large language model to obtain the target semantic embedding vector and performance index weight vector output by the large language model. The local reward of the target node is obtained based on the performance index weight vector and the performance index data of the target node. The network topology graph and the target semantic embedding vector are input into the graph neural network to obtain the region reward output by the graph neural network. The network topology diagram is constructed based on the multiple network nodes and the relationships between them; The local reward and the regional reward are merged to generate a comprehensive reward signal.

2. The method for generating dynamic rewards for wireless network agents according to claim 1, characterized in that, The step of inputting the network topology graph and the target semantic embedding vector into the graph neural network to obtain the region reward output by the graph neural network includes: The target semantic embedding vector can be concatenated as a global feature to the node features of each network node in the network topology graph, or the target semantic embedding vector can be added as a global additional node to the network topology graph. The hidden representation of each network node in the network topology graph is updated by weighting and summarizing the node features of each network node and its neighboring nodes using the graph neural network based on edge weights. The regional reward is generated through aggregation operations based on the updated hidden representation of the target node and its neighboring nodes within a preset hop range.

3. The method for generating dynamic rewards for wireless network agents according to claim 2, characterized in that, The edges in the network topology graph include a first type of edge representing geographical adjacency and a second type of edge representing frequency interference. The step of updating the hidden representation of each network node in the network topology graph by weighting and summarizing the node features of each network node and its neighboring nodes using the graph neural network based on edge weights includes: During message transmission, feature information of the network node's neighboring nodes is extracted; Based on the edge weight coefficients of the corresponding edges in the network topology graph, the feature information of the neighboring nodes is weighted and aggregated to obtain aggregated feature information; The aggregated feature information is fused with the current hidden representation of the network node, and a nonlinear transformation is performed on the fusion result to output the updated hidden representation.

4. The method for generating dynamic rewards for wireless network agents according to claim 1, characterized in that, The step of obtaining the local reward of the target node based on the performance index weight vector and the performance index data of the target node includes: After performing parameter adjustment actions on the target node, the indicator reward score is obtained based on the performance indicator weight vector and the change in the performance indicator data of the target node before and after the parameter adjustment actions are performed. Based on the performance index data of the target node at the first time point before the parameter adjustment action is executed, the first time point after the execution, and the second time point after the execution, calculate the trend reward score. The local reward is generated by weighting and combining the indicator reward score with the trend reward score.

5. The method for generating dynamic rewards for wireless network agents according to claim 4, characterized in that, The step of calculating the trend reward score based on the performance index data of the target node at the time points before, after, and after the parameter adjustment action, respectively, includes: Calculate the difference between the performance index data at the first time point after execution and the performance index data before execution to obtain the instantaneous performance change. The difference between the performance index data at the second time point after execution and the performance index data at the first time point after execution is calculated to obtain the continuous trend change amount; The trend reward score is obtained by weighted summing of the instantaneous performance change and the continuous trend change.

6. The method for generating dynamic rewards for wireless network agents according to claim 1, characterized in that, Before obtaining the local reward of the target node based on the performance metric weight vector and the performance metric data of the target node, the process further includes: Obtain historical network performance metrics data within a preset time window; The distribution characteristics of each of the historical network performance index data are analyzed using unsupervised clustering algorithms or probability distribution fitting algorithms. Extract the boundary points between cluster center values ​​or the inflection points of the mixed probability distribution, and determine the values ​​corresponding to the boundary points or inflection points as adaptive thresholds; the adaptive thresholds are used to measure the improvement level of the performance index data when calculating the local reward.

7. The method for generating dynamic rewards for wireless network agents according to claim 1, characterized in that, The step of fusing the local reward with the regional reward to generate a comprehensive reward signal includes: The local reward is normalized and mapped to convert it into a preset numerical range. The regional reward is normalized and mapped to convert it into a preset numerical range. The normalized local reward and the normalized regional reward are linearly weighted according to the preset fusion weight coefficient to generate the comprehensive reward signal.

8. The method for generating dynamic rewards for wireless network agents according to claim 7, characterized in that, After fusing the local reward and the regional reward to generate a comprehensive reward signal, the method further includes: The comprehensive reward signal is input into the reinforcement learning model to update the action policy of the reinforcement learning agent; The comprehensive reward signal is compared with a preset decision threshold to generate a status signal; wherein, when the comprehensive reward signal is greater than the decision threshold, a pass status signal is generated; when the comprehensive reward signal is not greater than the decision threshold, a fail status signal is generated. The state signal is output as a discrete evaluation signal independent of the comprehensive reward signal.

9. A dynamic reward generation system for wireless network intelligent agents, characterized in that, include: The first processing module is used to obtain the natural language optimization target and the performance index data of multiple network nodes, wherein the multiple network nodes include the target node to be optimized. The second processing module is used to input the natural language optimization target into the large language model and obtain the target semantic embedding vector and performance index weight vector output by the large language model. The third processing module is used to obtain the local reward of the target node based on the performance index weight vector and the performance index data of the target node. The fourth processing module is used to input the network topology graph and the target semantic embedding vector into the graph neural network to obtain the region reward output by the graph neural network; the network topology graph is constructed based on the multiple network nodes and the relationships between the network nodes; The fifth processing module is used to merge the local reward and the regional reward to generate a comprehensive reward signal.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating dynamic rewards for wireless network agents as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for generating dynamic rewards for wireless network agents as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for generating dynamic rewards for wireless network agents as described in any one of claims 1 to 8.

Images