Communication information determination method, apparatus, device, and readable storage medium

Abstract

The application discloses a communication information determination method and device, equipment and a readable storage medium. The method comprises the following steps: constructing a first intelligent agent corresponding to a first terminal-to-terminal W2W link in a first network environment; obtaining first real-time communication data, first historical communication data and a first action trajectory sequence of the first intelligent agent; inputting the first historical communication data into a target model to perform interference power value prediction, and obtaining a first predicted interference power value of the first intelligent agent; inputting the first real-time observation data, the first predicted interference power value and the first action trajectory sequence into the target model to obtain a plurality of first Q values corresponding to a plurality of first execution actions of the first intelligent agent; and determining a first target execution action based on the plurality of first Q values. The method improves the wireless resource allocation efficiency.

Description

Technical Field

[0001] This application belongs to the field of communication technology, specifically relating to a method, apparatus, device, and readable storage medium for determining communication information. Background Technology

[0002] With the increasing prevalence of smart home IoT and Wi-Fi technology, home networks typically have two types of communication links: one is the traditional terminal-to-router (Wi-Fi to Infrastructure, W2I) link where terminals connect to the wireless router, and the other is the terminal-to-Wi-Fi (Wi-Fi to Wi-Fi, W2W) link where devices communicate directly via Wi-Fi Direct. Because these two types of links share wireless spectrum resources, they are highly susceptible to co-channel and adjacent-channel interference, leading to reduced network throughput and significant transmission latency jitter.

[0003] To address the aforementioned interference issues, a unified scheduling framework of "master node selection—resource allocation—task scheduling" has been established. Specifically, a hierarchical weighted evaluation is first conducted based on indicators such as device hardware parameters, sensing communication capabilities, and device interoperability to select the master node. Then, the master node coordinates communication resources and service tasks of different protocols at the system level, and uses deep reinforcement learning algorithms to adaptively allocate time-frequency resources, thereby improving network service quality and overall resource utilization. However, this solution has significant drawbacks: in dynamically changing network scenarios such as fluctuations in neighboring Wi-Fi signal strength, sudden launches of high-latency sensitive services like high-definition video and online games, the scheduling strategy lags behind, easily causing network fluctuations and resulting in low efficiency in wireless resource allocation. Summary of the Invention

[0004] This application provides a method, apparatus, device, and readable storage medium for determining communication information, which solves the problem of low efficiency in wireless resource allocation in related technologies.

[0005] Firstly, a method for determining communication information is provided, the method comprising: Construct a first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first real-time communication data, the first historical communication data, and the first action trajectory sequence of the first intelligent agent are obtained. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent's selection of the first communication sub-band and the first transmission power. The first historical communication data is input into the target model to predict the interference power value, and the first predicted interference power value of the first intelligent agent is obtained. The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. A first target execution action is determined based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

[0006] Secondly, embodiments of this application provide a communication information determining device, the device comprising: The first construction module is used to construct the first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first acquisition module is used to acquire the first real-time communication data, the first historical communication data and the first action trajectory sequence of the first intelligent agent. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent to select the first communication sub-band and the first transmission power. The first prediction module is used to input the first historical communication data into the target model to predict the interference power value, and obtain the first predicted interference power value of the first intelligent agent. The first processing module is used to input the first real-time observation data, the first predicted interference power value and the first action trajectory sequence into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. The first determining module is configured to determine a first target execution action based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

[0007] Thirdly, embodiments of this application provide an electronic device, including a processor, the processor being used for: Construct a first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first real-time communication data, the first historical communication data, and the first action trajectory sequence of the first intelligent agent are obtained. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent's selection of the first communication sub-band and the first transmission power. The first historical communication data is input into the target model to predict the interference power value, and the first predicted interference power value of the first intelligent agent is obtained. The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. A first target execution action is determined based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

[0008] Fourthly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the communication information determination method as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the communication information determination method as described in the first aspect.

[0010] Sixthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the communication information determination method described above.

[0011] The communication information determination method of this application includes constructing a first agent corresponding to a first terminal-to-terminal W2W link in a first network environment; acquiring first real-time communication data, first historical communication data, and a first action trajectory sequence of the first agent, wherein the first action trajectory sequence consists of first communication data and a first execution action corresponding to each historical time step of the first agent in multiple historical time steps, and the first execution action is used to characterize the first agent's selection of a first communication sub-band and a first transmit power; inputting the first historical communication data into a target model to predict interference power values, thereby obtaining a first predicted interference power value of the first agent; and determining a first target execution action based on the multiple first Q values, wherein the first target execution action is the first execution action corresponding to the largest first Q value among the multiple first Q values, and the first target execution action is used to instruct the first W2W link to select the first communication sub-band and the first transmit power corresponding to the first target execution action. This method predicts future interference power values ​​based on the temporal characteristics of historical communication data using a target model. Then, it instructs the first W2W link to select the first communication sub-band and the first transmit power by determining the first target action based on the predicted interference power value. This allows for early detection of channel dynamic fluctuations, enabling the W2W link to adjust the channel and transmit power in advance, avoiding delays and network oscillations caused by centralized scheduling, dynamically adapting to Wi-Fi interference fluctuation scenarios, and thus improving the efficiency of wireless resource allocation. Attached Figure Description

[0012] Figure 1 This is one of the flowcharts of the communication information determination method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the target system provided in the real-time example of this application; Figure 3 This is a structural diagram of the initial model provided in the embodiments of this application; Figure 4 This is a structural diagram of a communication information determining device provided in an embodiment of this application; Figure 5 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] It is worth noting that the technologies described in this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in this application are often used interchangeably, and the described technologies can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies. However, the following description describes New Radio (NR) systems for illustrative purposes, and NR terminology is used in most of the following description. These technologies can also be applied to applications beyond NR systems, such as 6th Generation (6G) communication systems.

[0016] The following description, in conjunction with the accompanying drawings, further illustrates the communication information determination method, apparatus, device, storage medium, and program product proposed in the embodiments of the application.

[0017] Please see Figure 1 , Figure 1 A flowchart illustrating a communication information determination method provided in this application embodiment is shown in the figure. The method includes: Step 101: Construct the first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; In this step, the first network environment can be understood as a home Wi-Fi network, which includes a wireless router, multiple Wi-Fi terminals, and coexists terminal-to-router (Wi-Fi to Infrastructure, W2I) links and terminal-to-Wi-Fi (Wi-Fi to Wi-Fi, W2W) links for direct communication via Wi-FiDirect. Because the spectrum allocation and power control of home Wi-Fi are highly dynamic and high-dimensional, this application uses a partially observable Markov decision process model to map each first W2W link to a logical agent within a multi-agent reinforcement learning framework. Subband selection and transmit power control for subsequent links are handled by the first agent corresponding to the first W2W link. Wherein: The state space S of the first agent is used to characterize all environmental states such as channel gain, terminal location, and task information of the W2W link. The action space A of the first agent is used to characterize the selectable actions of each first agent, including subband selection and power control. It includes two parts: spectrum subband selection and power control. Therefore, the dimension of the action space is: (1); The observation space O of the first agent is used to characterize the local information directly seen by the first agent, including local measurement information and predicted interference information, such as information that each device can perceive itself, such as its own signal quality, received interference, plus predicted future interference. At each time step t, the W2W agent generates local observations through the observation function: (2); The aforementioned observation vectors integrate local measurement and prediction information, specifically including the W2W link channel gain. Wi-Fi to terminal interference gain Interference gain between Vehicle User Equipment (VUE) and Wi-Fi Interference gain between terminals Remaining data packet size Delay budget and the future Interference prediction of steps .

[0018] The transition probability P of the first agent is used to characterize the state changes of the network environment, and is jointly affected by channel dynamics and the actions of the first agent. The reward function R of the first agent is used to reflect the system optimization goal and incentivize the first agent to cooperate. The specific reward function can be found in the following formula: (3); in, , This is used as a weight to balance W2I throughput and W2W delivery rate. Discount factor It is used to balance short-term gains and long-term planning. The W2W delivery rate is estimated and defined as: (4).

[0019] Step 102: Obtain the first real-time communication data, the first historical communication data, and the first action trajectory sequence of the first intelligent agent. The first action trajectory sequence consists of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent's selection of the first communication sub-band and the first transmission power. In this step, the first real-time communication data of the first agent, i.e., the local observation data of the current time step, includes the channel gain of the first W2W link, the gain of various types of interference, the size of the remaining data packets of the terminal, the service delay budget, etc., which is the network state perceived by the first agent at the moment; the first historical communication data includes the observation data of the first W2W link in the most recent Th time steps, including channel gain, interference power and task information; the first action trajectory sequence is a chronological sequence of alternating "historical communication data + historical execution actions", which is the environmental observation data and the decisions made by the first agent in each historical time step since its operation; the first execution action is the first communication sub-band and the first transmission power selected by the first agent.

[0020] Step 103: Input the first historical communication data into the target model to predict the interference power value, and obtain the first predicted interference power value of the first intelligent agent; In this step, the first historical communication data can be input into the Long Short-Term Memory (LSTM) module of the target model. The LSTM module learns the channel timing pattern in the first historical communication data, infers future network changes based on the historical state, and outputs the first predicted interference power value. The first predicted interference power value can be understood as the interference power prediction result for future time steps.

[0021] Since this application focuses on efficient access and resource allocation for multiple terminals in home Wi-Fi, its time-domain correlation is modeled using the autocorrelation coefficient, defined as follows: (5); in, It is a zero-order Bessel function. For Doppler frequency, The time interval is specified. In the context of this application, the autocorrelation coefficient... When the value is close to 1, the channel changes gradually during the coherence time, thus exhibiting time-predictability.

[0022] Step 104: Input the first real-time observation data, the first predicted interference power value and the first action trajectory sequence into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. In this step, the target model also includes a Gated Recurrent Unit (GRU), which can input the first real-time observation data, the first predicted interference power value, and the first action trajectory sequence into the GRU module of the target model. The GRU handles long-term dependencies through forget gates and update gates, effectively capturing the channel timing characteristics in the data, making it more suitable for the dynamic environment of V2X compared to simple fully connected networks. The computational complexity of the GRU module is moderate; the computational cost of a single forward propagation is linearly related to the number of hidden units and the input dimension, suitable for the real-time processing requirements of W2W devices. The output of the GRU module is multiple first Q-values ​​corresponding to multiple first actions selectable by the first agent. The first Q-value represents the long-term comprehensive expected return that the first agent can obtain after performing a certain action. The expected return in this application is the revenue corresponding to the reward function of the first agent, namely the total throughput of the W2I link and the data packet delivery rate of the W2W link.

[0023] Step 105: Determine a first target execution action based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

[0024] In this step, among all the Q values ​​corresponding to the actions, the one with the largest value is selected. The action associated with this Q value is the first target action to be executed. The larger the Q value, the higher the long-term network-wide benefit and the better the network performance brought by this action.

[0025] In one implementation, a first agent corresponding to a first terminal-to-terminal W2W link in a first network environment is constructed; first real-time communication data, first historical communication data, and a first action trajectory sequence of the first agent are acquired, wherein the first action trajectory sequence consists of first communication data and a first execution action corresponding to each historical time step of the first agent in multiple historical time steps, and the first execution action is used to characterize the first agent's selection of a first communication sub-band and a first transmit power; the first historical communication data is input into a target model to predict interference power values, thereby obtaining a first predicted interference power value of the first agent; a first target execution action is determined based on the multiple first Q values, wherein the first target execution action is the first execution action corresponding to the largest first Q value among the multiple first Q values, and the first target execution action is used to instruct the first W2W link to select the first communication sub-band and the first transmit power corresponding to the first target execution action.

[0026] In this implementation, the target model predicts the future interference power value based on the time-series characteristics of historical communication data. Then, the first target action determined by the predicted interference power value instructs the first W2W link to select the first communication sub-band and the first transmit power. This allows for early detection of channel dynamic fluctuations, enabling the W2W link to adjust the channel and transmit power in advance, avoiding delays and network oscillations caused by centralized scheduling, dynamically adapting to Wi-Fi interference fluctuation scenarios, and thereby improving the efficiency of wireless resource allocation.

[0027] Optionally, the target model includes a gated recurrent unit (GRU) module and a long short-term memory (LSTM) network module; The step of inputting the first historical communication data of the first intelligent agent into the target model to predict the interference power value and obtain the first predicted interference power value of the first intelligent agent includes: The first historical communication data of the first intelligent agent is input into the LSTM module of the target model, and the temporal correlation of the first historical communication data is analyzed to obtain the first predicted interference power value of the first intelligent agent. The step of inputting the first real-time observation data, the first predicted interference power value, and the first action trajectory sequence into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent includes: The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the GRU module of the target model, and the forget gate and update gate mechanisms are used to calculate the multiple first Q values ​​corresponding to the multiple first execution actions of the first agent.

[0028] In one implementation, the target model includes a GRU module and an LSTM module.

[0029] As mentioned earlier, home Wi-Fi channels exhibit strong temporal correlation and gradual state changes, providing a theoretical basis for predicting future interference. The LSTM module of the target model utilizes this characteristic to predict the interference power in the next Tf steps based on the first agent's historical communication data, adding a look-ahead capability to the first agent's decision-making and solving the problems of "reaction lag and scheduling oscillations" in traditional solutions. The prediction process of the LSTM module is detailed in the following formula: (6); The input to this formula is the historical channel state sequence (i.e., the first historical communication data). In other words, it represents the channel gain, interference power, and mission information observed by device k over the past Th time steps; the output of this formula is the predicted interference power for the next Tf steps. , indicating the future Interference power estimation of step, where In hidden state, and These are the parameters for the fully connected layer.

[0030] Unlike traditional MARL methods that rely on "implicit history coding," the LSTM module of the target model explicitly outputs interference estimates, significantly enhancing the agent's ability to optimize interference constraints. When high interference is predicted for a subband in the future, the agent can select alternative channels in advance or reduce transmit power to avoid being affected by interference. This look-ahead capability is particularly important for optimizing the delivery rate of W2W direct connections, because the delivery rate depends on capacity stability across time steps; avoiding interference in advance is essential to ensure that data packets are transmitted within the delay constraints.

[0031] Because the channel state of home Wi-Fi does not change randomly but has strong temporal correlation (e.g., the autocorrelation coefficient is close to 1 as mentioned earlier), past states can influence future states. Traditional fully connected networks can only process current observations and cannot capture this historical dependency. The GRU module, however, is a recurrent neural network specifically designed for processing time-series data and excels at capturing long-term dependencies. The specific processing procedure of the GRU module can be seen in the following formula: (7); The input to the GRU module is... The observations of agent k from time step 1 to the current time step t and actions An alternating sequence is a complete record of all the "environmental states seen" and "decisions made" by the agent in the past, equivalent to the agent's historical decision log. These are the parameters of the GRU network. The GRU module uses forget and update gate mechanisms to remember historical information useful for future decisions and forget useless noise, effectively capturing the temporal characteristics of the channel. The output of the GRU module... Indicates that agent k is on the trajectory Select action The expected return is used for action selection and subsequent value decomposition.

[0032] In this implementation, the target model leverages the strong adaptability of the LSTM module to temporal features to uncover patterns in historical communication data, accurately predicting future interference and enabling forward-looking decision-making. The GRU module's gating mechanism efficiently processes action-observation trajectories, capturing temporal dependencies and rapidly calculating Q-values. These two modules work collaboratively, respectively performing interference prediction and action benefit assessment. This mitigates the impact of dynamic interference in the wireless environment, improves decision-making accuracy and computational efficiency, and ensures reasonable resource allocation and stable operation of the W2W link.

[0033] Optionally, the target model further includes a residual Q-network (RQN) module; The GRU module, which inputs the first real-time observation data, the first predicted interference power value, and the first action trajectory sequence into the target model, calculates multiple first Q values ​​corresponding to multiple first execution actions of the first agent using forget gate and update gate mechanisms, including: The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the GRU module of the target model, and the forget gate and update gate mechanisms are used to calculate the multiple initial first Q values ​​corresponding to the multiple first execution actions of the first agent. Determine the mean of the plurality of initial first Q values ​​and the maximum value among the plurality of initial first Q values; The mean and the maximum value, along with the first predicted interference power value, are input into the RQN module of the target model for calculation to obtain the first residual correction factor. The plurality of initial first Q values ​​are corrected using the first residual correction factor to obtain a plurality of first Q values.

[0034] In one implementation, traditional Q-learning algorithms are prone to estimation bias, leading to training oscillations and inaccurate decision-making. Therefore, this application introduces a Residual Q-Network (RQN) module to generate a correction factor to adjust the individual Q-value of each agent, thereby addressing the problems of cooperation bias and estimation error. The correction factor is calculated using the following formula: (8) The input features include the mean Q-value. and maximum value Capture the statistical characteristics of the action value distribution: (9) (10) Among them, the function For a fully connected neural network, the output scalar The Q-value is normalized using the ReLU activation function. A correction factor is used to adjust the individual Q-value; the adjustment process can be seen in the following formula: (11) The correction factor of the original RQN module depends only on , Ignoring future interference information may limit collaborative efficiency in high-interference W2W scenarios. In this implementation, the target model of this application enables the agent to dynamically adjust action priorities by fusing predicted states, prioritizing low-interference subbands and appropriate power.

[0035] The target model used in the above process is the initial model obtained after adjusting the network parameters of the GRU, LSTM, and RQN modules during the training phase. During execution, only forward inference is performed; parameters are not updated backward. Each W2W link corresponds to an independent agent, operating independently in a distributed manner, relying solely on local data for decision-making, without the need for centralized computation via routers. The reason why each agent can make near-globally optimal decisions based on its own local observations during the execution phase is that, during the training phase, all agents corresponding to W2W links are trained uniformly, allowing them to learn a cooperative, globally optimal strategy.

[0036] In traditional solutions, only the router knows global information, and the terminal can only passively follow instructions. Therefore, in order for the terminal to also be able to perceive its environment and cooperate with neighboring devices, see [reference needed]. Figure 2 This application first constructs a target system, setting up a "distributed collaborative sensing network" for home Wi-Fi devices. In this network, terminals share environmental awareness and their respective location information through W2W links. At the same time, one terminal within the cluster is selected to report the shared information to the Wi-Fi network through a W2I link, and the Wi-Fi network then broadcasts the cluster terminal information to terminals in a wider range.

[0037] In home Wi-Fi network systems, the performance of W2I and W2W links is directly limited by the signal-to-interference-plus-noise ratio (SINR), which is significantly affected by complex interference caused by shared spectrum. The SINR of the m-th W2I link is defined as: (12) in, CUE transmit power; The W2I channel gain includes path loss, shadowing fading, and small-scale fading. This indicates whether the k-th VUE occupies the subband of the m-th CUE. For VUE transmit power, For interference gain from the terminal to the Wi-Fi network, This represents the power of Gaussian white noise. The cumulative interference from W2W links to W2I links represents a core challenge in resource allocation optimization. The presence of interference degrades the signal quality of W2I links, limiting throughput improvement; therefore, it is necessary to mitigate its impact through reasonable subband allocation and power control.

[0038] Similarly, the SINR of the k-th W2W link is defined as: (13) in, For W2W channel gain, The interference gain from CUE to VUE. This represents the interference gain between VUEs.

[0039] Based on SINR, the channel capacity of W2I and W2W links is calculated using Shannon's formula and is defined as follows: (14) (15) Where B is the subband bandwidth.

[0040] The optimization goal of W2I links is usually to maximize total throughput. This is to support data transmission from the Wi-Fi network to the terminal. The logarithmic form of formula (14) shows that the improvement of SINR makes a significant contribution to capacity, but due to the interference term in formula (12), the actual throughput needs to be optimized by allocating resources to approach the theoretical upper limit. The throughput of the W2I link directly affects the service quality of the base station, and therefore has a high priority in home Wi-Fi network systems.

[0041] The performance of a W2W link revolves around reliable transmission, with the key metric being delivery rate, defined as the probability of successfully transmitting data packets under maximum latency constraints. (16) in, The capacity of W2W at time step t. For coherent time, For maximum delay, For data packet size, Indicates the number of available time steps.

[0042] To accurately describe the impact of interference on W2I and W2W performance, the single-link interference power of W2W on W2I is defined as follows: (17) In summary, the objective of this application is to coordinate the performance of W2I and W2W links, achieving high throughput and high reliability with limited spectrum resources, while effectively controlling interference to ensure communication quality. Therefore, the optimization objective can be defined as: (18) in, and Let P be the weight, and P be the set of discrete powers.

[0043] Based on the above optimization objectives, the training process of the target model will be described in detail below. It should be noted that the terms "first" and "second" used in this application are only for distinguishing between the actual execution process and the training process, and have no other meaning.

[0044] Optionally, before inputting the first historical communication data into the target model to predict the interference power value and obtain the first predicted interference power value of the first agent, the method further includes: Construct multiple second agents corresponding to multiple second W2W links in the second network environment; Within each time step, the second real-time communication data, the second historical communication data, and the second action trajectory sequence of each of the plurality of second agents are acquired. The second action trajectory sequence consists of the second communication data and the second execution action corresponding to each of the plurality of historical time steps. The second execution action is used to characterize the second agent's selection of the second communication sub-band and the second transmission power. The second historical communication data of each second agent is input into the initial model to predict the interference power value, thereby obtaining the second predicted interference power value of each second agent. The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial model for calculation to obtain the second Q value corresponding to each of the multiple second execution actions of each second agent. The second Q value is used to characterize the expected reward of the second agent performing the corresponding second execution action. A second target execution action is determined based on the plurality of second Q values. The second target execution action is the second execution action corresponding to the largest second Q value among the plurality of second Q values. The second target execution action is used to instruct the second W2W link to select the second communication subband and the second transmit power corresponding to the second target execution action. After the multiple second intelligent agents execute their respective second target execution actions, the total throughput of multiple terminal-to-router W2I links and the total delivery rate of the multiple second W2W links in the second network environment are calculated. Calculate the global reward value based on the total throughput and the total delivery rate; Using the training sample set corresponding to each time step, calculate the global joint Q value and the target Q value corresponding to each time step. The training sample set includes the second real-time communication data, the second target execution action and the global reward value of each second agent in the current time step, as well as the second real-time communication data and the second target execution action of each second agent in the next time step. Based on the global joint Q-value and the target Q-value, a loss function is constructed. With the goal of minimizing the loss function, the parameters of each module in the initial model are updated to obtain the target model.

[0045] In one implementation, multiple second agents are constructed, each corresponding to a second W2W link in a second network environment. At each time step, the second real-time communication data, second historical communication data, and second action trajectory sequence of each of the multiple second agents are acquired. This data is then input into an initial model to obtain the second Q-value corresponding to each of the multiple second execution actions of each second agent. The process of processing this data in the initial model can refer to the execution process of the aforementioned target model, and will not be elaborated further here.

[0046] The entire training process iterates in rounds and time steps. Each round represents a complete network interaction, and each time step represents a decision. Within each time step, the actions performed by all second agents on the selected second objective constitute a joint action. After all the second agents have performed these actions, the global reward value can be calculated based on the aforementioned formula (3).

[0047] Then, the second real-time communication data, second objective execution action, and global reward value of each second agent in the current time step, along with the second real-time communication data and second objective execution action of each second agent in the next time step, are combined into a training sample group and placed in the experience buffer. Next, random samples are taken from the experience buffer, and the global joint Q-value and target Q-value of the corresponding sample group are calculated. The loss function is then used to measure the difference between the global joint Q-value and the target Q-value. The error is backpropagated using the gradient descent algorithm to update the parameters of each module in the initial model, including LSTM, GRU, and RQN. After multiple iterations, the loss function continuously decreases, the model gradually learns the optimal scheduling strategy, and finally converges to the trained target model. The specific loss function is shown in the following formula: (19) in, These are the parameters for the GRU and RQN modules in the initial model; These are the parameters of the first model; For the global joint Q value, The target Q value.

[0048] The target Q-value is calculated from the parameters of the first model with fixed parameters. The parameters of the first model are obtained by periodically and synchronously replicating the initial model parameters and remain fixed during normal operation. They are used to calculate the target Q-value and serve to stabilize the training process and avoid oscillations.

[0049] In this implementation, centralized training is used to constrain the model until convergence using global reward and loss functions. During training, the model learns the globally optimal policy in a centralized manner. In actual operation, each device only needs to make decisions based on its own local observations, without frequent communication with the router, which reduces overhead and facilitates the expansion to more devices.

[0050] Optionally, the step of calculating the global joint Q-value and the target Q-value for each time step using the training sample group corresponding to each time step includes: Calculate the global joint Q value based on the second real-time communication data and the second target execution action corresponding to each second agent in the current time step; The target Q value is calculated based on the global reward value, the second real-time communication data of each second agent in the next time step, and the second target execution action.

[0051] In one implementation, based on the second real-time communication data of each second agent, the second target execution action corresponding to each second agent is obtained using the initial model. The global joint Q-value is obtained by summing the Q-values ​​corresponding to the second target execution actions of each second agent. The specific calculation process can be found in the following formula: (20) in, This indicates the second real-time communication data. Indicates the action to be performed for the second objective. This represents the Q value corresponding to the action performed for the second objective.

[0052] The objective Q-value is calculated from the global reward value, the second real-time communication data of each second agent in the next time step, and the second objective execution action. It represents the ideal expected total reward of the algorithm fit and is the standard answer that the initial model needs to fit during training. The specific calculation process is shown in the following formula: (twenty one) Where r is the global reward value, which is the real-time reward obtained after the initial model executes the joint action at the current time step; it is calculated by weighting the total W2I throughput and W2W delivery rate. This is the optimal global joint Q-value for the next time step; For real-time communication data at the next time step; Execute actions for the second target corresponding to each second agent in the next time step.

[0053] In this implementation, calculating the global joint Q-value and the target Q-value helps to accurately quantify the deviation between the model prediction and the standard value, ensuring the reliability of subsequent loss function calculations and stably guiding the iterative optimization of model parameters.

[0054] Optionally, the initial model includes an initial GRU module and an initial LSTM module; The step of inputting the second historical communication data of each second agent into the initial model to predict the interference power value, and obtaining the second predicted interference power value of each second agent, includes: The second historical communication data of each second agent is input into the initial LSTM module of the initial model, and the temporal correlation of the second historical communication data is analyzed to obtain the second predicted interference power value of each second agent. The step involves inputting the second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent into the initial model for calculation, to obtain the second Q value corresponding to each of the multiple second execution actions of each second agent, including: The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial GRU module of the initial model, and the second Q value corresponding to the multiple second execution actions of each second agent is calculated using the forget gate and update gate mechanism.

[0055] In one implementation, the initial model includes an initial GRU module and an initial LSTM module. The second historical communication data of each second agent can be input into the initial LSTM module to obtain the second predicted interference power value for each second agent. The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent can be input into the initial GRU module to obtain the second Q value corresponding to each of the multiple selectable second execution actions for each second agent. The specific process of processing this data can refer to the execution process of the aforementioned trained GRU module and LSTM module; the beneficial effects are similar, and will not be elaborated further here.

[0056] To ensure prediction accuracy, the initial LSTM module can be trained beforehand using mean squared error (MSE) loss to minimize the difference between the predicted and actual interference values, thus guaranteeing prediction accuracy. The specific training process can be found in the following formula: (7); in, The actual interference power is provided by the environment. The expected prediction error is defined as: (8); The implementation of the LSTM module can be divided into two stages: Pre-training phase: Train the LSTM model with historical data to teach it to predict future interference from the channel state sequence until the prediction error is small enough.

[0057] MARL Integration Phase: The trained LSTM model is put into the online process of multi-agent reinforcement learning. At each time step, the device generates interference predictions using the latest historical data and adds the prediction results to its own observation vector to guide subsequent channel and power decisions.

[0058] For a detailed implementation process, please refer to Algorithm 1 below: Input: Number of W2W agents K, history length T h Prediction steps T f Channel state dimension input dim ; Output: Predicted state, used to enhance observations; Initial stage: 1: Initialize the LSTM predictor for K agents. (parameter (Hidden unit number adaptation for real-time performance) 2: Initialize the history buffer H to store the channel state sequence of each agent; Pre-training phase: 1: for each round do; 2: Sampling environment data: Historical channel state data Actual interference data ; 3: for each W2W agent k, do; 4: Calculate the predicted state ; 5: Calculate the mean square error loss ; 6: Update via gradient descent , minimize ; 7: end for; 8: end for; MARL Integration Phase: 1: for each time step t, do; 2: for each W2W agent do; 3: Obtain historical channel status from buffer H ; 4: Calculate the predicted state ; 5: Update H, store the current state ; 6: end for; 7: Return the set of predicted states ; 8: end for.

[0059] Optionally, the initial model further includes an initial RQN module; The step of inputting the second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent into the initial GRU module of the initial model, and calculating the second Q value corresponding to each of the multiple selectable second execution actions of each second agent using forget gate and update gate mechanisms, includes: The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial GRU module, and multiple initial second Q values ​​corresponding to multiple second execution actions of each second agent are calculated using the forget gate and update gate mechanism. Determine the mean of the plurality of initial second Q values ​​and the maximum value among the plurality of initial second Q values; The mean and the maximum value, as well as the second predicted interference power value, are input into the initial RQN module for calculation to obtain the second residual correction factor; The plurality of initial second Q values ​​are corrected using the second residual correction factor to obtain a plurality of second Q values.

[0060] In one implementation, the Q value output by the initial GRU module can be corrected using the initial RQN module. The specific correction process is similar to the previously described process of correcting the Q value output by the GRU module using the RQN module, and the beneficial effects are similar. It will not be repeated here.

[0061] Next, see Figure 3 The structure diagram of the initial model, and the data processing process of the initial model for the second agent are roughly as follows: The second historical communication data of each second agent is input into the initial LSTM module of the initial model to analyze the temporal correlation of the second historical communication data and obtain the second predicted interference power value of each second agent. The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial GRU module. The forget gate and update gate mechanisms are used to calculate multiple initial second Q values ​​corresponding to multiple second execution actions of each second agent. The mean and the maximum value of the multiple initial second Q values ​​are determined. The mean, the maximum value, and the second predicted interference power value are input into the initial RQN module for calculation to obtain the second residual correction factor. The multiple initial second Q values ​​are corrected using the second residual correction factor to obtain multiple second Q values. The multiple second Q values ​​are fed into the aggregation network to generate a global joint action value function to guide the agent's action selection.

[0062] The training process of the initial model can also be represented by the following Algorithm 2: Inputs: Number of W2W agents K, number of sub-bands M, power set P, prediction state ; Output: Optimization strategy ; 1: Initialize the environment, GRU network, RQN network, and experience buffer D; 2: for each round e do; 3: for each time step t, do; 4: for each W2W agent do; 5: Obtain the predicted state , construct observation ; 6: Calculation ,according to , Calculate the correction factor ; 7: Calculate the corrected action. ; 8: end for; 9: Execute joint actions; 10: Calculate Rewards ; 11: Store and convert to the conversion area; 12: Sample small batches of data from it; 13: Calculate the joint Q value ; 14: Calculate the target Q value ; 15: Calculate the loss 16: Optimization Update parameters ; 17: end for; 18: Update the target network parameters; 19: end for.

[0063] Please see Figure 4 , Figure 4 This is a schematic diagram of a communication information determining device provided in an embodiment of this application. As shown in the figure, the device 400 includes: The first construction module 401 is used to construct a first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first acquisition module 402 is used to acquire the first real-time communication data, the first historical communication data and the first action trajectory sequence of the first intelligent agent. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent to select the first communication sub-band and the first transmission power. The first prediction module 403 is used to input the first historical communication data into the target model to predict the interference power value and obtain the first predicted interference power value of the first intelligent agent. The first processing module 404 is used to input the first real-time observation data, the first predicted interference power value and the first action trajectory sequence into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. The first determining module 405 is used to determine a first target execution action based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

[0064] Optionally, the target model includes a gated recurrent unit (GRU) module and a long short-term memory (LSTM) network module; The first prediction module 403 includes: The first analysis unit is used to input the first historical communication data of the first agent into the LSTM module of the target model, analyze the temporal correlation of the first historical communication data, and obtain the first predicted interference power value of the first agent. The first processing module 404 includes: The first computing unit is used to input the first real-time observation data, the first predicted interference power value and the first action trajectory sequence into the GRU module of the target model, and use the forget gate and update gate mechanism to calculate multiple first Q values ​​corresponding to multiple first execution actions of the first agent.

[0065] Optionally, the target model further includes a residual Q-network (RQN) module; The first computing unit includes: The first computing subunit is used to input the first real-time observation data, the first predicted interference power value and the first action trajectory sequence into the GRU module of the target model, and use the forget gate and update gate mechanism to calculate multiple initial first Q values ​​corresponding to multiple first execution actions of the first agent. The first determining subunit is used to determine the mean of the plurality of initial first Q values ​​and the maximum value among the plurality of initial first Q values; The second calculation subunit is used to input the mean, the maximum value, and the first predicted interference power value into the RQN module of the target model for calculation to obtain the first residual correction factor. The first correction unit is used to correct the plurality of initial first Q values ​​using the first residual correction factor to obtain a plurality of first Q values.

[0066] Optionally, the device further includes: The second building module is used to build multiple second intelligent agents corresponding to multiple second W2W links in the second network environment; The second acquisition module is used to acquire, in each time step, the second real-time communication data, the second historical communication data, and the second action trajectory sequence of each of the plurality of second agents. The second action trajectory sequence consists of the second communication data and the second execution action corresponding to each of the plurality of historical time steps. The second execution action is used to characterize the second agent's selection of the second communication sub-band and the second transmission power. The second prediction module is used to input the second historical communication data of each second agent into the initial model to predict the interference power value, and obtain the second predicted interference power value of each second agent. The second processing module is used to input the second real-time observation data, the second predicted interference power value and the second action trajectory sequence of each second agent into the initial model for calculation to obtain the second Q value corresponding to each of the multiple second execution actions of each second agent. The second Q value is used to characterize the expected reward of the second agent performing the corresponding second execution action. The second determining module is used to determine a second target execution action based on the plurality of second Q values. The second target execution action is the second execution action corresponding to the largest second Q value among the plurality of second Q values. The second target execution action is used to instruct the second W2W link to select the second communication subband and the second transmit power corresponding to the second target execution action. The first calculation module is used to calculate the total throughput of multiple terminal-to-router W2I links and the total delivery rate of multiple second W2W links in the second network environment after the multiple second agents execute their respective second target execution actions. The second calculation module is used to calculate the global reward value based on the total throughput and the total delivery rate; The third calculation module is used to calculate the global joint Q value and the target Q value corresponding to each time step using the training sample group corresponding to each time step. The training sample group includes the second real-time communication data, the second target execution action and the global reward value of each second agent in the current time step, and the second real-time communication data and the second target execution action of each second agent in the next time step. The third processing module is used to construct a loss function based on the global joint Q-value and the target Q-value, and update the parameters of each module in the initial model with the goal of minimizing the loss function, so as to obtain the target model.

[0067] Optionally, the third computing module includes: The third computing unit is used to calculate the global joint Q value based on the second real-time communication data and the second target execution action corresponding to each second agent in the current time step. The fourth calculation unit is used to calculate the target Q value based on the global reward value, the second real-time communication data of each second agent in the next time step, and the second target execution action.

[0068] Optionally, the initial model includes an initial GRU module and an initial LSTM module; The second prediction module includes: The second analysis unit is used to input the second historical communication data of each second agent into the initial LSTM module of the initial model, analyze the temporal correlation of the second historical communication data, and obtain the second predicted interference power value of each second agent. The second processing module includes: The fifth calculation unit is used to input the second real-time observation data, the second predicted interference power value and the second action trajectory sequence of each second agent into the initial GRU module of the initial model, and use the forget gate and update gate mechanism to calculate the second Q value corresponding to the multiple second execution actions of each second agent.

[0069] Optionally, the initial model further includes an initial RQN module; The fifth computing unit includes: The second calculation subunit is used to input the second real-time observation data, the second predicted interference power value and the second action trajectory sequence of each second agent into the initial GRU module, and use the forget gate and update gate mechanism to calculate multiple initial second Q values ​​corresponding to multiple second execution actions of each second agent; The second determining subunit is used to determine the mean of the plurality of initial second Q values ​​and the maximum value among the plurality of initial second Q values; The third calculation subunit is used to input the mean and the maximum value, as well as the second predicted interference power value, into the initial RQN module for calculation to obtain the second residual correction factor; The second correction subunit is used to correct the plurality of initial second Q values ​​using the second residual correction factor to obtain a plurality of second Q values.

[0070] The communication information determination device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments shown achieve the same technical effects, and will not be described again here to avoid repetition.

[0071] Specifically, see Figure 5 As shown in the illustration, this application also provides an electronic device, including a bus 501, an antenna 503, a bus interface 504, a processor 505, and a memory 506. The processor 505 is used for: Construct a first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first real-time communication data, the first historical communication data, and the first action trajectory sequence of the first intelligent agent are obtained. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent's selection of the first communication sub-band and the first transmission power. The first historical communication data is input into the target model to predict the interference power value, and the first predicted interference power value of the first intelligent agent is obtained. The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. A first target execution action is determined based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

[0072] exist Figure 5 In this design, a bus architecture (represented by bus 501) is used. Bus 501 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 505 and memory represented by memory 506. Bus 501 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Data processed by processor 505 is transmitted over the wireless medium via antenna 503. Furthermore, antenna 503 also receives data and transmits data to processor 505.

[0073] Processor 505 manages bus 501 and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 506 can be used to store data used by processor 505 during operation.

[0074] Optionally, the processor 505 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0075] Optionally, the target model includes a gated recurrent unit (GRU) module and a long short-term memory (LSTM) network module; The processor 505 is specifically used for: The first historical communication data of the first intelligent agent is input into the LSTM module of the target model, and the temporal correlation of the first historical communication data is analyzed to obtain the first predicted interference power value of the first intelligent agent. The step of inputting the first real-time observation data, the first predicted interference power value, and the first action trajectory sequence into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent includes: The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the GRU module of the target model, and the forget gate and update gate mechanisms are used to calculate the multiple first Q values ​​corresponding to the multiple first execution actions of the first agent.

[0076] Optionally, the target model further includes a residual Q-network (RQN) module; The processor 505 is specifically used for: The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the GRU module of the target model, and the forget gate and update gate mechanisms are used to calculate the multiple initial first Q values ​​corresponding to the multiple first execution actions of the first agent. Determine the mean of the plurality of initial first Q values ​​and the maximum value among the plurality of initial first Q values; The mean and the maximum value, along with the first predicted interference power value, are input into the RQN module of the target model for calculation to obtain the first residual correction factor. The plurality of initial first Q values ​​are corrected using the first residual correction factor to obtain a plurality of first Q values.

[0077] Optionally, the processor 505 is further configured to: Construct multiple second agents corresponding to multiple second W2W links in the second network environment; Within each time step, the second real-time communication data, the second historical communication data, and the second action trajectory sequence of each of the plurality of second agents are acquired. The second action trajectory sequence consists of the second communication data and the second execution action corresponding to each of the plurality of historical time steps. The second execution action is used to characterize the second agent's selection of the second communication sub-band and the second transmission power. The second historical communication data of each second agent is input into the initial model to predict the interference power value, thereby obtaining the second predicted interference power value of each second agent. The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial model for calculation to obtain the second Q value corresponding to each of the multiple second execution actions of each second agent. The second Q value is used to characterize the expected reward of the second agent performing the corresponding second execution action. A second target execution action is determined based on the plurality of second Q values. The second target execution action is the second execution action corresponding to the largest second Q value among the plurality of second Q values. The second target execution action is used to instruct the second W2W link to select the second communication subband and the second transmit power corresponding to the second target execution action. After the multiple second intelligent agents execute their respective second target execution actions, the total throughput of multiple terminal-to-router W2I links and the total delivery rate of the multiple second W2W links in the second network environment are calculated. Calculate the global reward value based on the total throughput and the total delivery rate; Using the training sample set corresponding to each time step, calculate the global joint Q value and the target Q value corresponding to each time step. The training sample set includes the second real-time communication data, the second target execution action and the global reward value of each second agent in the current time step, as well as the second real-time communication data and the second target execution action of each second agent in the next time step. Based on the global joint Q-value and the target Q-value, a loss function is constructed. With the goal of minimizing the loss function, the parameters of each module in the initial model are updated to obtain the target model.

[0078] Optionally, the processor 505 is specifically used for: Calculate the global joint Q value based on the second real-time communication data and the second target execution action corresponding to each second agent in the current time step; The target Q value is calculated based on the global reward value, the second real-time communication data of each second agent in the next time step, and the second target execution action.

[0079] Optionally, the initial model includes an initial GRU module and an initial LSTM module; The processor 505 is specifically used for: The second historical communication data of each second agent is input into the initial LSTM module of the initial model, and the temporal correlation of the second historical communication data is analyzed to obtain the second predicted interference power value of each second agent. The processor 505 is specifically used for: The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial GRU module of the initial model, and the second Q value corresponding to the multiple second execution actions of each second agent is calculated using the forget gate and update gate mechanism.

[0080] Optionally, the initial model further includes an initial RQN module; The processor 505 is specifically used for: The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial GRU module, and multiple initial second Q values ​​corresponding to multiple second execution actions of each second agent are calculated using the forget gate and update gate mechanism. Determine the mean of the plurality of initial second Q values ​​and the maximum value among the plurality of initial second Q values; The mean and the maximum value, as well as the second predicted interference power value, are input into the initial RQN module for calculation to obtain the second residual correction factor; The plurality of initial second Q values ​​are corrected using the second residual correction factor to obtain a plurality of second Q values.

[0081] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described communication information determination method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0082] This application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the communication information determination method embodiment described above and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0083] The processor mentioned above is the processor in the terminal described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0084] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0085] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for determining communication information, characterized in that, The method includes: Construct a first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first real-time communication data, the first historical communication data, and the first action trajectory sequence of the first intelligent agent are obtained. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent's selection of the first communication sub-band and the first transmission power. The first historical communication data is input into the target model to predict the interference power value, and the first predicted interference power value of the first intelligent agent is obtained. The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. A first target execution action is determined based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

2. The communication information determination method according to claim 1, characterized in that, The target model includes a gated recurrent unit (GRU) module and a long short-term memory (LSTM) network module. The step of inputting the first historical communication data of the first intelligent agent into the target model to predict the interference power value and obtain the first predicted interference power value of the first intelligent agent includes: The first historical communication data of the first intelligent agent is input into the LSTM module of the target model, and the temporal correlation of the first historical communication data is analyzed to obtain the first predicted interference power value of the first intelligent agent. The step of inputting the first real-time observation data, the first predicted interference power value, and the first action trajectory sequence into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent includes: The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the GRU module of the target model, and the forget gate and update gate mechanisms are used to calculate the multiple first Q values ​​corresponding to the multiple first execution actions of the first agent.

3. The communication information determination method according to claim 2, characterized in that, The target model also includes a residual Q-network (RQN) module; The GRU module, which inputs the first real-time observation data, the first predicted interference power value, and the first action trajectory sequence into the target model, calculates multiple first Q values ​​corresponding to multiple first execution actions of the first agent using forget gate and update gate mechanisms, including: The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the GRU module of the target model, and the forget gate and update gate mechanisms are used to calculate the multiple initial first Q values ​​corresponding to the multiple first execution actions of the first agent. Determine the mean of the plurality of initial first Q values ​​and the maximum value among the plurality of initial first Q values; The mean and the maximum value, along with the first predicted interference power value, are input into the RQN module of the target model for calculation to obtain the first residual correction factor. The plurality of initial first Q values ​​are corrected using the first residual correction factor to obtain a plurality of first Q values.

4. The communication information determination method according to claim 1, characterized in that, Before inputting the first historical communication data into the target model to predict the interference power value and obtain the first predicted interference power value of the first intelligent agent, the method further includes: Construct multiple second agents corresponding to multiple second W2W links in the second network environment; Within each time step, the second real-time communication data, the second historical communication data, and the second action trajectory sequence of each of the plurality of second agents are acquired. The second action trajectory sequence consists of the second communication data and the second execution action corresponding to each of the plurality of historical time steps. The second execution action is used to characterize the second agent's selection of the second communication sub-band and the second transmission power. The second historical communication data of each second agent is input into the initial model to predict the interference power value, thereby obtaining the second predicted interference power value of each second agent. The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial model for calculation to obtain the second Q value corresponding to each of the multiple second execution actions of each second agent. The second Q value is used to characterize the expected reward of the second agent performing the corresponding second execution action. A second target execution action is determined based on the plurality of second Q values. The second target execution action is the second execution action corresponding to the largest second Q value among the plurality of second Q values. The second target execution action is used to instruct the second W2W link to select the second communication subband and the second transmit power corresponding to the second target execution action. After the multiple second intelligent agents execute their respective second target execution actions, the total throughput of multiple terminal-to-router W2I links and the total delivery rate of the multiple second W2W links in the second network environment are calculated. Calculate the global reward value based on the total throughput and the total delivery rate; Using the training sample set corresponding to each time step, calculate the global joint Q value and the target Q value corresponding to each time step. The training sample set includes the second real-time communication data, the second target execution action and the global reward value of each second agent in the current time step, as well as the second real-time communication data and the second target execution action of each second agent in the next time step. Based on the global joint Q-value and the target Q-value, a loss function is constructed. With the goal of minimizing the loss function, the parameters of each module in the initial model are updated to obtain the target model.

5. The communication information determination method according to claim 4, characterized in that, The step of calculating the global joint Q-value and target Q-value for each time step using the training sample group corresponding to each time step includes: Calculate the global joint Q value based on the second real-time communication data and the second target execution action corresponding to each second agent in the current time step; The target Q value is calculated based on the global reward value, the second real-time communication data of each second agent in the next time step, and the second target execution action.

6. The communication information determination method according to claim 4, characterized in that, The initial model includes an initial GRU module and an initial LSTM module; The step of inputting the second historical communication data of each second agent into the initial model to predict the interference power value, and obtaining the second predicted interference power value of each second agent, includes: The second historical communication data of each second agent is input into the initial LSTM module of the initial model, and the temporal correlation of the second historical communication data is analyzed to obtain the second predicted interference power value of each second agent. The step involves inputting the second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent into the initial model for calculation, to obtain the second Q value corresponding to each of the multiple second execution actions of each second agent, including: The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial GRU module of the initial model, and the second Q value corresponding to the multiple second execution actions of each second agent is calculated using the forget gate and update gate mechanism.

7. The communication information determination method according to claim 6, characterized in that, The initial model also includes an initial RQN module; The step of inputting the second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent into the initial GRU module of the initial model, and calculating the second Q value corresponding to each of the multiple selectable second execution actions of each second agent using forget gate and update gate mechanisms, includes: The second real-time observation data, the second predicted interference power value, and the second action trajectory sequence of each second agent are input into the initial GRU module, and multiple initial second Q values ​​corresponding to multiple second execution actions of each second agent are calculated using the forget gate and update gate mechanism. Determine the mean of the plurality of initial second Q values ​​and the maximum value among the plurality of initial second Q values; The mean and the maximum value, as well as the second predicted interference power value, are input into the initial RQN module for calculation to obtain the second residual correction factor; The plurality of initial second Q values ​​are corrected using the second residual correction factor to obtain a plurality of second Q values.

8. A communication information determining device, characterized in that, The device includes: The first construction module is used to construct the first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first acquisition module is used to acquire the first real-time communication data, the first historical communication data and the first action trajectory sequence of the first intelligent agent. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent to select the first communication sub-band and the first transmission power. The first prediction module is used to input the first historical communication data into the target model to predict the interference power value, and obtain the first predicted interference power value of the first intelligent agent. The first processing module is used to input the first real-time observation data, the first predicted interference power value and the first action trajectory sequence into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. The first determining module is configured to determine a first target execution action based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

9. An electronic device, characterized in that, The electronic device includes a processor, the processor being used for: Construct a first intelligent agent corresponding to the first terminal-to-terminal W2W link in the first network environment; The first real-time communication data, the first historical communication data, and the first action trajectory sequence of the first intelligent agent are obtained. The first action trajectory sequence is composed of the first communication data and the first execution action corresponding to each historical time step of the first intelligent agent in multiple historical time steps. The first execution action is used to characterize the first intelligent agent's selection of the first communication sub-band and the first transmission power. The first historical communication data is input into the target model to predict the interference power value, and the first predicted interference power value of the first intelligent agent is obtained. The first real-time observation data, the first predicted interference power value, and the first action trajectory sequence are input into the target model to obtain multiple first Q values ​​corresponding to multiple first execution actions of the first agent. The first Q value is used to characterize the expected reward of the first agent performing the corresponding first execution action. A first target execution action is determined based on the plurality of first Q values. The first target execution action is the first execution action corresponding to the largest first Q value among the plurality of first Q values. The first target execution action is used to instruct the first W2W link to select the first communication subband and the first transmit power corresponding to the first target execution action.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the communication information determination method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the communication information determination method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps of the communication information determination method as described in any one of claims 1 to 7.

Images