Large-scale MIMO wireless communication codebook design method for digital twinning assisted reinforcement learning

Through the digital twin assisted reinforcement learning method, the problem of codebook design in large-scale MIMO technology requires frequent interaction with the real environment, and generate the best beam mode suitable for specific sites, reducing pilot overhead and improving beamforming gain.

CN118944717APending Publication Date: 2024-11-12SOUTHEAST UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411081417.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-12

Smart Images

  • Figure CN118944717A_ABST
    Figure CN118944717A_ABST
Patent Text Reader

Abstract

The invention discloses a large-scale MIMO wireless communication codebook design method for digital twin assisted reinforcement learning, and belongs to the technical field of wireless communication. Specifically, a digital twinning communication environment is constructed through information such as Gaussian noise, historical beam vectors reserved by a base station, receiving power and the like, and the method embeds signal composition information into a loss function of a neural network, so that the modeling efficiency of the digital twinning environment is effectively improved. And the deep reinforcement learning agent explores the wave beam in the digital twin environment, so that the overhead brought by interaction with the real environment is greatly reduced. A numerical result shows that compared with a classic beam-guided codebook, the average beam forming gain realized by the codebook designed by the digital twinning auxiliary reinforcement learning is obviously improved. Meanwhile, compared with a traditional reinforcement learning solution, the codebook performance of the digital twinning auxiliary reinforcement learning design is better, and the pilot frequency overhead is about 3% of that of the traditional reinforcement learning solution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence-assisted wireless communications, and specifically relates to a large-scale MIMO wireless communication codebook design method for digital twin-assisted reinforcement learning. Background Art

[0002] With the rapid growth of the number of access devices and applications, business scenarios have become increasingly complex and diverse, which has posed unprecedented challenges to the capacity of wireless communication systems. In this context, as a major breakthrough in the field of wireless communications, massive MIMO technology has attracted widespread attention in the industry. By densely deploying hundreds of antenna elements at the base station end, a large-scale array antenna system is constructed, which greatly improves the performance of the communication system. Specifically, massive MIMO technology can improve the spatial multiplexing and diversity gain effects of the communication system. At the same time, this technology can serve multiple users on the same time-frequency resources, and effectively improve the spectrum resources of the communication system. It is worth mentioning that massive MIMO combined with advanced precoding technology can finely control the beamforming of the antenna array so that its information transmission is aimed at the target user, which improves the communication quality of the target user and significantly reduces the interference to neighboring users. In addition, with the rapid development of technologies such as the Internet of Things, cloud computing, and big data, future wireless communication networks will face the challenges of massive device access and diversified application scenarios. Massive MIMO technology provides solid support for these emerging applications with its excellent performance. By continuously optimizing the configuration of the antenna array and the precoding algorithm, this technology can easily cope with the communication needs in different scenarios and provide strong guarantees for the high-speed and reliable connection of massive devices.

[0003] With the application of large-scale MIMO technology, expensive digital transceiver architectures cannot be fully deployed due to cost issues. Therefore, the industry tends to adopt pure analog or hybrid analog / digital architectures to achieve beamforming. In current beamforming technology, the widely used scheme is based on predefined codebooks. These codebooks consist of a large number of single-lobe beams. In order to cover all potential directions as much as possible, the beam pointing that constitutes the predefined codebook is usually evenly distributed throughout the space, so that the size of the predefined codebook is generally large, which also causes huge overhead for beam training and scanning. At the same time, in non-line-of-sight scenarios, the uniform distribution of the beams of the predefined codebook makes it possible that the included beams are not the optimal single-lobe beams, which causes the loss of beamforming gain.

[0004] For a long time, the codebook design problem has been focused on two dimensions: general environment and specific site. However, in most scenarios, beamforming codebooks designed based on general environments do not always provide the optimal beam, and such codebooks are often large in size to cover all potential directions, although the beams in some specific directions may not be frequently used in actual applications. For the codebook design of a specific site, it is usually necessary to collect a large amount of user channel state information data sets from a specific environment. However, the collection cost of channel state information data is high, and the channel state information data collected from real communication systems is often difficult to fully reflect the channel distribution conditions of all possible locations in the service area.

[0005] Deep reinforcement learning (DRL) has attracted great research attention in optimization problems and has become one of the means to solve codebook design. However, in practical applications, DRL usually requires a lot of interaction with the environment, which greatly increases the overhead of pilot and spectrum resources. In order to solve the problems of data collection and environmental interaction, a virtual environment for a specific site is urgently needed.

[0006] Digital twin, as the core technology of the next generation of mobile communications, is reshaping our understanding of the interaction between the real world and the virtual world. Its core lies in the high-precision simulation of the actual environment behavior through the deep integration of physical models and operation history data. Therefore, this invention aims to propose a modeling method for the digital twin environment to reduce the large amount of interaction between DRL and the real environment. Summary of the invention

[0007] The purpose of the present invention is to provide a large-scale MIMO wireless communication codebook design method assisted by digital twin reinforcement learning to address the problem that DRL needs to frequently interact with the real environment in the codebook design process. By cleverly embedding signal composition information into the loss function of the neural network, the equivalent channel information of a specific site is reconstructed using only Gaussian noise, a small amount of historical base station beam vector information and received power data, and the DRL agent can interactively learn in such a digital twin environment, and finally form a beam codebook adapted to the specific site;

[0008] In order to achieve the above object, the present invention adopts the following technical solution: a large-scale MIMO wireless communication codebook design method assisted by digital twin reinforcement learning, characterized in that it includes:

[0009] Step 1: Initialize the wireless communication environment, collect sensing information based on randomly generated sensing beams, and obtain M user clusters to complete user clustering;

[0010] Step 2: The environments of M user clusters interact with their corresponding reinforcement learning agents respectively to obtain M interaction data sets;

[0011] Step 3: construct M digital twin environment models using the M interaction data sets to simulate the environment of the M user clusters, and optimize the loss function of the digital twin environment model based on the Newton gradient algorithm;

[0012] Step 4: The M digital twin environment models are interacted with their corresponding reinforcement learning agents multiple times to generate the optimal beam patterns under the M digital twin environment models;

[0013] Step 5: Combine the optimal beam patterns generated under the M digital twin environment models into a codebook for a specific site.

[0014] Furthermore, the step 1 comprises the following steps:

[0015] Step 11: Initialize the wireless communication environment, generate a feasible beamforming vector set using constant modulus and finite resolution phase shifters, and randomly select a subset from the vector set to generate a sensing beam;

[0016] Step 12: using the previously generated sensing beam, receiving data sent by line-of-sight and non-line-of-sight users in the indoor environment, and then the base station captures and records the average received power of the data sent by line-of-sight and non-line-of-sight users in the indoor environment, and then obtains the approximate distribution of users in the wireless scenario based on the average received power;

[0017] Step 13: Based on the approximate distribution of users in the wireless scenario and the K-means algorithm, the K users in the scenario are divided into M user clusters, namely in It is a channel set of K users. There is no intersection between users in user clusters, thus completing the clustering of users.

[0018] Furthermore, the step 2 comprises the following steps:

[0019] Step 21: Actor1 network in the reinforcement learning agent is based on the current first state s t Make the corresponding first action a t Then the Critic network in the reinforcement learning agent is based on the first state-action group (s t ,a t ), output the corresponding Q 1 and Q 2 value;

[0020] Step 22: Set the first state s in step 21 t and the first action a tThe corresponding beamforming vector w and the average beamforming gain G corresponding to w are stored in the data set D m , and generate the first quadruple (s t ,a t ,r t ,s t+1 ) and stored in the experience replay memory_1 to train the Actor1 network and the Critic network, thereby completing an interaction;

[0021] Step 23: After a round of interaction is completed, the environment of the wireless communication scene changes once. The clustering of users in the new environment is completed based on the K-means algorithm and the sensing beam to obtain a new user cluster, that is, At the same time, the reinforcement learning agent always tracks the beamforming vector that achieves the best beamforming gain and generates a set of “temporary optimal” beamforming vectors Then use the Hungarian matching algorithm and the "temporary best" beamforming vector Assign new user clusters to reinforcement learning agents;

[0022] Step 24: After the new user cluster assignment is completed, a new round of interaction between the reinforcement learning agent and the environment of the user cluster is started, and steps 21 to 23 are repeated until multiple rounds of interaction are completed to obtain M interaction data sets.

[0023] Furthermore, the digital twin environment model in step 3 includes: an input layer, a hidden layer, and an output layer, wherein the randomly generated noise q enters the hidden layer through the input layer, is activated by the sine function sin in the hidden layer, and then outputs the real part of the equivalent channel through the hidden layer. and the imaginary part And the real part of the equivalent channel and the imaginary part Reshape the matrix into a matrix that conforms to the channel structure through the reshape function and Finally, it is output by the output layer.

[0024] Furthermore, the loss function of the digital twin environment model in step 3 is:

[0025]

[0026] Among them, V represents the size of the training set, ω is the parameter of the digital twin environment model, G p and They respectively represent the actual value of the pth beamforming gain and the beamforming gain value predicted by the digital twin environment model.

[0027] Furthermore, the specific steps of each digital twin environment model in step 4 interacting multiple times with its corresponding reinforcement learning agent include:

[0028] Step 41: Assigning the digital twin environment model to the reinforcement learning agent;

[0029] Step 42: The network parameters of Actor 1 in the reinforcement learning agent are passed to the network of Actor 2 in the reinforcement learning agent in a soft update manner to update the network parameters of Actor 2. The update expression of the network parameters of Actor 2 is:

[0030]

[0031] in, are the parameters of Actor1 network and Actor2 network respectively, ζ is the weight coefficient for updating network parameters;

[0032] Step 43: Update the Actor2 network with updated network parameters according to the second state s t ', generate the second action a t 'Provide it to the digital twin environment model, the digital twin environment model performs a performance pre-verification of the beam vector, and gives the reinforcement learning agent a second reward r according to the performance pre-verification of the beam vector t ', and generate the second quadruple (s t ',a t ',r t ',s t+1 '), and stored in the experience playback memory_2, and then the second quadruple (s t ',a t ',r t ',s t+1 ') Train the Actor2 network and the Critic network so that the Critic network can better guide the Actor2 network to output the beam pattern under the digital twin environment model;

[0033] Step 44: Repeat steps 41-43 until the number of interactions reaches d times, and obtain the beam patterns under d digital twin environment models, and screen the beam patterns under the d digital twin environment models to obtain the optimal beam pattern under the digital twin environment model.

[0034] Furthermore, after every c times of interaction in step 44, the loss function of the digital twin environment model is re-optimized once to fine-tune and update the parameter ω of the digital twin environment model.

[0035] Beneficial effects: By utilizing the excellent simulation and reconstruction capabilities of the digital twin technology, the present invention can pre-verify the average beamforming gain that may be achieved by the beam vector generated by the DRL agent in actual applications in the wireless communication environment simulated by the digital twin. At the same time, based on this gain result, the digital twin provides the DRL agent with corresponding reward feedback to evaluate the performance of the current beam vector and provide guidance for its subsequent optimization direction. At the same time, in this way, it can gradually iterate the optimization until the optimal beam pattern suitable for user clustering is generated, and all the best beam patterns are combined into a codebook for a specific site. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram of the implementation steps of the large-scale MIMO wireless communication codebook design method for digital twin assisted reinforcement learning in an embodiment of the present invention.

[0037] Figure 2 This is a flow chart of a method for designing a large-scale MIMO wireless communication codebook using digital twin-assisted reinforcement learning in an embodiment of the present invention.

[0038] Figure 3 It is a user distribution diagram of a wireless communication scenario and a specific site in an embodiment of the present invention.

[0039] Figure 4 The figure is a modeling and interaction flowchart of the digital twin environment used in the embodiment of the present invention.

[0040] Figure 5 It is a result diagram showing the simulation performance in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further explained below in conjunction with the accompanying drawings.

[0042] like Figure 1-2 As shown, the present invention provides a large-scale MIMO wireless communication codebook design method for digital twin assisted reinforcement learning, including:

[0043] Step 1: Initialize the wireless communication environment, collect sensing information based on randomly generated sensing beams, and obtain M user clusters to complete the clustering of users.

[0044] Step 2: The environments of M user clusters interact with their corresponding reinforcement learning agents respectively to obtain M interaction data sets.

[0045] Step 3: Use the M interaction data sets to build M digital twin environment models to simulate the environment of M user clusters, and optimize the loss function of the digital twin environment model based on the Newton gradient algorithm.

[0046] Step 4: The M digital twin environment models are interacted with their corresponding reinforcement learning agents multiple times to generate the optimal beam patterns under the M digital twin environment models.

[0047] Step 5: Combine the optimal beam patterns generated under the M digital twin environment models into a codebook for a specific site.

[0048] The present invention considers a large-scale MIMO system with N T The total number of potential users served by a base station with antennas is B, and users use a single antenna for communication. A fully analog transceiver architecture is used, and a γ-bit quantized phase shifter network is equipped to achieve beamforming. Therefore, the received signal during uplink transmission is expressed as:

[0049] y=w H h u s+w H n,

[0050] in, is the uplink channel vector between user u and the base station. is the noise vector received at the base station, is the symbol transmitted by the user and satisfies the average transmission power

[0051] w represents the beamforming vector of the base station, which can be expressed as:

[0052]

[0053] represents the phase in the beamforming vector w, and is 2 γ The elements in the set Θ are generated in (-π,π] in an equally spaced manner. Therefore, the signal-to-noise ratio at user u is:

[0054]

[0055] It is expressed as the transmit signal-to-noise ratio. Due to the constant modulus constraint ||w|| 2 = 1, select the beamforming vector from the codebook W to maximize the received signal-to-noise ratio of the target user:

[0056]

[0057] Therefore, the design problem of the beam codebook can be expressed as:

[0058]

[0059]

[0060]

[0061] Among them, w nm It's a codebook The nth element of the mth beam vector in, is the size of the codebook. The constraint in the formula is to make the generated beam vector meet the requirements of the phase shifter. Since the essence of the above problem is to find the optimal solution in a huge and discrete search space, the codebook design problem is difficult to solve.

[0062] In order to solve the problem that the above codebook design problem is difficult to solve.

[0063] like Figure 3 As shown, in step 1, first, the wireless communication environment is initialized, and then a feasible beamforming vector set is generated using a constant modulus and finite resolution phase shifter, and a subset is randomly selected from the vector set to generate a sensing beam.

[0064] Then, the previously generated sensing beam is used to receive data sent by line-of-sight and non-line-of-sight users in the indoor environment. The base station then captures and records the average received power of data sent by line-of-sight and non-line-of-sight users in the indoor environment, and then obtains the approximate distribution of users in the wireless scenario based on the average received power.

[0065] Finally, based on the approximate distribution of users in the wireless scenario and the K-means algorithm, the K users in the scenario are divided into M non-overlapping user clusters, namely in is a channel set of K users, and there is no intersection between users in the user clusters.

[0066] After the above steps, the codebook design problem is transformed into M easy-to-solve beam pattern optimization sub-problems. The beam pattern optimization sub-problem expression is:

[0067]

[0068]

[0069]

[0070] Among them, w n is the nth element in the beamforming vector, The mth user cluster.

[0071] In step 2, M reinforcement learning agents interact with the environments of M user clusters to obtain interaction data for building M digital twin environment models, thereby improving the modeling efficiency of the digital twin environment model.

[0072] Step 2 specifically includes:

[0073] Step 21: Actor1 network in the reinforcement learning agent is based on the current first state s t Make the corresponding first action a t Then the Critic network in the reinforcement learning agent is based on the first state-action group (s t ,a t ), output the corresponding Q 1 and Q 2 value.

[0074] Step 22: Set the first state s in step 21 t and the first action a t The corresponding beamforming vector w and the average beamforming gain G corresponding to w are stored in the data set At the same time, the first quadruple (s t ,a t ,r t ,s t+1 ) and stored in the experience replay memory_1 for training the Actor1 network and the Critic network, thus completing an interaction.

[0075] Step 23: After a round of interaction is completed, the environment of the wireless communication scene changes once. The clustering of users in the new environment is completed based on the K-means algorithm and the sensing beam to obtain a new user cluster, that is, At the same time, the reinforcement learning agent always tracks the beamforming vector that achieves the best beamforming gain and generates a set of “temporary optimal” beamforming vectors Then use the Hungarian matching algorithm and the "temporary best" beamforming vector New user clusters are assigned to reinforcement learning agents, where the optimal number of interactions per round is 50.

[0076] Step 24: After the new user cluster assignment is completed, a new round of interaction between the reinforcement learning agent and the user cluster environment is started, and steps 21 to 23 are repeated until multiple rounds of interaction are completed to obtain M interaction data sets, each of which includes multiple data sets And at this time the data set Store multiple first states s in t and the first action a t The corresponding beamforming vector w and the average beamforming gain G corresponding to w, where the best round is 20 times

[0077] In step 23, when new user clusters are assigned, it can be obtained by calculation The average beamforming gain of the beamforming vector in the new user cluster is:

[0078]

[0079] The average beamforming gain x is obtained by using the above mm' , assign the new user cluster to the reinforcement learning agent, and the assignment formula is:

[0080]

[0081] Where Z is a permutation matrix.

[0082] Then use the Hungarian matching algorithm to obtain M associated tuples Completed from The mapping from the original dataset to the new user clusters completes the continuous beam pattern learning of the reinforcement learning agent.

[0083] like Figure 4 As shown, in step 3, the interaction data obtained in step 2 is used to construct M digital twin environment models to simulate the environment of M user clusters for interacting with M reinforcement learning agents to reduce the large amount of interaction between the reinforcement learning agents and the real environment. At the same time, the loss function of the digital twin environment model is optimized based on the Newton gradient algorithm to improve the accuracy of the digital twin model.

[0084] In step 3, the digital twin environment model includes:

[0085] Input layer, hidden layer and output layer, where the randomly generated noise q enters the hidden layer through the input layer, is activated by the sine function sin in the hidden layer, and then outputs the real part of the equivalent channel through the hidden layer and the imaginary part And the real part of the equivalent channel and the imaginary part Reshape the matrix into a matrix that conforms to the channel structure through the reshape function and Finally, the output is output by the output layer. The input layer dimension is 8, and the hidden layer dimension is N T ×2, the real part of the equivalent channel and the imaginary part The dimension is N T ×N, the hidden layer activation and fully connected layer processing process can be expressed as follows:

[0086] x=sin(ω 1 q+b 1 ),

[0087]

[0088]

[0089] Where ω1 ,ω 2 ,ω 3 is the weight matrix between the fully connected layers, consisting of elements ω j,c It is composed of the weight of the connection between neuron j and neuron c.

[0090] And according to user h u The beamforming gain composition feature constrains the equivalent channel of the fully connected output, and we can get

[0091] g=w H h u ,

[0092]

[0093] Among them, w r ,w i and h r ,h i are the real and imaginary parts of the beam vector w and the user h u The real and imaginary parts of the channel.

[0094] Therefore, the loss function of the digital twin environment model is:

[0095]

[0096]

[0097]

[0098] in, They are the equivalent channel matrices The nth column vector in V represents the size of the training set. ω is the parameter of the digital twin environment model, which is represented by the weight matrix ω 1 ,ω 2 ,ω 3 constitute. and denote the pth beamforming vector w p The real and imaginary parts of .

[0099] The loss function of the above digital twin environment model is optimized based on the Newton gradient algorithm.

[0100] In step 4, the M digital twin environment models interact with their corresponding reinforcement learning agents multiple times to generate the optimal beam patterns under M different digital twin environment models, thereby solving the M beam pattern optimization sub-problems.

[0101] The specific steps of multiple interactions between each digital twin environment model and its corresponding reinforcement learning agent in step 4 include:

[0102] Step 41: Assign the digital twin environment model to the reinforcement learning agent.

[0103] Step 42: The network parameters of Actor 1 in the reinforcement learning agent are passed to the network of Actor 2 in the reinforcement learning agent in a soft update manner to update the network parameters of Actor 2. The update expression of the network parameters of Actor 2 is:

[0104]

[0105] in, are the parameters of Actor1 network and Actor2 network respectively, and ζ is the weight coefficient for updating the network parameters.

[0106] Step 43: Update the Actor2 network with updated network parameters according to the second state s t ', generate the second action a t 'Provide it to the digital twin environment model, the digital twin environment model performs a performance pre-verification of the beam vector, and gives the reinforcement learning agent a second reward r according to the performance pre-verification of the beam vector t ', and generate the second quadruple (s t ',a t ',r t ',s t+1 '), and stored in the experience playback memory_2, and then the second quadruple (s t ',a t ',r t ',s t+1 ') Train the Actor2 network and the Critic network so that the Critic network can better guide the Actor2 network to output the beam pattern under the digital twin environment model;

[0107] Step 44: Repeat steps 41-43 until the number of interactions reaches d times, and obtain d beam patterns under the digital twin environment model, and screen the d beam patterns under the digital twin environment model to obtain the best beam pattern under the digital twin environment model, where the optimal d is 40000.

[0108] In order to ensure that all actions finally applied to the system are effective, all generated actions need to be quantified. The quantification formula is:

[0109]

[0110] a k Represents all actions of the system, including the first action a t and the second action a t', and the quantized phase vector is passed through is converted into a beamforming vector.

[0111] The second reward adopts a binary mechanism and is selected from the set {1, -1}. The specific rules are as follows:

[0112]

[0113] In the formula, G t ,G t-1 They represent the average beamforming gain obtained in the current phase and the average beamforming gain at the previous moment respectively.

[0114] The same rules as above are used to process the first reward, so that the first reward r t .

[0115] After the number of interactions in step 44 reaches c times, the loss function of the digital twin environment model is re-optimized once to fine-tune and update the parameters ω of the digital twin environment model until the number of interactions reaches d times, completing the interaction between the reinforcement learning agent and the updated digital twin environment model, generating the optimal beam pattern under the digital twin environment model, making the digital twin environment model more accurate, and preventing the feedback provided to the reinforcement learning agent from being inaccurate due to the error of the model, so that the learning efficiency of reinforcement learning decreases, resulting in the performance of the generated codebook. The optimal value of c is 10000.

[0116] In step 5, the optimal beam patterns generated under the M digital twin environment models are combined into a codebook for a specific site.

[0117] In all the steps of the above embodiments, the reinforcement learning agents are deep reinforcement learning agents, and the algorithm used is the Soft Actor-Critic (SAC) algorithm.

[0118] like Figure 5 It is shown that the online digital twin-assisted SAC algorithm (DT_SAC) of the present invention only interacts with the real environment 1000 times to achieve the codebook design of a specific site. Compared with the traditional SAC solution, the pilot overhead is about 3% of the traditional SAC solution, and the codebook performance designed by the DT_SAC algorithm is improved by about 2%-6%. The 8-beam codebook designed by the DT_SAC algorithm is comparable to the performance of the classic beam steering codebook with 32 beams. As the codebook size increases, the performance far exceeds the classic beam steering codebook.

[0119] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A codebook design method for large-scale MIMO wireless communication based on digital twin assisted reinforcement learning, characterized in that: include: Step 1: Initialize the wireless communication environment, collect sensing information based on randomly generated sensing beams, and obtain M user clusters to complete user clustering; Step 2: The environments of M user clusters interact with their corresponding reinforcement learning agents respectively to obtain M interaction data sets; Step 3: construct M digital twin environment models using the M interaction data sets to simulate the environment of the M user clusters, and optimize the loss function of the digital twin environment model based on the Newton gradient algorithm; Step 4: The M digital twin environment models are interacted with their corresponding reinforcement learning agents multiple times to generate the optimal beam patterns under the M digital twin environment models; Step 5: Combine the optimal beam patterns generated under the M digital twin environment models into a codebook for a specific site.

2. The method for designing a codebook for large-scale MIMO wireless communication using digital twin-assisted reinforcement learning according to claim 1, characterized in that: The step 1 comprises the following steps: Step 11: Initialize the wireless communication environment, generate a feasible beamforming vector set using constant modulus and finite resolution phase shifters, and randomly select a subset from the vector set to generate a sensing beam; Step 12: using the previously generated sensing beam, receiving data sent by line-of-sight and non-line-of-sight users in the indoor environment, and then the base station captures and records the average received power of the data sent by line-of-sight and non-line-of-sight users in the indoor environment, and then obtains the approximate distribution of users in the wireless scenario based on the average received power; Step 13: Based on the approximate distribution of users in the wireless scenario and the K-means algorithm, the K users in the scenario are divided into M user clusters, namely in It is a channel set of K users. There is no intersection between users in user clusters, thus completing the clustering of users.

3. The method for designing a massive MIMO wireless communication codebook for digital twin-assisted reinforcement learning according to claim 1, characterized in that: The step 2 comprises the following steps: Step 21: Actor1 network in the reinforcement learning agent is based on the current first state s t Make the corresponding first action a t Then the Critic network in the reinforcement learning agent is based on the first state-action group (s t ,a t ), output the corresponding Q1 and Q2 values; Step 22: Set the first state s in step 21 t and the first action a t The corresponding beamforming vector w and the average beamforming gain G corresponding to w are stored in the data set At the same time, the first quadruple (s t ,a t ,r t ,s t+1 ) and stored in the experience replay memory_1 to train the Actor1 network and the Critic network, thereby completing an interaction; Step 23: After a round of interaction is completed, the environment of the wireless communication scene changes once. The clustering of users in the new environment is completed based on the K-means algorithm and the sensing beam to obtain a new user cluster, that is, At the same time, the reinforcement learning agent always tracks the beamforming vector that achieves the best beamforming gain and generates a set of "temporary optimal" beamforming vectors Then use the Hungarian matching algorithm and the "temporary best" beamforming vector Assign new user clusters to reinforcement learning agents; Step 24: After the new user cluster assignment is completed, a new round of interaction between the reinforcement learning agent and the environment of the user cluster is started, and steps 21 to 23 are repeated until multiple rounds of interaction are completed to obtain M interaction data sets.

4. The method for designing a massive MIMO wireless communication codebook for digital twin-assisted reinforcement learning according to claim 1, characterized in that: The digital twin environment model in step 3 includes: an input layer, a hidden layer, and an output layer, wherein the randomly generated noise q enters the hidden layer through the input layer, is activated by the sine function sin in the hidden layer, and then outputs the real part of the equivalent channel through the hidden layer. and the imaginary part And the real part of the equivalent channel and the imaginary part Reshape the matrix into a matrix that conforms to the channel structure through the reshape function and Finally, it is output by the output layer.

5. The method for designing a codebook for large-scale MIMO wireless communication using digital twin-assisted reinforcement learning according to claim 1, characterized in that: The loss function of the digital twin environment model in step 3 is: Among them, V represents the size of the training set, ω is the parameter of the digital twin environment model, G p and They respectively represent the actual value of the pth beamforming gain and the beamforming gain value predicted by the digital twin environment model.

6. The method for designing a codebook for large-scale MIMO wireless communication using digital twin-assisted reinforcement learning according to claim 1, characterized in that: The specific steps of multiple interactions between each digital twin environment model and its corresponding reinforcement learning agent in step 4 include: Step 41: Assigning the digital twin environment model to the reinforcement learning agent; Step 42: The network parameters of Actor 1 in the reinforcement learning agent are passed to the network of Actor 2 in the reinforcement learning agent in a soft update manner to update the network parameters of Actor 2. The update expression of the network parameters of Actor 2 is: in, are the parameters of Actor1 network and Actor2 network respectively, ζ is the weight coefficient for updating network parameters; Step 43: Update the Actor2 network with updated network parameters according to the second state s t ', generate the second action a t 'Provided to the digital twin environment model, the digital twin environment model performs a performance pre-verification of the beam vector, and gives the reinforcement learning agent a second reward rt' according to the performance pre-verification of the beam vector, and generates a second quadruple (s t ',a t ',r t ',s t+1 '), and stored in the experience playback memory_2, and then the second quadruple (s t ',a t ',r t ',s t+1 ') Train the Actor2 network and the Critic network so that the Critic network can better guide the Actor2 network to output the beam pattern under the digital twin environment model; Step 44: Repeat steps 41-43 until the number of interactions reaches d times, and obtain the beam patterns under d digital twin environment models, and screen the beam patterns under the d digital twin environment models to obtain the optimal beam pattern under the digital twin environment model.

7. The method for designing a codebook for large-scale MIMO wireless communications using digital twin-assisted reinforcement learning according to claim 6, characterized in that: After the number of interactions in step 44 reaches c times, the loss function of the digital twin environment model is re-optimized once to fine-tune and update the parameter ω of the digital twin environment model.

Citation Information

Cited By

  • Beam management method and system based on digital twinning

    CN120186656A

  • Method for constructing channel fingerprint twin, and system

    WO2025232288A1