Dynamic beam forming control system and method based on multi-target reinforcement learning

By using a dynamic beamforming control system based on multi-objective reinforcement learning, the problems of insufficient adaptability and high computational complexity of existing technologies in complex dynamic wireless communication environments are solved. Predictive beamforming and multi-objective optimization are achieved, improving connection robustness and communication real-time performance, and reducing system energy consumption.

CN121308790APending Publication Date: 2026-01-09LIAONING TECHNICAL UNIVERSITY
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202511602767.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-11-04
Publication Date
2026-01-09

AI Technical Summary

Technical Problem

Existing beamforming technologies suffer from insufficient adaptability, multi-objective optimization conflicts, lack of scene understanding and prediction capabilities, poor coordination and insufficient network-level optimization, high computational complexity and poor real-time performance when facing complex and dynamically changing wireless communication environments, making it difficult to meet rapidly changing communication needs.

Method used

A dynamic beamforming control system based on multi-objective reinforcement learning is adopted. The scene perception module collects and processes multimodal environmental data to generate scene situation data. The central control module generates the optimal cooperative beamforming action based on the multi-objective reinforcement learning algorithm, and executes the optimal radio frequency beam through the cooperative antenna array module. Combined with the system feedback evaluation module and experience storage module, continuous learning and optimization are carried out to achieve the updating of multi-objective rewards and the self-improvement of system status.

Benefits of technology

It significantly improves performance in complex dynamic environments, enables predictive beamforming, enhances connectivity robustness, dynamically optimizes multiple key performance indicators, reduces system energy consumption, meets real-time communication requirements, and requires no large amount of pre-labeled data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121308790A_ABST
    Figure CN121308790A_ABST
Patent Text Reader

Abstract

The invention discloses a dynamic beam forming control system and method based on multi-target reinforcement learning, and relates to the technical field of intelligent antennas and arrays, and the system comprises a scene sensing module which collects and processes multi-mode original environment data, and generates scene situation data; the central control module is used for constructing a comprehensive state vector at the current moment according to the scene situation data and the channel state information, and generating an optimal collaborative beamforming action based on a multi-target reinforcement learning algorithm; the cooperative antenna array module is used for executing an optimal cooperative beam forming action and generating an optimal radio frequency beam; the system feedback evaluation module is used for monitoring the system performance after the optimal cooperative beam forming action is executed, calculating a multi-target reward vector, updating the system state and obtaining a comprehensive state vector at the next moment; and the experience storage module is used for constructing experience tuples and storing the experience tuples to the experience playback pool, and regularly updating parameters of the strategy network and the value network based on the experience tuples. According to the invention, the beamforming control performance in a wireless communication environment can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of smart antenna and array technology, and relates to, but is not limited to, a dynamic beamforming control system and method based on multi-objective reinforcement learning. Background Technology

[0002] With the rapid development of wireless communication technology, especially in 5G and future 6G communication systems, technologies such as Massive Multiple-Input Multiple-Output (MIMO) and Intelligent Reflecting Surface (IRS) are crucial. One of the core aspects of these advanced technologies is beamforming, which, by precisely controlling the signal amplitude and phase of each antenna element in an antenna array, focuses wireless energy onto specific users or areas, thereby significantly improving signal coverage, data throughput, spectral efficiency, and system capacity.

[0003] In existing technologies, beamforming techniques mainly include classic methods such as Maximum Ratio Combining (MRC) and Zero Forcing (ZF) based on Channel State Information (CSI) feedback. In addition, schemes based on convex optimization or heuristic algorithms have also been widely studied and applied. The ultimate goal of these methods is to find the optimal beamforming weights in a mathematical model. While existing beamforming technologies have improved the performance of wireless communication systems to some extent, they also have significant limitations when dealing with increasingly complex and dynamically changing wireless communication environments. These limitations are mainly manifested in the following ways: Existing technologies heavily rely on real-time feedback of Channel Indicator (CSI). In scenarios involving high-speed mobile users or dynamic obstructions, the channel changes often outpace the CSI feedback and update cycle, causing beamforming strategies to fail to track channel changes in a timely manner, thus affecting connection stability and service quality. In complex communication needs, users and network operators often expect to optimize multiple conflicting performance metrics simultaneously. However, existing beamforming strategies typically optimize only a single metric, making it difficult to achieve an effective trade-off between multiple conflicting objectives and failing to meet the increasingly diverse needs of service scenarios. Furthermore, existing technologies often treat the wireless channel as a "black box," lacking a deep understanding and rational interpretation of the physical communication environment. The inability to proactively acquire and utilize key physical environment information leads to a lack of foresight in beamforming strategies, making it impossible to predict potential channel changes in advance, thus limiting the robustness of the system and user experience. In complex network environments composed of multiple base stations or multiple antenna elements, existing beamforming strategies lack effective coordination mechanisms. When beamforming is performed independently by each base station or antenna element, inter-cell interference is easily generated, and it is difficult to achieve network-level joint optimization, limiting the overall network resource utilization efficiency and performance. For large-scale antenna arrays, existing beamforming algorithms based on convex optimization and other methods require a large number of iterative calculations, resulting in extremely high computational complexity. This makes them difficult to deploy and apply in high real-time scenarios and cannot meet rapidly changing communication needs. In addition, most machine learning-based beamforming schemes rely on large amounts of labeled data or adopt single-objective reinforcement learning, making it difficult to simultaneously address the core challenge of multi-objective collaborative optimization in dynamic environments.

[0004] Therefore, there is an urgent need for a more comprehensive dynamic beamforming control system to address the common problems in existing technologies when dealing with beamforming control in complex dynamic wireless communication environments, such as insufficient adaptability and large lag, multi-objective optimization conflicts, lack of scene understanding and prediction capabilities, poor coordination and insufficient network-level optimization, high computational complexity and poor real-time performance. This would enable a significant improvement in the overall performance, robustness, energy efficiency, and user experience of the system in complex dynamic environments. Summary of the Invention

[0005] This application provides a dynamic beamforming control system and method based on multi-objective reinforcement learning.

[0006] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a dynamic beamforming control system based on multi-objective reinforcement learning. The system includes a scene perception module, a central control module, a cooperative antenna array module, a system feedback evaluation module, and an experience storage module, wherein: The scene perception module collects and processes multimodal raw environmental data to generate scene situational data, including physical environment data and user equipment status information. The central control module constructs the current comprehensive state vector based on the scene situational data and CSI, and generates the optimal cooperative beamforming action using a multi-objective reinforcement learning (MORL) algorithm, resolving the optimal cooperative beamforming action into beamforming control parameters. The cooperative antenna array module executes the optimal cooperative beamforming action to generate the optimal radio frequency beam. The system feedback evaluation module monitors the system performance after the cooperative antenna array module executes the optimal cooperative beamforming action, calculates the multi-objective reward vector, updates the system state, and obtains the comprehensive state vector for the next moment. The experience storage module constructs experience tuples using the current comprehensive state vector, the optimal cooperative beamforming action, the multi-objective reward vector, and the next comprehensive state vector, stores the experience tuples in the experience replay pool, and periodically updates the parameters of the policy (Actor) network and the value (Critic) network based on the experience tuples.

[0007] The technical solution provided in this application collects and processes multimodal raw environmental data through a scene perception module, predicts user movement and environmental changes, and generates scene situational data. This enables deep perception of the physical environment, automatic identification and adaptation to different communication scenarios, significantly improving performance in complex dynamic environments. It also achieves predictive beamforming, adjusting beams in advance to avoid interruptions, demonstrating strong foresight and prediction capabilities, and greatly enhancing connection robustness. In the central control module, a comprehensive state vector for the current moment is constructed based on scene situational data and CSI, and the optimal cooperative beamforming action is generated based on the MORL algorithm. This achieves simultaneous dynamic optimization of multiple key performance indicators, determining Pareto optimal or satisfactory solutions based on actual service requirements and network status, overcoming the limitations of single-objective optimization, and further resolving the optimal cooperative beamforming action into beamforming control parameters. The cooperative antenna array module executes the optimal cooperative beamforming action to generate the optimal radio frequency beam. In the system feedback evaluation module… In this module, the system performance after the cooperative antenna array module performs the optimal cooperative beamforming action is monitored. The multi-target reward vector is calculated and the system state is updated to obtain the comprehensive state vector at the next moment. In the experience storage module, experience tuples are constructed using the comprehensive state vector at the current moment, the optimal cooperative beamforming action, the multi-target reward vector, and the comprehensive state vector at the next moment. The experience tuples are stored in the experience replay pool, and the parameters of the Actor network and Critic network are updated periodically based on the experience tuples to achieve continuous learning and self-optimization of the system. This process does not require a large amount of pre-labeled data, allowing the system performance to continuously improve over time. After offline training, online decision-making is based on neural network forward computation, which has low complexity and fast response speed, meeting the real-time communication requirements. In addition, the technical solution provided in this application embodiment can intelligently reduce the transmission power or put some modules into sleep mode while ensuring performance, thereby significantly reducing system energy consumption.

[0008] Optionally, the system further includes a collaborative control module, which coordinates multiple base stations to perform joint beamforming and performs cross-base station joint optimization and interference coordination.

[0009] Optionally, the scene perception module includes a data acquisition unit, a data preprocessing unit, a user distribution perception unit, a prediction and inference unit, and a scene classification and feature extraction unit, wherein: the data acquisition unit is used to acquire multimodal raw environmental data based on multi-sensor fusion technology; the data preprocessing unit is used to perform timestamp alignment, coordinate unification, and noise reduction filtering on the multimodal raw environmental data to generate multimodal environmental data; the user distribution perception unit is used to process the multimodal environmental data through a wireless positioning algorithm to generate spatial distribution information of users; the prediction and inference unit is used to perform time-series prediction and spatial feature extraction based on the spatial distribution information and multimodal environmental data through a machine learning model to obtain user trajectory prediction results, real-time environmental maps, and potential channel event prediction results; the scene classification and feature extraction unit is used to classify the current scene and generate scene feature vectors based on the user trajectory prediction results, real-time environmental maps, and potential channel event prediction results through a deep learning model.

[0010] Optionally, the prediction inference unit includes a spatial feature extraction subunit, a trajectory prediction subunit, and a state prediction and anomaly detection subunit, wherein: the spatial feature extraction subunit is used to process image and environmental map data using a convolutional neural network, identify static and dynamic obstacles, and construct and update a real-time environmental map; the trajectory prediction subunit is used to process user movement trajectory data using a long short-term memory network, a gated recurrent unit, or a Transformer neural network, predict the future movement trajectory of the user device, and obtain the user trajectory prediction result; the state prediction and anomaly detection subunit is used to perform system state prediction using a Markov chain model, and predict potential channel blocking events based on the user trajectory prediction result and the real-time environmental map, and obtain the potential channel event prediction result.

[0011] Optionally, the collaborative control module includes a spatial information exchange unit, a joint optimization unit, an interference coordination unit, and a policy distribution unit, wherein: the information exchange unit is used to acquire scene situation data and CSI of each base station through a preset interface; the joint optimization unit is used to determine a global collaborative beamforming strategy based on the scene situation data and CSI of each base station to maximize the overall network performance; the interference coordination unit is used to generate cross-base station joint beamforming control parameters according to the global collaborative beamforming strategy and perform centralized or distributed interference coordination to suppress inter-cell interference; and the policy distribution unit is used to distribute the coordinated joint beamforming control parameters to each base station.

[0012] Optionally, the central control module includes a state construction unit, a multi-objective reinforcement learning agent unit, and a beamforming strategy parsing unit, wherein: the state construction unit is used to fuse scene situational data and CSI to generate a comprehensive state vector at the current moment; the scene situational data includes user spatial distribution information, user trajectory prediction results, real-time environment map, and scene feature vectors; the CSI includes channel matrix, signal-to-noise ratio, and interference level; the multi-objective reinforcement learning agent unit is used to make intelligent decisions based on the comprehensive state vector at the current moment to generate the optimal cooperative beamforming action; the beamforming strategy parsing unit is used to parse the optimal cooperative beamforming action into beamforming control parameters, which include beamforming weight parameters, power allocation parameters, antenna selection parameters, and time-frequency resource allocation parameters.

[0013] Optionally, the multi-objective reinforcement learning agent unit includes an action generation subunit, a cumulative reward calculation subunit, and a multi-objective optimization unit, wherein: the action generation subunit is used to generate cooperative beamforming actions based on the comprehensive state vector at the current time through an Actor network; the cumulative reward calculation subunit is used to evaluate the expected long-term cumulative reward of the comprehensive state vector at the current time and the cooperative beamforming actions through a Critic network, the expected long-term cumulative reward being calculated based on a multi-objective reward function; the multi-objective optimization unit is used to handle the conflicts between multiple optimization objectives using Pareto optimization, ε-constraints, or weight adaptive mechanisms to maximize the multi-objective reward function and obtain the optimal cooperative beamforming action, the multi-objective reward function being constructed through system throughput, communication latency, network energy consumption, user fairness, interference suppression level, and coverage.

[0014] Optionally, the system feedback evaluation module includes a system performance monitoring unit, a reward calculation unit, and a state update unit, wherein: the system performance monitoring unit is used to monitor the system performance indicators in real time after the cooperative antenna array module performs optimal cooperative beamforming, and the system performance indicators include system throughput, communication latency, network energy consumption, interference level, and user service quality satisfaction; the reward calculation unit is used to calculate a multi-objective reward vector based on a multi-objective reward function according to the system performance indicators; the state update unit is used to obtain the comprehensive state vector at the next moment according to changes in the environment and system, and the comprehensive state vector at the next moment integrates new scene situation data and CSI.

[0015] Secondly, embodiments of this application provide a dynamic beamforming control method based on multi-objective reinforcement learning. The method is applied to a dynamic beamforming control system based on multi-objective reinforcement learning. The system includes a scene perception module, a central control module, a cooperative antenna array module, a system feedback evaluation module, and an experience storage module. The method includes: acquiring and processing multimodal raw environmental data to generate scene situational data, whereby the multimodal raw environmental data includes physical environment data and user equipment status information; constructing a comprehensive state vector at the current moment based on the scene situational data and CSI; and generating optimal cooperative beamforming control based on the MORL algorithm. The beamforming action is parsed into beamforming control parameters; the optimal cooperative beamforming action is executed to generate the optimal radio frequency beam; the system performance after the cooperative antenna array module executes the optimal cooperative beamforming action is monitored, the multi-target reward vector is calculated and the system state is updated, and the comprehensive state vector at the next time step is obtained; an experience tuple is constructed using the comprehensive state vector at the current time step, the optimal cooperative beamforming action, the multi-target reward vector, and the comprehensive state vector at the next time step, and the experience tuple is stored in the experience replay pool, and the parameters of the Actor network and Critic network are updated periodically based on the experience tuple.

[0016] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps in the above-described dynamic beamforming control method based on multi-objective reinforcement learning.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the aforementioned dynamic beamforming control method based on multi-objective reinforcement learning.

[0018] The beneficial effects of the technical solutions provided in this application include at least the following: This application provides a dynamic beamforming control system and method based on multi-objective reinforcement learning. Through a scene perception module, it collects and processes multimodal raw environmental data, predicts user movement and environmental changes, and generates scene situational data. This allows for deep perception of the physical environment, automatic identification and adaptation to different communication scenarios, significantly improving performance in complex dynamic environments. It also achieves predictive beamforming, adjusting beams in advance to avoid interruptions, demonstrating strong foresight and prediction capabilities, and greatly enhancing connection robustness. In the central control module, a comprehensive state vector is constructed based on scene situational data and CSI (Content Situation Indicator). The optimal cooperative beamforming action is generated based on the MORL algorithm, simultaneously and dynamically optimizing multiple key performance indicators. Pareto optimal or satisfactory solutions are determined based on actual service requirements and network status, overcoming the limitations of single-objective optimization. Furthermore, the optimal cooperative beamforming action is parsed into beamforming control parameters. The cooperative control module enables intelligent coordination among multiple antennas or base stations, effectively suppressing interference and improving overall network performance and resource utilization. Cooperative antenna arrays are also included. The column module executes the optimal cooperative beamforming action to generate the optimal radio frequency beam. In the system feedback evaluation module, the system performance after the cooperative antenna array module executes the optimal cooperative beamforming action is monitored, the multi-target reward vector is calculated and the system state is updated to obtain the comprehensive state vector at the next moment. In the experience storage module, an experience tuple is constructed using the comprehensive state vector at the current moment, the optimal cooperative beamforming action, the multi-target reward vector, and the comprehensive state vector at the next moment. The experience tuple is stored in the experience replay pool, and the parameters of the Actor network and Critic network are updated periodically based on the experience tuple to achieve continuous learning and self-optimization of the system. This process does not require a large amount of pre-labeled data, so the system performance continues to improve over time. After offline training, online decision-making is based on neural network forward computation, which has low complexity and fast response speed, meeting the real-time communication requirements. In addition, the technical solution provided in this application embodiment can intelligently reduce the transmission power or put some modules into sleep mode while ensuring performance, thereby significantly reducing system energy consumption. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A schematic diagram of a dynamic beamforming control system based on multi-objective reinforcement learning provided in an embodiment of this application; Figure 2 A schematic diagram of a scene perception module provided in an embodiment of this application; Figure 3A schematic diagram of a central control module provided in an embodiment of this application; Figure 4 A schematic diagram of a network structure for intelligent decision-making based on the MO-DDPG algorithm is provided in an embodiment of this application; Figure 5 A schematic diagram of a system feedback evaluation module provided in an embodiment of this application; Figure 6 A flowchart illustrating a dynamic beamforming control method based on multi-objective reinforcement learning, provided for embodiments of this application; Figure 7 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0022] It should be noted that the terms "first, second, and third" used in the embodiments of this application are merely to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0023] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0024] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0025] In view of the current problems in dynamic beamforming control in the field of smart antenna and array technology, this application provides a dynamic beamforming control system and method based on multi-objective reinforcement learning.

[0026] The technical solution of this application is described below, starting with the system implementation of this application.

[0027] Please refer to Figure 1 It shows a schematic diagram of a dynamic beamforming control system based on multi-objective reinforcement learning provided in an embodiment of this application, such as... Figure 1 As shown, the system includes a scene perception module 01, a central control module 02, a cooperative antenna array module 03, a system feedback evaluation module 04, and an experience storage module 05. The scene perception module 01, central control module 02, cooperative antenna array module 03, system feedback evaluation module 04, and experience storage module 05 are interconnected. The scene perception module 01 is used to collect and process multimodal raw environmental data to generate scene situational data. The multimodal raw environmental data includes physical environment data and user equipment status information. The central control module 02 is used to construct the comprehensive state vector at the current moment based on the scene situational data and CSI, and generate the optimal cooperative beamforming action based on the MORL algorithm. The optimal cooperative beamforming action is then parsed into beamforming control parameters. The cooperative antenna array module 03 is used to execute the optimal cooperative beamforming action and generate the optimal radio frequency beam. The system feedback evaluation module 04 is used to monitor the system performance after the cooperative antenna array module executes the optimal cooperative beamforming action, calculate the multi-target reward vector and update the system state, and obtain the comprehensive state vector at the next moment. The experience storage module 05 is used to construct experience tuples based on the comprehensive state vector at the current moment, the optimal cooperative beamforming action, the multi-target reward vector and the comprehensive state vector at the next moment, store the experience tuples in the experience replay pool, and periodically update the parameters of the Actor network and the Critic network based on the experience tuples.

[0028] In the embodiments of this application, please refer to Figure 2 It shows a schematic diagram of a scene perception module provided in an embodiment of this application, such as Figure 2As shown, the scene perception module 01 includes a data acquisition unit, a data preprocessing unit, a user distribution perception unit, a prediction and inference unit, and a scene classification and feature extraction unit. The data acquisition unit is used to acquire multimodal raw environmental data based on multi-sensor fusion technology. Specifically, the multimodal raw environmental data includes physical environment data and user equipment status information. In the specific acquisition of physical environment data, visual sensors, such as optical camera arrays, acquire image data to identify the number of users, user locations, and obstacles. Radar sensors, such as lidar or millimeter-wave radar, acquire point cloud data to achieve accurate ranging, construct a 3D environmental map, and track user movement trajectories. Environmental sensors acquire more refined environmental parameters, such as temperature and humidity sensors. In the specific acquisition of user equipment status information, positioning sensors, such as the Global Positioning System (GPS) or BeiDou satellites, acquire user equipment positioning data. The network core network acquires geographic location and map data to enrich the physical environment data. In addition, radio frequency sensors acquire wireless signal data, including Received Signal Strength Indicator (RSSI) and CSI. The technical solution provided in this application uses multi-sensor fusion technology to acquire multimodal raw environmental data, which can maximize the acquisition of environmental data and improve the comprehensiveness of the data.

[0029] In this embodiment, the data preprocessing unit performs timestamp alignment, coordinate unification, and noise reduction filtering on the original multimodal environmental data to generate multimodal environmental data. Specifically, firstly, the original multimodal environmental data is timestamped using a data synchronizer. Then, a coordinate system transformation is performed on each timetamp-aligned environmental data, converting them to a unified coordinate system. Finally, a fusion filtering algorithm based on wavelet transform and Kalman filtering is used to perform noise reduction filtering on the environmental data in the unified coordinate system, generating multimodal fused data. The user distribution sensing unit processes the multimodal environmental data using a wireless positioning algorithm to perform user positioning and distribution analysis, generating spatial distribution information for users. Specifically, it utilizes Angle of Arrival (AoA), Time Difference of Arrival (TDoA), RSSI, etc., comprehensively utilizing data collected by multiple sensors in the multimodal fused data to achieve accurate perception of user location and spatial distribution feature analysis, generating spatial distribution information for users.

[0030] In this embodiment, the prediction inference unit is used to perform temporal prediction and spatial feature extraction based on spatial distribution information and multimodal environmental data using a machine learning model, to obtain user trajectory prediction results, real-time environmental maps, and potential channel event prediction results. Specifically, in the spatial feature extraction subunit, spatial data such as image and environmental map data are processed using a convolutional neural network to extract spatial features, identify static and dynamic obstacles, and construct and update the real-time environmental map. In the trajectory prediction subunit, temporal data such as user movement trajectory data are processed using a long short-term memory network, a gated recurrent unit, or a Transformer neural network to capture its dynamic changes, predict the future movement trajectory of the user device, and obtain user trajectory prediction results. In the state prediction and anomaly detection subunit, a Markov chain model is used for system state prediction, identifying static obstacles such as buildings and walls, and dynamic obstacles such as pedestrians and vehicles in the environment. Based on user trajectory prediction results, real-time environmental maps, and obstacle movement recognition results, potential channel blocking events are predicted, allowing for early prediction of possible signal obstruction events and obtaining potential channel event prediction results. Furthermore, spatial distribution information is processed through a positioning algorithm, enabling high-precision estimation of the user's current location and movement speed. The technical solution provided in this application, through in-depth processing and intelligent prediction of spatial distribution information and multimodal environmental data, provides a solid foundation for the realization of predictive beamforming.

[0031] In this embodiment, the scene classification and feature extraction unit is used to automatically classify the current scene based on the user trajectory prediction results, real-time environment map and potential channel event prediction results, using a deep learning model. The scene classification includes indoor dense scenes, outdoor sparse scenes, urban macro base station coverage scenes, suburban coverage scenes, high-speed mobile scenes, industrial IoT scenes, etc. At the same time, it generates feature vectors that reflect the characteristics of the current scene, i.e., scene feature vectors, to provide a basis for subsequent decision-making.

[0032] In the embodiments of this application, please refer to Figure 3 It shows a schematic diagram of a central control module provided in an embodiment of this application, such as Figure 3As shown, the central control module 02 includes a state construction unit, a multi-objective reinforcement learning agent unit, and a beamforming strategy parsing unit. The state construction unit fuses scene situational data and CSI to generate a comprehensive state vector for the current moment. Specifically, scene situational data is acquired from the scene perception module 01, including user spatial distribution information, user trajectory prediction results, real-time environment map, and scene feature vectors. CSI is directly acquired from the communication link, and includes the channel matrix, signal-to-noise ratio, and interference level. By fusing scene situational data and CSI, a high-dimensional comprehensive state vector for the current moment is generated. This comprehensive state vector can represent complete information about the current communication environment and system state. The formula for expressing the comprehensive state vector for the current moment is as follows: ; In the formula, This represents the overall state vector at the current moment; Represents the user location matrix; Represents the base station location matrix; Represents the channel matrix; Indicates the interference spectrum; This represents the scene feature vector.

[0033] In this embodiment, the multi-objective reinforcement learning agent unit is used to make intelligent decisions based on the comprehensive state vector at the current moment, generating an optimal cooperative beamforming action. Specifically, after receiving the comprehensive state vector at the current moment, the multi-objective reinforcement learning agent unit makes intelligent decisions based on its built-in MORL algorithm. This MORL algorithm can be the Multi-Objective Deep Deterministic Policy Gradient (MO-DDPG) algorithm, Pareto Q-Learning (PQL), etc., thereby generating an optimal cooperative beamforming action. Please refer to [reference needed]. Figure 4 This document illustrates a network structure diagram for intelligent decision-making based on the MO-DDPG algorithm, provided in an embodiment of this application. The intelligent decision-making process is completed using a multi-objective reinforcement learning agent unit. This agent unit includes an action generation subunit, a cumulative reward calculation subunit, and a multi-objective optimization unit. The action generation subunit generates cooperative beamforming actions based on the current integrated state vector through the Actor network. Figure 4In the network structure, the input layer receives the comprehensive state vector at the current moment, the long short-term memory network layer processes the temporal features in the comprehensive state vector at the current moment to capture dynamic changes in the environment, the attention mechanism layer weights and focuses on the key features in the comprehensive state vector at the current moment to improve decision accuracy, the fully connected layer performs nonlinear transformation and fusion of features, and the output layer outputs the cooperative beamforming action. The formula for the cooperative beamforming action is expressed by the following formula: ; in, Indicates cooperative beamforming action, This represents the beam weight vector. Additionally, in the target Action network, parameters are synchronized from the main Actor network using a soft update mechanism (such as Polyak averaging) to improve training stability.

[0034] Furthermore, the cumulative reward calculation subunit is used to evaluate the expected long-term cumulative reward of the current integrated state vector and the cooperative beamforming action through the Critic network. This expected long-term cumulative reward is calculated based on a multi-objective reward function, which simultaneously considers and balances at least two objectives among system throughput, communication latency, network energy consumption, user fairness, interference suppression level, and coverage range. This enables the communication system to intelligently determine the optimal or satisfactory solution on the Pareto front across multiple dimensions such as coverage, capacity, and energy efficiency. The expected long-term cumulative reward represents the sum of the discounted cumulative multi-objective rewards expected to be obtained in the future after performing the cooperative beamforming action, under the current integrated state vector. The expected long-term cumulative reward is mainly used to evaluate the long-term value of the cooperative beamforming action, and then uses this long-term value to guide the optimization of the beamforming strategy, achieving a long-term trade-off between multiple objectives. Figure 4 In the network structure, the current state vector is received through the state input layer, and spatial features are extracted through convolutional layers. Co-beamforming actions are received through the action input layer and processed through the embedding layer. The current state vector and co-beamforming actions are fused through a fusion layer (e.g., by concatenation or weighted summation). Deep feature extraction and nonlinear mapping are performed through fully connected layers. The expected long-term cumulative reward is output through a reward output layer. Furthermore, in the target Critic network, parameters are synchronized from the main Critic network using a soft update mechanism to calculate the expected long-term cumulative reward and reduce estimation bias. The multi-objective reward function is calculated by weighting the rewards of each objective, and its expression is given by the following formula: ; In the formula, Represents a multi-objective reward function; It represents system throughput and measures the amount of data transmitted per unit of time. It represents communication latency and measures the time required for a signal to travel from the sending end to the receiving end; Network energy efficiency measures the efficiency of energy utilization. It represents user fairness and can measure the degree of fairness in the allocation of resources to different users; It indicates the level of interference suppression and can measure the degree of interference inside and outside the system; It indicates the coverage area and can measure the effective service range and signal quality of a signal; ~ This indicates a dynamically adjustable weighting coefficient, which can be dynamically configured according to different scenario types, business needs, or operator strategies.

[0035] In a multi-objective reward function, the function can be maximized by maximizing positive reward terms and minimizing negative penalty terms. Positive reward terms include throughput rewards, energy efficiency rewards, fairness rewards, and coverage rewards. Throughput reward is the term expressed in the multi-objective reward function formula. This item is used to encourage improvements in system data transmission capabilities. The system throughput is calculated using the following formula: ; In the formula, Indicates actual throughput; This represents the theoretical maximum throughput; This represents the signal-to-interference-plus-noise ratio. In video streaming scenarios, the weighting method is to increase the system throughput. In low-power scenarios, the weight adjustment method is to reduce the weight of system throughput. Energy efficiency rewards are expressed in the multi-objective reward function formula. This item is used to optimize energy utilization. The formula for calculating network energy efficiency is expressed as follows: ; In the formula, Indicates the number of bits transmitted; Representing energy, the weighting method in IoT devices is to increase network energy efficiency. Under mains-powered base stations, the weighting method is to reduce the weight of network energy efficiency. Fairness reward is the formula expressed by a multi-objective reward function. This item is used to ensure the fair allocation of resources among users. User fairness is calculated using Jain's Fairness Index, and the formula for calculating user fairness is expressed as follows: ; In the formula, Indicates the number of users. Indicates user The throughput is considered, and the fairness range is 0 to 1, where 0 represents the most unfair and 1 represents perfect fairness. The coverage reward is expressed in the multi-objective reward function formula. This item is used to extend the effective service area and reduce coverage blind spots. The formula for calculating the coverage area is expressed as follows: ; In the formula, Indicates user The reference signal received power; Indicates the service threshold, such as -110dBm; This indicates the optimal signal strength that the system can achieve, when When, receive a positive reward, when At that time, a negative penalty is imposed.

[0036] The negative penalty term includes the interference penalty, which is the penalty expressed in the multi-objective reward function formula. The term is used to suppress interference within the system. The formula for calculating the interference suppression level is expressed as follows: ; In the formula, This represents the interference power spectral density. In dense base station deployments, the weighting method is to increase the interference suppression level. In a single-base station scenario, the weight adjustment method is to reduce the weight of interference suppression level. In the multi-objective reward function formula The term represents latency. The latency term is designed with flexibility. In low-latency services such as 5G ultra-reliable low-latency communication scenarios and ordinary services such as enhanced mobile broadband scenarios, the communication latency calculation formula is expressed as follows: ; In the formula, This indicates the communication latency under low-latency services; This represents communication latency under normal business conditions. Under different communication scenarios, the weight coefficients for system throughput, communication latency, network energy efficiency, user fairness, interference suppression level, and coverage range in the multi-objective reward function are respectively... ~ Dynamic changes: In one embodiment, the dynamic weighting mechanism is shown in Table 1 below. In the highway scenario, since coverage continuity is the most critical factor, the weighting coefficients corresponding to the coverage range are... The maximum value is 0.8; in stadium event scenarios (i.e., video scenarios), throughput is prioritized, therefore the weighting coefficient corresponding to system throughput is... The maximum value is 0.6; in smart factory scenarios, the optimization of both energy efficiency and latency is prioritized, therefore the weighting coefficients corresponding to communication latency and network energy efficiency are adjusted accordingly. and The values ​​were adjusted to 0.3 and 0.1; in rural wide-area coverage scenarios (i.e., remote areas), energy efficiency and coverage are the core objectives, therefore the weighting coefficients corresponding to network energy efficiency and coverage range were adjusted accordingly. and Adjusted to 0.1 and 0.7.

[0037] Table 1 (Dynamic Weighting Mechanism Table) In this embodiment, the multi-objective optimization unit is used to handle the conflict between multiple optimization objectives using Pareto optimization, ε-constraint method, or weight adaptive mechanism, so as to maximize the multi-objective reward function and obtain the optimal cooperative beamforming action. Further, in the beamforming strategy parsing unit, the optimal cooperative beamforming action is parsed and transformed into specific, executable beamforming control parameters. These beamforming control parameters include beamforming weight parameters, power allocation parameters, antenna selection parameters, and time-frequency resource allocation parameters. Specifically, the beamforming weight vector is the complex weight applied to each antenna element on the cooperative antenna array, i.e., amplitude and phase, used to jointly determine the direction, gain, and shape of the synthesized beam; the power allocation vector is the transmit power dynamically allocated to different users or beams; the antenna selection vector is used to select an active subset of antennas in a large-scale array to reduce power consumption and complexity; and the time-frequency resource allocation parameters are used for fine-grained allocation of time-frequency resources as needed.

[0038] In this embodiment, when the system is deployed in a multi-base station or multi-access point collaborative scenario such as macro base stations or micro base stations, the collaborative control module 06 coordinates multiple base stations to perform joint beamforming and cross-base station joint optimization and interference coordination. The collaborative control module 06 includes a spatial information exchange unit, a joint optimization unit, an interference coordination unit, and a policy distribution unit. The information exchange unit is used to obtain scene situation data and CSI of each base station through a preset interface, realizing efficient sharing and synchronization of scene situation data and CSI among multiple base stations or multiple access points. Specifically, a reliable data transmission channel is established between each base station or multiple access points through a standard interface or a dedicated high-speed communication link. The system collects and forwards scene situation data such as user spatial distribution information, user trajectory prediction results, real-time environmental maps, and scene feature vectors of each base station in real time, and synchronizes the channel matrix, signal-to-noise ratio, interference level, and other CSI of each base station. The information exchange unit supports both periodic and event-triggered data exchange modes to ensure the real-time nature and effectiveness of the information.

[0039] In this embodiment, the joint optimization unit is used to determine the globally optimal or suboptimal cooperative beamforming strategy based on the shared information of each base station, namely scene situation data and CSI, to maximize the overall network performance. Specifically, a global optimization problem model is constructed, with the objective function being a weighted sum of multi-objective reward functions. This simultaneously optimizes system throughput, communication latency, network energy efficiency, user fairness, interference suppression level, and coverage. Distributed or centralized optimization algorithms (such as distributed alternating direction multiplier method, centralized convex optimization, and multi-agent cooperative learning algorithm) are used to solve this global optimization problem, thereby outputting a global cooperative beamforming strategy, including the beamforming weight matrix of each base station, power allocation scheme, and time-frequency resource allocation strategy. The weights of each objective can be adaptively adjusted according to network load, service type, and scene characteristics. The interference coordination unit is used to generate joint beamforming control parameters across base stations according to the global cooperative beamforming strategy, and to perform centralized or distributed interference coordination to suppress inter-cell interference and improve the signal-to-noise ratio of edge users. In the centralized interference coordination, the central controller uniformly calculates and issues interference coordination instructions, while in the distributed interference coordination, each base station achieves local interference coordination based on consensus algorithms using local information and neighboring cell information.

[0040] In this embodiment, the policy distribution unit distributes the coordinated joint beamforming control parameters to each base station to ensure policy consistency and execution synchronization. Specifically, the parsed joint beamforming control parameters are distributed to each base station through a standard interface. The transmission process supports parameter encoding and compression to reduce transmission overhead and improve distribution efficiency. An integrated synchronization controller ensures that each base station executes the same global coordinated beamforming policy at the same time or within the same frame period, avoiding performance fluctuations caused by timing deviations. In addition, through policy version management and rollback mechanisms, it is possible to quickly recover to the previous stable state when policy execution is abnormal. The execution scheduler schedules the execution sequence of beamforming policies of each base station to ensure network-level coordination. The performance feedback processor monitors network performance indicators (such as inter-cell interference levels) after coordinated execution and sends feedback information back to the central control module 02 for subsequent policy optimization and online learning.

[0041] In this embodiment, the cooperative antenna array module 03 receives the optimal cooperative beamforming action generated by the central control module 02, executes the parsed beamforming control parameters, and then generates the optimal radio frequency beam. Specifically, the cooperative antenna array module 03 is the physical execution center of the system, containing multiple cooperative antenna elements. These multiple antenna elements can form a Massive MIMO array or be distributed antenna elements. The cooperative antenna array module 03 receives the beamforming control parameters issued by the central control module 02, precisely adjusts the radio frequency beam, and obtains the optimal radio frequency beam, thereby achieving directional transmission and efficient reception of signals. This module mainly supports communication in the current mainstream millimeter wave (mmWave) and sub-6GHz frequency bands.

[0042] In the embodiments of this application, please refer to Figure 5 It shows a schematic diagram of a system feedback evaluation module provided in an embodiment of this application, such as Figure 5As shown, the system feedback evaluation module 04 includes a system performance monitoring unit, a reward calculation unit, and a status update unit. The system performance monitoring unit is used to monitor the system performance indicators after the cooperative antenna array module 03 performs the optimal cooperative beamforming action in real time. The system performance indicators include system throughput, communication latency, network power consumption, interference level, and user service quality satisfaction. Specifically, firstly, multi-source performance data is acquired in real time after the cooperative antenna array module 03 performs optimal cooperative beamforming. This includes CSI, system monitoring indicators, resource usage status, user experience data, and environmental sensor data. System monitoring indicators include end-to-end communication latency, bit error rate, and packet loss rate; resource usage status includes base station CPU utilization, memory usage, and RF unit power consumption; user experience data includes user feedback on service quality satisfaction, complaint rate, and service experience quality; and environmental sensor data includes real-time interference level measurements, channel change rate, and dynamic obstacle impact assessment. Secondly, the multi-source performance data is preprocessed. A data synchronizer aligns the timestamps of the multi-source performance data to ensure data timing consistency. A data sampler downsamples high-frequency data and upsamples or interpolates sparse data to unify data granularity. A noise filter uses wavelet transform or Kalman filtering algorithms to denoise the performance data, eliminating measurement errors and instantaneous fluctuations. Finally, based on key performance indicators (Key... The Performance Indicator (KPI) calculator calculates system performance metrics, including system throughput, communication latency, network energy consumption, interference level, and user service quality satisfaction. In addition, a reliability calculator calculates system communication reliability metrics such as bit error rate and connection interruption probability; an efficiency calculator calculates spectral efficiency and energy efficiency; a trend analyzer performs time-series analysis on system performance metrics to identify performance trends; a correlation analyzer analyzes the correlation between multiple variables to identify key factors affecting system performance; and an anomaly detector detects abnormal system states based on statistical methods or machine learning models.

[0043] In this embodiment, the reward calculation unit determines a multi-objective reward vector based on a multi-objective reward function according to system performance indicators. The multi-objective reward function integrates multiple conflicting system performance objectives into a single reward value through a dynamic weighting mechanism. The dynamic weights can be adaptively adjusted based on the scene type characteristics output by the scene perception module. The reward value and penalty value are calculated by a reward calculator and a penalty calculator based on the deviation between the actual system performance and the target system performance. By combining the reward value and the penalty value, the multi-objective reward vector can be calculated. Unlike the expected long-term cumulative reward, the multi-objective reward vector is the immediate reward value, composed of multiple objectives, actually observed after the system performs cooperative beamforming at each time step. The expected long-term cumulative reward can be used to evaluate the immediate effect after the current cooperative beamforming action is executed.

[0044] In this embodiment, the state update unit is used to obtain the comprehensive state vector for the next moment based on changes in the environment and system. This comprehensive state vector incorporates new scene situation data and CSI. Specifically, the state update unit receives the latest scene situation data and CSI, performs feature reduction and encoding on the latest scene situation data and CSI using a state encoder, extracts key state features, and performs data standardization and normalization to improve learning stability. A state aggregator then merges the updated scene situation data and CSI to construct the comprehensive state vector for the next moment, whose dimension and structure are consistent with the comprehensive state vector at the current moment to ensure temporal continuity. In optional embodiments, a report generator generates a system performance analysis report, including historical performance trends and optimization effect evaluations. An alarm manager triggers alarm signals and records system logs when abnormal performance or communication failures are detected. A dashboard updater updates the visual monitoring interface in real time, displaying system status and performance indicators.

[0045] In this embodiment, the experience storage module 05 is used to construct an experience tuple using the current integrated state vector, the optimal cooperative beamforming action, the multi-objective reward vector, and the next integrated state vector. The experience tuple is then stored in the experience replay pool, and the parameters of the Actor network and Critic network are periodically updated based on the experience tuple. Specifically, this is done by combining the current integrated state vector... Optimal Cooperative Beamforming Action Multi-objective reward vector and the comprehensive state vector at the next moment Constructing experience tuples The model stores experience tuples in an experience replay pool to provide data for model training and learning. Batch learning is performed using data from each experience tuple to complete offline training of the MORL agent, or continuous learning to complete online fine-tuning of the MORL agent. Based on the data stored in the experience replay pool, the parameters of the MORL agent's Actor and Critic networks are periodically updated by randomly sampling small batches of data. This allows the model to continuously learn and improve from real-world interactions, continuously optimizing its decision-making capabilities and supporting transfer learning to accelerate adaptation to new scenarios.

[0046] In this embodiment, the system supports iterative looping, generating the comprehensive state vector for the next time step. When, a state transition is performed, and As the new comprehensive state vector at the current moment, each module in the system repeatedly executes the corresponding operation. Through a continuous "perception-decision-execution-evaluation-learning" cycle, the system achieves dynamic adaptation to the wireless communication environment and continuous performance optimization until the system performance converges or meets the preset operating requirements. Through this powerful online learning mechanism, the system achieves rapid adaptation to long-term changes and evolution of the environment.

[0047] The following description uses a specific embodiment to illustrate the above-mentioned dynamic beamforming control system based on multi-objective reinforcement learning. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0048] In a specific embodiment, in a typical urban central business district scenario, the area has a large number of high-rise buildings, which can easily lead to severe signal obstruction and multipath effects. The area is densely populated with users and their devices, including a large number of mobile users walking or using public transportation, as well as fixed users in office buildings, creating dynamic user "tidal" hotspots. Inter-cell interference is severe, network capacity requirements are huge, and users have high demands for network connection stability. A dynamic beamforming control system is deployed in this area, with multiple macro and micro base stations densely deployed to form a multi-layered coverage network. All base stations are equipped with Massive MIMO antenna arrays to complete the deployment of cooperative antenna array modules. High-definition optical cameras and millimeter-wave radars are deployed at key intersections, building facades, and base station towers to complete the deployment of scene perception modules. The central control module and cooperative control module functions are integrated and deployed on a regional mobile edge computing server, connected to all base stations via a high-speed backhaul network, achieving low-latency centralized control. The scene perception module operates in real time. Cameras utilize target detection algorithms (such as YOLOv5) to identify pedestrian hotspots and vehicle flow directions on the street. Millimeter-wave radar precisely tracks the trajectories of individual high-speed moving vehicles. Based on several seconds of continuous observation, the Long Short-Term Memory (LSTM) network model predicts that bus N will enter the intersection area covered by base stations A and B in 5 seconds, and may be briefly obscured by a large building. The system automatically classifies this area as a "dense urban-high-speed moving mixed scene," thus completing scene perception and prediction. The central control module's state construction unit integrates scene situational data and CSI (Content Situation Indicator). Scene situational data includes the real-time location, speed, predicted trajectory, building occlusion map, and scene classification labels for all users and vehicles. CSI includes CSI reported by all base stations and CSI between users within their respective coverage areas, thus completing comprehensive state construction. The MORL agent receives the comprehensive state and, based on the scene label of "dense urban-high-speed moving mixed scene," dynamically adjusts the reward function weights to: high throughput (…). =0.4), high fairness ( =0.3), strong interference suppression ( =0.2), medium energy efficiency ( =0.1). For buses about to be blocked, the MORL agent generates a forward-looking cooperative multipoint transmission decision to ensure uninterrupted connectivity, thus completing the MORL decision. Further, the beamforming strategy parsing unit parses the MORL agent's actions into specific instructions, instructing base stations 1 and 2 to jointly serve the users on the bus. Base station 1's beam tracks the vehicle until it enters the leading edge of the blocking area, while base station 2, based on the predicted trajectory, generates a high-gain "waiting beam" in advance at the position where the bus will exit the blocking area. For other densely populated pedestrian areas, the system employs Multi-User Multiple-Input Multiple-Output (MU-MIMO) technology to generate multiple spatially orthogonal beams, maximizing throughput and user fairness. Simultaneously, the system continuously monitors the total network throughput, the instantaneous rate of bus users, the interference level between base stations, and the total power consumption, calculating multi-objective rewards accordingly and storing the experience tuples in the experience replay pool for continuous online optimization of the MORL model. By comparing the base station based on dynamic beamforming control using multi-objective reinforcement learning in this embodiment with a base station using traditional maximum ratio combining beamforming, the technical solution provided in this embodiment has the following significant advantages: In terms of edge user experience, the average download speed of bus users located at the cell edge or on the move is increased by about 55%, and the drop call rate is reduced by nearly 80%. In terms of network capacity, the total system throughput of the entire cooperative area is increased by about 40%. In terms of interference level, the average inter-cell interference power is reduced by 6-8 dB. In summary, the technical solution provided in this embodiment has significant advantages in effectively solving the complex communication challenges of high density, high interference, and high mobility in urban areas.

[0049] In summary, the dynamic beamforming control system based on multi-objective reinforcement learning provided in this application collects and processes multimodal raw environmental data through a scene perception module, predicts user movement and environmental changes, and generates scene situational data. This allows for deep perception of the physical environment, automatic identification and adaptation to different communication scenarios, significantly improving performance in complex dynamic environments. It also achieves predictive beamforming, adjusting beams in advance to avoid interruptions, demonstrating strong foresight and prediction capabilities, and greatly enhancing connection robustness. In the central control module, a comprehensive state vector for the current moment is constructed based on scene situational data and CSI, and the optimal cooperative beamforming action is generated based on the MORL algorithm. This enables simultaneous dynamic optimization of multiple key performance indicators, determining Pareto optimal or satisfactory solutions based on actual service requirements and network status, overcoming the limitations of single-objective optimization, and further resolving the optimal cooperative beamforming action into beamforming control parameters. The cooperative control module enables intelligent coordination of multiple antennas or base stations, effectively suppressing interference and improving overall network performance and resource utilization. The linear array module performs optimal cooperative beamforming to generate the optimal radio frequency beam. In the system feedback evaluation module, the system performance after the cooperative antenna array module performs optimal cooperative beamforming is monitored, the multi-target reward vector is calculated and the system state is updated to obtain the comprehensive state vector at the next moment. In the experience storage module, an experience tuple is constructed using the comprehensive state vector at the current moment, the optimal cooperative beamforming action, the multi-target reward vector, and the comprehensive state vector at the next moment. The experience tuple is stored in the experience replay pool, and the parameters of the Actor network and Critic network are updated periodically based on the experience tuple to achieve continuous learning and self-optimization of the system. This process does not require a large amount of pre-labeled data, allowing the system performance to continuously improve over time. After offline training, online decision-making is based on neural network forward computation, which has low complexity and fast response speed, meeting the real-time communication requirements. In addition, the technical solution provided in this application embodiment can intelligently reduce the transmission power or put some modules into sleep mode while ensuring performance, thereby significantly reducing system energy consumption.

[0050] The above is a description of the system embodiments of this application. Based on the foregoing embodiments, the method embodiments of this application are described below.

[0051] Please refer to Figure 6 The diagram illustrates a flowchart of a dynamic beamforming control method based on multi-objective reinforcement learning, provided in an embodiment of this application. This method is applied to applications such as... Figure 1 The diagram illustrates a dynamic beamforming control system based on multi-objective reinforcement learning. For details not disclosed in the method embodiments, please refer to the system embodiments. The system includes a scene perception module, a central control module, a cooperative antenna array module, a system feedback evaluation module, and an experience storage module. These modules are interconnected. Figure 6 As shown, the method includes the following steps S610 to S650.

[0052] Step S610: Collect and process multimodal raw environmental data to generate scene situational data. The multimodal raw environmental data includes physical environment data and user equipment status information.

[0053] In this embodiment, multimodal raw environmental data is acquired based on multi-sensor fusion technology; the multimodal raw environmental data is processed by timestamp alignment, coordinate unification, and noise reduction filtering to generate multimodal environmental data; the multimodal environmental data is processed by a wireless positioning algorithm to generate spatial distribution information of the user; based on the spatial distribution information and multimodal environmental data, a machine learning model is used to perform time-series prediction and spatial feature extraction to obtain user trajectory prediction results, real-time environmental map, and potential channel event prediction results; based on the user trajectory prediction results, real-time environmental map, and potential channel event prediction results, a deep learning model is used to classify the current scene and generate scene feature vectors.

[0054] In this embodiment, image and environmental map data are processed by a convolutional neural network to identify static and dynamic obstacles, and a real-time environmental map is constructed and updated. User movement trajectory data is processed by a long short-term memory network, a gated recurrent unit, or a Transformer neural network to predict the future movement trajectory of the user equipment, thus obtaining the user trajectory prediction result. A Markov chain model is used to predict the system state, and potential channel blocking events are predicted based on the user trajectory prediction result and the real-time environmental map, thus obtaining the potential channel event prediction result.

[0055] Step S620: Construct a comprehensive state vector for the current moment based on scene situation data and channel state information, and generate the optimal cooperative beamforming action based on a multi-objective reinforcement learning algorithm, and parse the optimal cooperative beamforming action into beamforming control parameters.

[0056] In this embodiment, the scene situation data and CSI are fused to generate a comprehensive state vector at the current moment. The scene situation data includes the spatial distribution information of users, the user trajectory prediction results, the real-time environment map, and the scene feature vector. The CSI includes the channel matrix, signal-to-noise ratio, and interference level. Based on the comprehensive state vector at the current moment, intelligent decision-making is performed to generate the optimal cooperative beamforming action. The optimal cooperative beamforming action is parsed into beamforming control parameters, which include beamforming weight parameters, power allocation parameters, antenna selection parameters, and time-frequency resource allocation parameters.

[0057] In this embodiment, a cooperative beamforming action is generated based on the comprehensive state vector at the current moment using an Actor network. A Critic network is used to evaluate the expected long-term cumulative reward of the comprehensive state vector at the current moment and the cooperative beamforming action. The expected long-term cumulative reward is calculated based on a multi-objective reward function. Pareto optimization, ε-constraints, or a weighted adaptive mechanism are used to handle conflicts between multiple optimization objectives, maximizing the multi-objective reward function to obtain the optimal cooperative beamforming action. The multi-objective reward function is constructed based on system throughput, communication latency, network energy consumption, user fairness, interference suppression level, and coverage.

[0058] In this embodiment, multiple base stations are coordinated to perform joint beamforming and cross-base station joint optimization and interference coordination. Specifically, scenario situation data and CSI of each base station are obtained through a preset interface; a global cooperative beamforming strategy is determined based on the scenario situation data and CSI of each base station to maximize the overall network performance; according to the global cooperative beamforming strategy, cross-base station joint beamforming control parameters are generated, and centralized or distributed interference coordination is performed to suppress inter-cell interference; the coordinated joint beamforming control parameters are distributed to each base station.

[0059] Step S630: Perform optimal cooperative beamforming to generate the optimal radio frequency beam.

[0060] Step S640: Monitor the system performance after the cooperative antenna array module performs the optimal cooperative beamforming action, calculate the multi-target reward vector and update the system state, and obtain the comprehensive state vector at the next moment.

[0061] In this embodiment, the system performance indicators after the cooperative antenna array module performs optimal cooperative beamforming are monitored in real time. The system performance indicators include system throughput, communication latency, network power consumption, interference level, and user service quality satisfaction. Based on the system performance indicators, a multi-objective reward vector is calculated based on a multi-objective reward function. According to changes in the environment and system, the comprehensive state vector at the next moment is obtained, and the comprehensive state vector at the next moment incorporates new scene situation data and CSI.

[0062] Step S650: Construct an experience tuple using the current integrated state vector, the optimal cooperative beamforming action, the multi-objective reward vector, and the next integrated state vector. Store the experience tuple in the experience replay pool and periodically update the parameters of the policy network and the value network based on the experience tuple.

[0063] It should be noted that, in the embodiments of this application, if the above-mentioned dynamic beamforming control method based on multi-objective reinforcement learning is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0064] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of any of the dynamic beamforming control methods based on multi-objective reinforcement learning described in the above embodiments. Correspondingly, embodiments of this application also provide a computer program product, which, when executed by a processor of an electronic device, is used to implement the steps of any of the dynamic beamforming control methods based on multi-objective reinforcement learning described in the above embodiments.

[0065] Based on the same technical concept, this application provides an electronic device for implementing a dynamic beamforming control method based on multi-objective reinforcement learning as described in the above method embodiments. Figure 7 This is a hardware entity diagram of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the electronic device 700 includes a memory 710 and a processor 720. The memory 710 stores a computer program that can run on the processor 720. When the processor 720 executes the program, it implements the steps in any of the embodiments of this application of a dynamic beamforming control method based on multi-objective reinforcement learning.

[0066] The memory 710 is configured to store instructions and applications executable by the processor 720, and can also cache data to be processed or already processed by the processor 720 and various modules in the electronic device (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0067] The processor 720 executes a program to implement the steps of a dynamic beamforming control method based on multi-objective reinforcement learning, as described above. The processor 720 typically controls the overall operation of the electronic device 700.

[0068] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0069] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0070] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0071] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0072] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0074] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0075] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0076] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause the device automatic test line to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0077] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0078] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0079] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A dynamic beamforming control system based on multi-objective reinforcement learning, characterized in that, The system includes: The scene perception module is used to collect and process multimodal raw environmental data and generate scene situation data. The multimodal raw environmental data includes physical environment data and user equipment status information. The central control module is used to construct the comprehensive state vector at the current moment based on the scene situation data and channel state information, and generate the optimal cooperative beamforming action based on the multi-objective reinforcement learning algorithm, and parse the optimal cooperative beamforming action into beamforming control parameters. The cooperative antenna array module is used to perform optimal cooperative beamforming to generate the optimal radio frequency beam. The system feedback evaluation module is used to monitor the system performance after the cooperative antenna array module performs the optimal cooperative beamforming action, calculate the multi-target reward vector and update the system state, and obtain the comprehensive state vector at the next moment. The experience storage module is used to construct experience tuples by using the current integrated state vector, the optimal cooperative beamforming action, the multi-objective reward vector, and the next integrated state vector. The experience tuples are stored in the experience replay pool, and the parameters of the policy network and value network are updated periodically based on the experience tuples.

2. The system according to claim 1, characterized in that, The system also includes a collaborative control module, which coordinates multiple base stations to perform joint beamforming and performs cross-base station joint optimization and interference coordination.

3. The system according to claim 1, characterized in that, The scene perception module includes a data acquisition unit, a data preprocessing unit, a user distribution perception unit, a prediction and inference unit, and a scene classification and feature extraction unit, wherein: The data acquisition unit is used to acquire multimodal raw environmental data based on multi-sensor fusion technology; The data preprocessing unit is used to perform timestamp alignment, coordinate unification, and noise reduction filtering on the original multimodal environmental data to generate multimodal environmental data. The user distribution sensing unit is used to process multimodal environmental data through a wireless positioning algorithm to generate spatial distribution information of users; The prediction and inference unit is used to perform time-series prediction and spatial feature extraction based on spatial distribution information and multimodal environment data through machine learning models, so as to obtain user trajectory prediction results, real-time environment maps and potential channel event prediction results. The scene classification and feature extraction unit is used to classify the current scene and generate scene feature vectors based on user trajectory prediction results, real-time environment map and potential channel event prediction results through a deep learning model.

4. The system according to claim 3, characterized in that, The prediction and reasoning unit includes a spatial feature extraction subunit, a trajectory prediction subunit, and a state prediction and anomaly detection subunit, wherein: The spatial feature extraction subunit is used to process image and environmental map data based on convolutional neural network, identify static and dynamic obstacles, and construct and update real-time environmental map. The trajectory prediction subunit is used to process user movement trajectory data based on long short-term memory network, gated recurrent unit or Transformer neural network, predict the future movement trajectory of user equipment, and obtain user trajectory prediction result. The state prediction and anomaly detection subunit is used to predict the system state using a Markov chain model, and to predict potential channel blocking events based on user trajectory prediction results and real-time environment maps, thereby obtaining potential channel event prediction results.

5. The system according to claim 2, characterized in that, The collaborative control module includes a space information exchange unit, a joint optimization unit, an interference coordination unit, and a strategy distribution unit, wherein: The information exchange unit is used to obtain scene situation data and channel status information of each base station through a preset interface; The joint optimization unit is used to determine a global cooperative beamforming strategy based on the scenario situation data and channel state information of each base station, so as to maximize the overall network performance. The interference coordination unit is used to generate joint beamforming control parameters across base stations according to the global cooperative beamforming strategy, and to perform centralized or distributed interference coordination to suppress inter-cell interference. The strategy distribution unit is used to distribute the coordinated joint beamforming control parameters to each base station.

6. The system according to claim 1, characterized in that, The central control module includes a state construction unit, a multi-objective reinforcement learning agent unit, and a beamforming policy parsing unit, wherein: The state construction unit is used to fuse scene situation data and channel state information to generate a comprehensive state vector at the current moment. The scene situation data includes the spatial distribution information of users, user trajectory prediction results, real-time environment map and scene feature vector. The channel state information includes channel matrix, signal-to-noise ratio and interference level. The multi-objective reinforcement learning agent unit is used to make intelligent decisions based on the comprehensive state vector at the current moment and generate the optimal cooperative beamforming action. The beamforming strategy parsing unit is used to parse the optimal cooperative beamforming action into beamforming control parameters, which include beamforming weight parameters, power allocation parameters, antenna selection parameters, and time-frequency resource allocation parameters.

7. The system according to claim 6, characterized in that, The multi-objective reinforcement learning agent unit includes an action generation subunit, a cumulative reward calculation subunit, and a multi-objective optimization unit, wherein: The action generation subunit is used to generate cooperative beamforming actions based on the comprehensive state vector at the current moment through a policy network. The cumulative reward calculation subunit is used to evaluate the expected long-term cumulative reward of the current integrated state vector and the cooperative beamforming action through the value network. The expected long-term cumulative reward is calculated based on the multi-objective reward function. The multi-objective optimization unit is used to handle the conflict between multiple optimization objectives by employing Pareto optimization, ε-constraint method or weight adaptive mechanism, so as to maximize the multi-objective reward function and obtain the optimal cooperative beamforming action. The multi-objective reward function is constructed by considering system throughput, communication latency, network energy consumption, user fairness, interference suppression level and coverage.

8. The system according to claim 1, characterized in that, The system feedback evaluation module includes a system performance monitoring unit, a reward calculation unit, and a status update unit, wherein: The system performance monitoring unit is used to monitor the system performance indicators in real time after the cooperative antenna array module performs the optimal cooperative beamforming action. The system performance indicators include system throughput, communication latency, network power consumption, interference level, and user service quality satisfaction. The reward calculation unit is used to calculate a multi-objective reward vector based on a multi-objective reward function according to system performance indicators; The state update unit is used to obtain the comprehensive state vector at the next moment according to changes in the environment and system. The comprehensive state vector at the next moment integrates new scenario situation data and channel state information.

9. A dynamic beamforming control method based on multi-objective reinforcement learning, characterized in that, A method for a dynamic beamforming control system based on multi-objective reinforcement learning is provided. The system includes a scene perception module, a central control module, a cooperative antenna array module, a system feedback evaluation module, and an experience storage module. Collect and process multimodal raw environmental data to generate scene situational data. The multimodal raw environmental data includes physical environment data and user equipment status information. The comprehensive state vector at the current moment is constructed based on the scene situation data and channel state information, and the optimal cooperative beamforming action is generated based on the multi-objective reinforcement learning algorithm. The optimal cooperative beamforming action is then parsed into beamforming control parameters. Perform optimal cooperative beamforming actions to generate the optimal radio frequency beam; Monitor the system performance after the cooperative antenna array module performs the optimal cooperative beamforming action, calculate the multi-target reward vector and update the system state, and obtain the comprehensive state vector at the next moment. Experience tuples are constructed using the current state vector, the optimal cooperative beamforming action, the multi-objective reward vector, and the next state vector. These experience tuples are then stored in the experience replay pool, and the parameters of the policy network and value network are periodically updated based on the experience tuples.

10. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method of claim 9.

Citation Information

Cited By

  • 6G terahertz communication optimization method based on machine learning

    CN121567229A