Multi-uav beam adjustment method and device, and electronic device

Abstract

The application discloses a multi-unmanned aerial vehicle beam adjustment method and device and electronic equipment. The method comprises the following steps: obtaining a pre-allocated beam scheme of an unmanned aerial vehicle and position information in a flight process; determining a state vector of the unmanned aerial vehicle according to the position information, and processing the state vector through a target encoder in a strategy network to obtain an association probability matrix and a beam parameter vector, wherein the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, the association probability matrix is used to reflect a service relationship between the unmanned aerial vehicle and a base station beam, and the beam parameter vector is used to reflect a parameter configuration of the base station beam; and adjusting the pre-allocated beam scheme according to the association probability matrix and the beam parameter vector. The application solves the technical problems of discontinuous communication coverage, unstable signal quality and low resource scheduling efficiency of a static beam scheme in the related art under high-speed movement and complex scenes of multi-unmanned aerial vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication, in particular to a multi-unmanned aerial vehicle (UAV) beam adjustment method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology and the rise of low-altitude economy, aerial UAV networks gradually become an important communication infrastructure connecting the ground and airspace, supporting a variety of aerial application scenarios such as logistics distribution, intelligent inspection, and real-time live broadcast. However, due to the dynamic nature of unmanned aerial vehicles and the complex and variable airspace environment, higher requirements are placed on the reliability, coverage range, and data transmission rate of unmanned aerial vehicle network communication.

[0003] On the one hand, the flight trajectory of unmanned aerial vehicles is variable, unlike the linear coverage optimization of high-speed railways or highways. It is difficult for operators to predict the precise flight path of unmanned aerial vehicles, making it difficult for ground base stations to track and optimize beams to meet the high-speed mobile requirements of unmanned aerial vehicles.

[0004] On the other hand, unmanned aerial vehicles usually have predictability of preset routes and trajectories. However, the current aerial network fails to fully utilize this feature, especially in multi-unmanned aerial vehicle cooperative scenarios. A relatively rigid beam configuration method, such as fixed base station beams or static cell strategies, is still used, hindering the improvement of signal quality and resource scheduling efficiency.

[0005] To address the above problems, no effective solutions have been proposed so far. SUMMARY

[0006] Embodiments of the present application provide a multi-unmanned aerial vehicle beam adjustment method, device, and electronic equipment to at least solve the technical problems of discontinuous communication coverage, unstable signal quality, and low resource scheduling efficiency in the high-speed movement and complex scenarios of multi-unmanned aerial vehicles due to the static beam scheme in the related art.

[0007] According to an aspect of the embodiments of the present application, a multi-unmanned aerial vehicle (UAV) beam adjustment method is provided. The method comprises: obtaining a pre-allocated beam scheme of the UAV and position information of the UAV in a flight process, wherein the pre-allocated beam scheme comprises an initial beam allocation scheme corresponding to a plurality of UAVs; determining a state vector of the UAV according to the position information, and processing the state vector through a target encoder in a strategy network to obtain a correlation probability matrix and a beam parameter vector, wherein the strategy network is used to adjust a base station beam service decision for the UAV, the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, and outputs the correlation probability matrix and the beam parameter vector through different branches, the correlation probability matrix is used to reflect a service relationship between the UAV and a base station beam, and the beam parameter vector is used to reflect a parameter configuration of the base station beam; and adjusting the pre-allocated beam scheme according to the correlation probability matrix and the beam parameter vector.

[0008] Optionally, the determining of the state vector of the UAV according to the position information comprises: determining a first vector, a second vector and a third vector respectively according to the position information, wherein the first vector is used to reflect angle coordinate information and speed change information of the UAV relative to the base station, the second vector is used to reflect historical beam parameter information of the UAV, and the third vector is used to reflect an angle difference between the UAV and the base station beam; and fusing the first vector, the second vector and the third vector to obtain the state vector.

[0009] Optionally, the processing of the state vector through the target encoder in the strategy network to obtain the correlation probability matrix and the beam parameter vector comprises: determining an input feature corresponding to the state vector, and determining a first feature matrix of the strategy network according to the input feature; processing the first feature matrix through the target encoder in the strategy network to output a second feature matrix; performing normalization processing on the second feature matrix through a first branch in the strategy network to obtain the correlation probability matrix; and performing a pooling operation on the second feature matrix through a second branch in the strategy network to obtain the beam parameter vector.

[0010] Optionally, the strategy network is obtained by training in the following manner: obtaining a simulation state vector of the UAV and a simulation action vector, wherein the simulation action vector is a simulation correlation probability matrix and a simulation beam parameter vector generated by using a pre-allocated algorithm; processing the simulation state vector through an initial strategy network to obtain a predicted action vector, wherein the predicted action vector comprises a predicted correlation probability matrix and a predicted beam parameter vector output by the initial strategy network; determining a first loss function required for training the initial strategy network, and determining a first loss between the simulation action vector and the predicted action vector through the first loss function; determining optimal parameters of the initial strategy network by using a gradient descent method according to the first loss to obtain the strategy network.

[0011] Optionally, the method further comprises: determining an action vector of the UAV according to the association probability matrix and the beam parameter vector, wherein the action vector is used to reflect a beam service decision taken by the UAV; splicing the action vector and the state vector to obtain a spliced vector, and processing the spliced vector through a value network to obtain an evaluation value, wherein the value network is used to evaluate the value of the beam service decision taken by the UAV; and optimizing the policy network and the value network according to the evaluation value and a reward function, wherein the reward function is used to quantify the signal quality, effective coverage rate and beam stability of the UAV after adopting a target beam allocation scheme, and the target beam allocation scheme is a pre-allocated beam scheme adjusted according to the association probability matrix and the beam parameter vector.

[0012] Optionally, the reward function is determined by: determining a service probability of a target beam to a target UAV, and determining a signal reception power of the target UAV after accessing the target beam, and determining a first reward function according to the service probability and the signal reception power, wherein the target UAV is any one of the plurality of UAVs, the target beam is an optimal beam corresponding to the target UAV in the target beam scheme, and the first reward function is used to quantify the signal quality received by the target UAV from the target beam; determining a second reward function according to the signal reception power and a preset probability, wherein the second reward function is used to balance the signal strength and effective coverage degree of the target beam to the target UAV; determining a third reward function according to a beam configuration in the beam parameter vector and a historical beam configuration, wherein the third reward function is used to suppress the change of the beam parameter; and performing weighted processing on the first reward function, the second reward function and the third reward function to obtain the reward function.

[0013] Optionally, the optimization of the policy network and the value network according to the evaluation value and the reward function comprises: determining a target evaluation value according to the reward function, wherein the target evaluation value is used to represent an expected evaluation value under the action vector; determining a second loss function required for optimizing the value network, and determining a second loss between the evaluation value and the target evaluation value through the second loss function; optimizing the value network according to the second loss and a preset soft update rate, wherein the preset soft update rate is used to smooth the update process of the value network and the policy network; determining a third loss function required for optimizing the policy network, and determining a third loss under the action vector through the third loss function; and optimizing the policy network according to the third loss and the preset soft update rate.

[0014] According to another aspect of the embodiments of the present application, a multi-UAV beam adjustment apparatus is also provided, comprising: an acquisition module configured to acquire a pre-allocated beam scheme of the UAVs and position information of the UAVs in a flight process, wherein the pre-allocated beam scheme comprises an initial beam allocation scheme corresponding to the plurality of UAVs; a processing module configured to determine a state vector of the UAVs according to the position information, and process the state vector through a target encoder in a strategy network to obtain an association probability matrix and a beam parameter vector, wherein the strategy network is used to adjust a service decision of a base station beam to the UAVs, the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, and outputs the association probability matrix and the beam parameter vector through different branches, the association probability matrix is used to reflect a service relationship between the UAVs and the base station beams, and the beam parameter vector is used to reflect a parameter configuration of the base station beams; and an adjustment module configured to adjust the pre-allocated beam scheme according to the association probability matrix and the beam parameter vector.

[0015] According to still another aspect of the embodiments of the present application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory is configured to store program instructions; and the processor is connected with the memory and is configured to execute the above-mentioned multi-UAV beam adjustment method.

[0016] According to still another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, comprising a stored computer program, wherein a device in which the non-volatile storage medium is located executes the above-mentioned multi-UAV beam adjustment method by running the computer program.

[0017] According to still another aspect of the embodiments of the present application, a computer program product is also provided, comprising computer instructions, which, when executed by a processor, implement the above-mentioned multi-UAV beam adjustment method.

[0018] In the embodiment of the present application, the pre-allocated beam scheme of the unmanned aerial vehicle and the position information of the unmanned aerial vehicle in the flight process are obtained, wherein the pre-allocated beam scheme includes an initial beam allocation scheme corresponding to a plurality of unmanned aerial vehicles; the state vector of the unmanned aerial vehicle is determined according to the position information, and the state vector is processed by a target encoder in a strategy network to obtain an association probability matrix and a beam parameter vector, wherein the strategy network is used to adjust the service decision of the base station beam to the unmanned aerial vehicle, the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, and outputs the association probability matrix and the beam parameter vector through different branches, the association probability matrix is used to reflect the service relationship between the unmanned aerial vehicle and the base station beam, and the beam parameter vector is used to reflect the parameter configuration of the base station beam; the pre-allocated beam scheme is adjusted according to the association probability matrix and the beam parameter vector, so as to achieve the purpose of intelligently optimizing the service decision of the base station beam to the plurality of unmanned aerial vehicles, thereby realizing the technical effects of improving the communication quality and resource utilization efficiency, enhancing the beam tracking stability and response speed, and further solving the technical problems of communication coverage discontinuity, unstable signal quality and low resource scheduling efficiency of the static beam scheme in the related art under the high-speed movement and complex scene of the plurality of unmanned aerial vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0020] Figure 1 FIG. 1 is a hardware structure diagram of a computer terminal for implementing a beam adjustment method for a plurality of unmanned aerial vehicles according to an embodiment of the present application;

[0021] Figure 2 FIG. 2 is a flowchart of a beam adjustment method for a plurality of unmanned aerial vehicles according to an embodiment of the present application;

[0022] Figure 3 FIG. 3 is a structure diagram of an Actor-Critic network according to an embodiment of the present application;

[0023] Figure 4 FIG. 4 is a structure diagram of a beam adjustment apparatus for a plurality of unmanned aerial vehicles according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable personnel in the technical field to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, some nouns or terms appearing in the process of explaining the embodiments of the present application are applicable to the following explanations:

[0027] SSB (Synchronization Signal Block, Synchronization Signal Block): a signal structure in 5G communication system, used for terminal device to perform initial access and cell measurement. SSB contains primary synchronization signal (PSS, Primary Synchronization Signal), secondary synchronization signal (SSS, Secondary Synchronization Signal) and physical broadcast channel (PBCH, Physical Broadcast Channel), these signals and channels are used to help terminal synchronization and receive cell basic broadcast information.

[0028] MDP (Markov Decision Process, Markov Decision Process): a mathematical model for describing how to make decisions based on the current state to achieve the optimal result in an uncertain environment. MDP is a basic concept in reinforcement learning, which defines the state space, action space, transition probability and reward function, and provides a framework for intelligent agents to learn the optimal strategy in the environment.

[0029] RSRP (Reference Signal Received Power): is an important indicator for measuring network signal quality in 4G and 5G communication systems. In a wireless network, the higher the RSRP value, the stronger the power of the reference signal received by the terminal, which usually means better communication quality.

[0030] DDPG (Deep Deterministic Policy Gradient): is an algorithm used in reinforcement learning to handle continuous action space. Compared with traditional policy-based reinforcement learning methods, DDPG combines the Actor-Critic architecture, using two deep neural networks to learn policy and value function respectively, and is particularly suitable for solving continuous control problems in complex environments.

[0031] MQTT (Message Queuing Telemetry Transport): a lightweight communication protocol based on the publish / subscribe model, widely used in Internet of Things (IoT) and remote data transmission scenarios.

[0032] In order to solve the problem of poor transmission efficiency of unmanned aerial vehicle network communication in the related art, the embodiment of the present application provides a multi-unmanned aerial vehicle beam adjustment method, which can be run in Figure 1 The computer terminal is described below.

[0033] The multi-unmanned aerial vehicle beam adjustment method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal, or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing a multi-unmanned aerial vehicle beam adjustment method is shown. As Figure 1 shown, the computer terminal 10 can include one or more (shown in the figure as 102a, 102b, …, 102n) processors (the processor can include but not limited to a microprocessor MCU or a programmable logic device FPGA processing device), a memory 104 for storing data, and a transmission module 106 for communication function through wired and / or wireless network connection. In addition, it can also include a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a BUS bus. Those skilled in the art can understand, Figure 1 The structure shown is only for illustration, and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can also include more Figure 1more or less components than those shown, or in configurations with different configurations of components than those shown. Figure 1

[0034] It should be noted that the one or more processors and / or other data processing circuitry described above can be generally referred to herein as "data processing circuitry". The data processing circuitry can be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuitry can be a single standalone processing module, or incorporated in whole or in part within any of the other elements of the computer terminal 10. As referred to in embodiments of the present application, the data processing circuitry functions as a processor to control, for example, the selection of the variable resistance terminal path in connection with the interface.

[0035] The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the multi- UAV beam adjustment method in embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, i.e. implements the multi- UAV beam adjustment method described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory disposed remotely with respect to the processor, which can be connected to the computer terminal 10 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission module 106 is configured to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC) that can be connected to other network devices through a base station so as to be able to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module configured to communicate with the Internet in a wireless manner.

[0037] The display can be, for example, a touch screen type liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10.

[0038] It should be noted that in some alternative embodiments, the computer terminal shown above Figure 1 It should be noted that the computer terminal shown above can include hardware elements (including circuitry), software elements (including computer code stored on a computer readable medium), or a combination of both hardware and software elements. It should be noted that,​Figure 1 is merely one instance of a particular, specific example and is intended to show the types of components that can be present in the computer terminal described above.

[0039] Under the above operating environment, an embodiment of the present application provides a multi-unmanned aerial vehicle beam adjustment method, it should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0040] Figure 2 is a flowchart of a multi-unmanned aerial vehicle beam adjustment method according to an embodiment of the present application, as shown in Figure 2 the method comprises the following steps:

[0041] Step S202, obtaining a pre-allocated beam scheme of the unmanned aerial vehicle and position information of the unmanned aerial vehicle in the flight process, wherein the pre-allocated beam scheme comprises an initial beam allocation scheme corresponding to a plurality of unmanned aerial vehicles.

[0042] Step S204, determining a state vector of the unmanned aerial vehicle according to the position information, and processing the state vector through a target encoder in a policy network to obtain an association probability matrix and a beam parameter vector, wherein the policy network is used to adjust the service decision of the base station beam to the unmanned aerial vehicle, the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, and outputs the association probability matrix and the beam parameter vector through different branches, the association probability matrix is used to reflect the service relationship between the unmanned aerial vehicle and the base station beam, and the beam parameter vector is used to reflect the parameter configuration of the base station beam.

[0043] Step S206, adjusting the pre-allocated beam scheme according to the association probability matrix and the beam parameter vector.

[0044] Through the above steps S202 to S206, the purpose of intelligently optimizing the service decision of the base station beam to the multi-unmanned aerial vehicle is achieved, thereby realizing the technical effects of improving the communication quality and resource utilization efficiency, enhancing the beam tracking stability and response speed, and further solving the technical problems of communication coverage discontinuity, unstable signal quality and low resource scheduling efficiency of the static beam scheme in the related art under the high-speed movement and complex scene of the multi-unmanned aerial vehicle. The following will be described in detail.

[0045] In an embodiment of the present application, taking a single base station that can customize 7 SSB beams while serving N unmanned aerial vehicles as an example, through an agent (including a policy network (Actor) and a value network (Critic)), the azimuth angle, the downtilt angle, the beam width and other parameters of the beam are dynamically adjusted according to the real-time position and service quality of the unmanned aerial vehicle, and the optimal coverage scheme of the multi-unmanned aerial vehicle is realized.

[0046] In the above step S202, the following two key information are mainly collected:

[0047] First, the pre-allocated beam scheme of the UAV. For example, based on the initial flight path information of the UAV, the trajectory is predicted, and the initial beam allocation scheme corresponding to multiple UAVs is generated, that is, each UAV is assigned the base station beam that is most likely to provide the best service, and the key parameters such as the azimuth angle, the downtilt angle and the beam width of the beam are preliminarily configured.

[0048] Second, the position information of the UAV during flight. For example, through the MQTT protocol, the current position information and the service quality indicators such as signal strength (RSRP) of the UAV are uploaded to the ground control center or the base station in real time. Among them, the lightweight characteristics and the publish / subscribe mode of the MQTT protocol ensure the efficiency and reliability of data transmission, and even under limited bandwidth and unstable network conditions, the information can be updated in time.

[0049] In the above step S204, the state vector of each UAV is constructed according to the real-time position information of each UAV, and is sent to the policy network (Actor) for processing. Among them, the Actor network innovatively introduces a shared encoder module, and uses a Transformer encoder (i.e., a target encoder) as the core structure of the shared encoder module. This structure can effectively mine the mutual spatial distribution patterns between UAVs and the high-order interaction relationship between the beams through the multi-head self-attention mechanism for global modeling of each state in the input sequence, and outputs two key vectors, namely the association probability matrix and the beam parameter vector, through different branches.

[0050] It should be noted that in the embodiments of the present application, the SSB beam tracking problem of the base station to the UAV can be modeled as a Markov decision process. By defining a reasonable state space, action space and reward function, the beam tracking problem can be converted into a form suitable for solving by a reinforcement learning algorithm, providing a learning framework for adaptive optimization of beam parameters.

[0051] I. State space.

[0052] Optionally, the state vector of the UAV is determined according to the position information, including: determining a first vector, a second vector and a third vector according to the position information, respectively, wherein the first vector is used to reflect the angle coordinate information and the speed change information of the UAV relative to the base station, the second vector is used to reflect the historical beam parameter information of the UAV, and the third vector is used to reflect the angle difference between the UAV and the base station beam; and the first vector, the second vector and the third vector are fused to obtain the state vector. The specific expression is as follows:

[0053]

[0054] wherein, denotes the state vector at time t, denotes the first vector, denotes the second vector, denotes the third vector.

[0055] For the first vector:

[0056]

[0057] wherein, denotes the UAV at time t, i.e.:

[0058]

[0059] wherein, denotes the angle coordinate information of the UAV relative to the base station, denotes the angle coordinate information of the UAV relative to the base station, denotes the velocity change information of the UAV relative to the base station. Specifically, in order to enhance the accuracy of the base station beam direction selection, the application adopts a modeling method based on polar coordinate domain, constructs an angle coordinate system with the north direction and the horizontal plane as the reference datum, and uses it to represent the position relationship of the UAV relative to the base station. Specifically, first, the azimuth angle

[0060] and the pitch angle of each UAV at any time t are calculated, and the mechanical azimuth angle and the downtilt angle of the base station antenna are subtracted respectively, so as to obtain the relative angle coordinate of the UAV relative to the current antenna direction:

[0061]

[0062] By using the relative angle representation, the agent only needs to pay attention to the angle difference of the UAV relative to the main direction of the antenna, avoiding the perception of the antenna orientation information, thereby simplifying the state space of the reinforcement learning and improving the learning efficiency.

[0063]

[0064] Specifically, the angular velocity of the UAV in the polar coordinate domain relative to the base station, corresponding to the change rate of the azimuth angle and the pitch angle direction, is calculated as follows:

[0065]

[0066] ​​​

[0067] For the second vector:

[0068]

[0069] In the formula, This indicates the parameter configuration of the seven SSB beams at the previous moment, including but not limited to the electronic azimuth, electronic elevation, and beamwidth information for each beam.

[0070] For the third vector:

[0071]

[0072] In the formula, and These represent the azimuth and elevation difference matrices between the UAV and the seven beams, respectively. For example, its first line, number Column elements It is a drone With beam The azimuth difference value, The definition is similar. This represents the operation of expanding a matrix into column vectors by rows. This angular difference feature can enhance the agent's ability to perceive the angular deviation between the beam and the target, thereby helping to improve the accuracy and convergence efficiency of the beam pointing strategy.

[0073] II. Action Space.

[0074] In this embodiment, an association probability matrix is ​​innovatively introduced. This is used to explicitly model the service relationship between UAVs and SSB beams. The association probability matrix... It can be regarded as an attention mechanism with structure perception capabilities. It can not only enable the agent to perceive the service intensity of each beam to the drone, but also actively learn the matching pattern between the beam and the drone during the training process, thereby realizing the joint optimization of service relationship and beam parameters, rather than just passively adjusting the beam parameters.

[0075] Specifically, the correlation probability matrix The line, number Column elements Indicates drone Beam The probability of providing the service, and satisfying:

[0076]

[0077] Based on this correlation probability matrix , the action vector at time t is determined :

[0078]

[0079] wherein, , represents the beam information at time t; , represents the electronic azimuth angle of the beam , represents the electronic pitch angle of the beam , represents the beam width of the beam .

[0080] III. Reward function.

[0081] In the embodiments of the present application, in order to realize efficient tracking and dynamic optimization control of the base station SSB beam pair multi-target unmanned aerial vehicle, a reward function based on multi-target joint optimization is also designed , which comprises: determining the service probability of the target beam pair to the target unmanned aerial vehicle, and determining the signal receiving power of the target unmanned aerial vehicle after accessing the target beam, and determining a first reward function according to the service probability and the signal receiving power, wherein the target unmanned aerial vehicle is any one of the plurality of unmanned aerial vehicles, the target beam is the optimal beam corresponding to the target unmanned aerial vehicle in the target beam scheme, and the first reward function is used to quantify the signal quality received by the target unmanned aerial vehicle from the target beam; determining a second reward function according to the signal receiving power and a preset probability, wherein the second reward function is used to balance the signal strength and effective coverage degree of the target beam pair to the target unmanned aerial vehicle; determining a third reward function according to the beam configuration in the beam parameter vector and the historical beam configuration, wherein the third reward function is used to suppress the change of the beam parameter; and performing weighted processing on the first reward function, the second reward function and the third reward function to obtain the reward function.

[0082] The specific expression of the reward function is as follows:

[0083]

[0084] wherein, is a weighting coefficient, which is used to balance the optimization objectives among the first reward function , the second reward function and the third reward function .

[0085] For the first reward function , which is used to quantify the signal quality received by the unmanned aerial vehicle, a soft association mechanism is adopted to construct, and the calculation method is as follows:

[0086]

[0087] wherein, is the beam (i.e., the target beam) serving the UAV (i.e., the target UAV), is the UAV using the beam for communication. The first reward function can achieve a signal strength oriented learning goal, prompting the policy network to actively adjust the beam parameters to enhance the overall communication quality.

[0088] For the second reward function , which is used to encourage the policy to achieve complete target coverage while ensuring the strength of the main service beam, the calculation method is as follows:

[0089]

[0090] wherein, is an indicator function, which is valid coverage when the maximum received power of the UAV exceeds the set threshold , and is 1, otherwise 0. The second reward function can significantly reduce the uncovered probability of edge users, thereby enhancing the overall service integrity of the system.

[0091] For the third reward function , which is used to punish the degree of mutation of the current beam configuration and the beam configuration at the last time, to inhibit the violent jump of the beam parameters, and to avoid the delay caused by frequent adjustment to the communication system. The calculation method is as follows:

[0092]

[0093] The third reward function effectively improves the continuity and stability of the beam control policy by constraining the variation amplitude of the beam parameters at two consecutive time points.

[0094] Optionally, the state vector is processed by a target encoder in the policy network to obtain an association probability matrix and a beam parameter vector, including: determining an input feature corresponding to the state vector, and determining a first feature matrix inputting the policy network according to the input feature; processing the first feature matrix by the target encoder in the policy network to output a second feature matrix; normalizing the second feature matrix by a first branch in the policy network to obtain the association probability matrix; and performing a pooling operation on the second feature matrix by a second branch in the policy network to obtain the beam parameter vector.

[0095] In the embodiments of the present application, based on the DDPG reinforcement learning framework, a value network (Critic) and a policy network (Actor) are constructed for jointly solving the optimal beam configuration strategy. To enhance the modeling capability of the Actor network for the coupling between beam allocation and parameter adjustment, a shared encoder module is innovatively introduced into the Actor network, which takes the Transformer encoder as the core to perform unified feature extraction on the input state vector After unified feature extraction, the two parallel branches respectively output the association probability matrix and the beam parameter vector , realizing the structure-level joint learning of the association probability matrix and the beam parameter vector , and significantly improving the coordination and consistency between the two types of decisions. The overall Actor-Critic network framework is shown in Figure 3 , and the process analysis is as follows:

[0096] 1. Determine the input features.

[0097] First, based on the input state vector , the input features of each UAV are constructed as follows:

[0098]

[0099] wherein represents the coordinate information and velocity information of the UAV ; represents the azimuth angle difference and the elevation angle difference between the UAV and the 7 SSB beams; represents the beam state .

[0100] Subsequently, in the Actor network, the input features are spliced to obtain the first feature matrix input into the Transformer encoder, wherein N represents the number of input features .

[0101] 2. Semantic enhancement.

[0102] The first feature matrix is processed by the Transformer encoder in the Actor network to output the second feature matrix :

[0103]

[0104] The Transformer encoder consists of basic modules such as a multi-head self-attention mechanism, residual connections, layer normalization structure, and feedforward neural network, and outputs a second feature matrix. The first in OK For drones The context-enhanced representation includes its interactive semantic information in the global sequence.

[0105] 3. Branch processing.

[0106] The encoded second feature matrix The input is fed into two branches, which output the correlation probability matrix respectively. With beam parameter vector .

[0107] For the correlation probability matrix First, the second characteristic matrix After the input hidden layer, a set of fully connected layers is fed in, followed by normalization using the Sigmoid function to obtain the soft correlation probability between each UAV and each beam. The specific structure definition is as follows:

[0108]

[0109] In the formula, This is the network weight matrix. For bias vectors, This represents the element-wise Sigmoid function.

[0110] For beam parameter vector First, the second characteristic matrix After being fed into the hidden layer, the data is passed to a pooling layer for pooling operations, and then fed into a fully connected layer to obtain the control parameters for each beam. The specific structure is defined as follows:

[0111]

[0112]

[0113] In the formula, This represents the global context feature vector obtained after pooling operations. This application uses average pooling, i.e. This can also be extended to other pooling mechanisms; , These represent the weights and biases of the beam parameter prediction branch, respectively. This indicates the feature pooling operation.

[0114] 4. Determine the network's evaluation value.

[0115] In the embodiments of the present application, the Critic network adopts a multi-layer perceptron structure, and the original state vector is spliced with the action vector , i.e., the fused correlation probability matrix and the beam parameter vector , as input, processed through a series of fully connected layers and ReLU activation functions in the hidden layer, and the output of the last fully connected layer is taken as the evaluation value of the corresponding action , which is used to guide the optimization process of the Actor network and the Critic network in combination with the reward function.

[0116] In the above step S206, the pre-allocated beam scheme of the unmanned aerial vehicle is dynamically adjusted according to the correlation probability matrix and the beam parameter vector obtained in step S204. This adjustment includes both the reallocation of the beam service object and the real-time optimization of the beam parameters (such as azimuth angle, downtilt angle, and beam width).

[0117] In the embodiments of the present application, step S208 is further included: determining the action vector of the unmanned aerial vehicle according to the correlation probability matrix and the beam parameter vector, wherein the action vector is used to reflect the beam service decision made by the unmanned aerial vehicle; splicing the action vector and the state vector to obtain a spliced vector, and processing the spliced vector through a value network to obtain an evaluation value, wherein the value network is used to evaluate the value of the beam service decision made by the unmanned aerial vehicle; and optimizing the policy network and the value network according to the evaluation value and a reward function, wherein the reward function is used to quantify the signal quality, effective coverage rate and beam stability of the unmanned aerial vehicle after adopting the target beam allocation scheme, and the target beam allocation scheme is the pre-allocated beam scheme adjusted according to the correlation probability matrix and the beam parameter vector.

[0118] In order to accelerate the convergence speed of the policy Actor network, guide the agent to quickly master the optimization direction in the early stage of training, and effectively reduce the resource overhead in the online training process, the present application proposes an agent training and optimization method that combines expert policy supervised learning and reinforcement learning adaptive optimization. This method fully utilizes expert knowledge to pre-train the policy network, and combines reinforcement learning to fine-tune the performance, which can efficiently complete the construction of the optimal service relationship between the unmanned aerial vehicle and the base station beam and the configuration of the beam parameters. Specifically, it includes the following two stages:

[0119] I. Behavior cloning pre-training (i.e., Actor network pre-training).

[0120] Optionally, the policy network is trained by: obtaining a simulation state vector and a simulation action vector of the UAV, wherein the simulation action vector is a simulation correlation probability matrix and a simulation beam parameter vector generated by using a pre-allocation algorithm; processing the simulation state vector by an initial policy network to obtain a predicted action vector, wherein the predicted action vector includes a predicted correlation probability matrix and a predicted beam parameter vector output by the initial policy network; determining a first loss function required for training the initial policy network, and determining a first loss between the simulation action vector and the predicted action vector by the first loss function; and determining optimal parameters of the initial policy network by using a gradient descent method according to the first loss to obtain the policy network. The specific process analysis is as follows:

[0121] 1. Construct training data.

[0122] Construct sample data , wherein, represents a simulation state vector of the UAV, represents a simulation action vector of the UAV, , including a correlation probability matrix generated by a pre-allocation algorithm and a beam parameter vector .

[0123] 2. Determine the loss function required for training the Actor network .

[0124] The behavior cloning stage adopts a mean square error loss, and the specific expression is as follows:

[0125]

[0126] In the formula, that is, the first loss function described above, represents a loss weighting coefficient, which is used to control the learning balance of the two branches; represents a predicted action vector output by the Actor network, including a predicted correlation probability matrix and a predicted beam parameter vector .

[0127] 3. Update the parameters of the Actor network.

[0128] Determine the first loss between the simulation action vector and the predicted action vector by the first loss function, and update the parameters of the Actor network by using a gradient descent method according to the first loss.

[0129] II. Reinforcement learning fine-tuning (i.e., optimizing the Actor-Critic network).

[0130] Optionally, the policy network and the value network are optimized according to the evaluation value and the reward function, including: determining a target evaluation value according to the reward function, wherein the target evaluation value is used to represent an expected evaluation value under the action vector; determining a second loss function required for optimizing the value network, and determining a second loss between the evaluation value and the target evaluation value through the second loss function; optimizing the value network according to the second loss and a preset soft update rate, wherein the preset soft update rate is used to smooth the update process of the value network and the policy network; and determining a third loss function required for optimizing the policy network, and determining a third loss under the action vector through the third loss function; and optimizing the policy network according to the third loss and the preset soft update rate.

[0131] In the embodiments of the present application, the DDPG reinforcement learning framework is adopted to fine-tune the policy of the Actor network and synchronously train the Critic network, so that the agent can further optimize the policy in the interaction process with the environment, and break through the limitations of the expert policy. The specific process analysis is as follows:

[0132] 1. Determine the loss function required for optimizing the Critic network .

[0133] The training of the Critic network is based on the mean square temporal difference error, and the specific expression is as follows:

[0134]

[0135] In the formula, that is, the second loss function, represents the Q value (real evaluation value) output by the Critic network, represents the target Q value (expected evaluation value under the service decision), that is, the approximate "real" Q value, and the specific expression is as follows:

[0136]

[0137] In the formula, represents the reward function; is a discount factor; represents the state-action value function of the current Critic network, and the parameter is , which is used to estimate the action value under the current beam service policy, represents the optimized Critic network parameter; is the Actor network, and the parameter is , which is used to output the optimal beam policy under the given state, represents the state vector at t+1.

[0138] The training target of the Critic network is to minimize thereby improving the accuracy of the Q value estimation.

[0139] 2. Determine the loss function required for optimizing the Actor network .

[0140] The optimization goal of the Actor network is to maximize the action value under the current policy, which is expressed as follows:

[0141]

[0142] In the formula, i.e. the third loss function described above, which is used to improve the expected evaluation value of the current beam service policy in the Critic network through backpropagation.

[0143] 3. Optimize the Actor-Critic network.

[0144] According to the second loss obtained from the second loss function and the third loss obtained from the third loss function, an experience replay mechanism and a soft update target network are used, and the specific expression is as follows:

[0145]

[0146]

[0147] In the formula, represents a preset soft update rate, represents the optimized Critic network parameters, represents the optimized Actor network parameters.

[0148] The present application has been simulated and verified in a scenario where 20 unmanned aerial vehicles are operating simultaneously within the coverage range of a space domain base station. The traditional static beam scheme and the dynamic beam adjustment scheme proposed by the present application are compared respectively.

[0149] The verification results show that for the planning unmanned aerial vehicles with trajectory prediction information, the average received signal strength is improved by 4.5 dB, significantly enhancing the coverage quality; while for the black flying unmanned aerial vehicles not included in the trajectory planning, the average received signal strength is decreased by 5.8 dB, effectively suppressing the communication ability of unauthorized targets.

[0150] In the embodiments of the present application, by introducing the modeling mode of the association probability matrix and the Markov decision process, the joint optimization of the service relationship between the base station beam and the multiple unmanned aerial vehicles and the beam parameters is realized. Specifically, the present application utilizes the structure-aware association modeling mechanism, and through the soft association mechanism and the state representation based on the polar coordinate domain, not only the accuracy of the beam allocation decision of the intelligent agent is enhanced, but also the response speed and stability of the beam tracking are improved. In addition, by designing the policy network structure of the fusion shared encoder, the present application can simultaneously optimize the beam parameters and the service relationship on the basis of unified feature extraction, significantly improving the consistency and coordination of the decision, and thus improving the efficiency of the beam tracking.

[0151] According to the embodiments of the present application, a beam adjustment device for multiple unmanned aerial vehicles is provided. It should be noted that the beam adjustment device for multiple unmanned aerial vehicles in the embodiments of the present application can be used to execute the beam adjustment method for multiple unmanned aerial vehicles provided by the embodiments of the present application. The beam adjustment device for multiple unmanned aerial vehicles provided by the embodiments of the present application is introduced below.

[0152] Figure 4 is a structural diagram of a beam adjustment device for multiple unmanned aerial vehicles according to the embodiments of the present application. As shown in the figure, Figure 4 the device comprises:

[0153] The acquisition module 40 is configured to acquire a pre-allocated beam scheme of the unmanned aerial vehicle and position information of the unmanned aerial vehicle in the flight process, wherein the pre-allocated beam scheme comprises an initial beam allocation scheme corresponding to the multiple unmanned aerial vehicles.

[0154] The processing module 42 is configured to determine a state vector of the unmanned aerial vehicle according to the position information, and process the state vector through a target encoder in a policy network to obtain an association probability matrix and a beam parameter vector, wherein the policy network is used to adjust the service decision of the base station beam to the unmanned aerial vehicle, the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, and outputs the association probability matrix and the beam parameter vector through different branches, the association probability matrix is used to reflect the service relationship between the unmanned aerial vehicle and the base station beam, and the beam parameter vector is used to reflect the parameter configuration of the base station beam.

[0155] The adjustment module 44 is configured to adjust the pre-allocated beam scheme according to the association probability matrix and the beam parameter vector.

[0156] The acquisition module, the processing module and the adjustment module in the multi-unmanned aerial vehicle beam adjustment device achieve the purpose of intelligently optimizing the multi-unmanned aerial vehicle service decision of the base station beam, thereby achieving the technical effects of improving the communication quality and resource utilization efficiency, enhancing the beam tracking stability and response speed, and further solving the technical problems of communication discontinuous coverage, unstable signal quality and low resource scheduling efficiency of the static beam scheme in the related art under the high-speed movement and complex scene of the multi-unmanned aerial vehicle.

[0157] In the multi-unmanned aerial vehicle beam adjustment device provided in the embodiments of the application, the processing module is further configured to determine a first vector, a second vector and a third vector according to the position information, wherein the first vector is used to reflect the angle coordinate information and the speed change information of the unmanned aerial vehicle relative to the base station, the second vector is used to reflect the historical beam parameter information of the unmanned aerial vehicle, and the third vector is used to reflect the angle difference between the unmanned aerial vehicle and the base station beam; and the state vector is obtained by fusing the first vector, the second vector and the third vector.

[0158] In the multi-unmanned aerial vehicle beam adjustment device provided in the embodiments of the application, the processing module is further configured to determine the input feature corresponding to the state vector, and determine the first feature matrix of the input policy network according to the input feature; the first feature matrix is processed by the target encoder in the policy network to output the second feature matrix; the second feature matrix is normalized by the first branch in the policy network to obtain the association probability matrix; and the second feature matrix is subjected to the pooling operation by the second branch in the policy network to obtain the beam parameter vector.

[0159] In the multi-unmanned aerial vehicle beam adjustment device provided in the embodiments of the application, the processing module is further configured to determine the service probability of the target beam to the target unmanned aerial vehicle, and determine the signal receiving power of the target unmanned aerial vehicle after accessing the target beam, and determine the first reward function according to the service probability and the signal receiving power, wherein the target unmanned aerial vehicle is any one of the plurality of unmanned aerial vehicles, the target beam is the optimal beam corresponding to the target unmanned aerial vehicle in the target beam scheme, and the first reward function is used to quantify the signal quality received by the target unmanned aerial vehicle from the target beam; the second reward function is determined according to the signal receiving power and the preset probability, wherein the second reward function is used to balance the signal strength and the effective coverage degree of the target beam to the target unmanned aerial vehicle; the third reward function is determined according to the beam configuration in the beam parameter vector and the historical beam configuration, wherein the third reward function is used to suppress the change of the beam parameter; and the first reward function, the second reward function and the third reward function are subjected to the weighted processing to obtain the reward function.

[0160] In the multi-unmanned aerial vehicle beam adjustment apparatus provided in the embodiments of the present application, the training module 46 is further configured to obtain a simulation state vector and a simulation action vector of the unmanned aerial vehicle, wherein the simulation action vector is a simulation correlation probability matrix and a simulation beam parameter vector generated by using a pre-allocation algorithm; process the simulation state vector by using an initial policy network to obtain a predicted action vector, wherein the predicted action vector includes a predicted correlation probability matrix and a predicted beam parameter vector output by the initial policy network; determine a first loss function required for training the initial policy network, and determine a first loss between the simulation action vector and the predicted action vector by using the first loss function; determine optimal parameters of the initial policy network by using a gradient descent method according to the first loss to obtain a policy network.

[0161] In the multi-unmanned aerial vehicle beam adjustment apparatus provided in the embodiments of the present application, the optimization module 48 is further configured to determine an action vector of the unmanned aerial vehicle according to the correlation probability matrix and the beam parameter vector, wherein the action vector is used to reflect a beam service decision made by the unmanned aerial vehicle; splice the action vector and the state vector to obtain a spliced vector, and process the spliced vector by using a value network to obtain an evaluation value, wherein the value network is used to evaluate the value of the beam service decision made by the unmanned aerial vehicle; optimize the policy network and the value network according to the evaluation value and a reward function, wherein the reward function is used to quantify the signal quality, the effective coverage rate and the beam stability of the unmanned aerial vehicle after using a target beam allocation scheme, and the target beam allocation scheme is a pre-allocated beam scheme adjusted according to the correlation probability matrix and the beam parameter vector.

[0162] In the multi-unmanned aerial vehicle beam adjustment apparatus provided in the embodiments of the present application, the optimization module is further configured to determine a target evaluation value according to the reward function, wherein the target evaluation value is used to represent an expected evaluation value under the action vector; determine a second loss function required for optimizing the value network, and determine a second loss between the evaluation value and the target evaluation value by using the second loss function; optimize the value network according to the second loss and a preset soft update rate, wherein the preset soft update rate is used to smooth the update process of the value network and the policy network; and determine a third loss function required for optimizing the policy network, and determine a third loss under the action vector by using the third loss function; optimize the policy network according to the third loss and the preset soft update rate.

[0163] The embodiments of the present application further provide an electronic device, including a memory and a processor, wherein the memory is configured to store program instructions; the processor is connected with the memory and is configured to execute the above-mentioned multi-unmanned aerial vehicle beam adjustment method.

[0164] It should be noted that the above-mentioned electronic device is used to execute the above-mentioned multi-unmanned aerial vehicle beam adjustment method. Figure 2The multi-UAV beam adjustment method shown in the method is applicable to the electronic device, and thus the related explanations in the multi-UAV beam adjustment method are also applicable to the electronic device, which will not be repeated here.

[0165] The embodiment of the present application further provides a nonvolatile storage medium, which comprises a stored computer program, wherein a device where the nonvolatile storage medium is located executes the multi-UAV beam adjustment method by running the computer program.

[0166] It should be noted that the nonvolatile storage medium is used for executing the multi-UAV beam adjustment method. Figure 2 The multi-UAV beam adjustment method shown in the method is applicable to the nonvolatile storage medium, and thus the related explanations in the multi-UAV beam adjustment method are also applicable to the nonvolatile storage medium, which will not be repeated here.

[0167] The embodiment of the present application further provides a computer program product, which comprises computer instructions, and the computer instructions are executed by a processor to realize the multi-UAV beam adjustment method.

[0168] It should be noted that the computer program product is used for executing the multi-UAV beam adjustment method. Figure 2 The multi-UAV beam adjustment method shown in the method is applicable to the computer program product, and thus the related explanations in the multi-UAV beam adjustment method are also applicable to the computer program product, which will not be repeated here.

[0169] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0170] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0171] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit embodiment described above is only illustrative, and for example, the division of units can be a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0172] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0173] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0174] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0175] The above is only the preferred embodiment of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A method for adjusting a beam of multiple unmanned aerial vehicles, characterized in that, The method comprises: obtaining a pre-allocated beam scheme of a UAV and position information of the UAV in a flight process, wherein the pre-allocated beam scheme comprises an initial beam allocation scheme corresponding to a plurality of UAVs; determining a state vector of the UAV according to the position information, and processing the state vector through a target encoder in a policy network to obtain an association probability matrix and a beam parameter vector, wherein the policy network is used to adjust a service decision of a base station beam on the UAV, the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, and outputs the association probability matrix and the beam parameter vector through different branches, the association probability matrix is used to reflect a high-order interaction relationship between a mutual spatial distribution mode of the UAVs and the base station beams, and the beam parameter vector is used to reflect a parameter configuration of the base station beams; adjusting the pre-allocated beam scheme according to the association probability matrix and the beam parameter vector; determining the state vector of the UAV according to the position information comprises: determining a first vector, a second vector and a third vector respectively according to the position information, wherein the first vector is used to reflect angle coordinate information and velocity change information of the UAV relative to a base station, the second vector is used to reflect historical beam parameter information of the UAV, and the third vector is used to reflect an angle difference between the UAV and the base station beams; and fusing the first vector, the second vector and the third vector to obtain the state vector.

2. The method of claim 1, wherein, processing the state vector through the target encoder in the policy network to obtain the association probability matrix and the beam parameter vector comprises: determining an input feature corresponding to the state vector, and determining a first feature matrix input into the policy network according to the input feature; processing the first feature matrix through the target encoder in the policy network to output a second feature matrix; performing normalization processing on the second feature matrix through a first branch in the policy network to obtain the association probability matrix; and performing a pooling operation on the second feature matrix through a second branch in the policy network to obtain the beam parameter vector.

3. The method of claim 1, wherein, The policy network is trained in the following manner: obtaining a simulation state vector and a simulation action vector of the UAV, wherein the simulation action vector is a simulation association probability matrix and a simulation beam parameter vector generated by using a pre-allocated algorithm; processing the simulation state vector through an initial policy network to obtain a predicted action vector, wherein the predicted action vector comprises a predicted association probability matrix and a predicted beam parameter vector output by the initial policy network; determining a first loss function required for training the initial policy network, and determining a first loss between the simulation action vector and the predicted action vector through the first loss function; determining optimal parameters of the initial policy network through a gradient descent method according to the first loss to obtain the policy network.

4. The method of claim 1, wherein, The method further comprises: determine an action vector of the UAV according to the association probability matrix and the beam parameter vector, wherein the action vector is used to reflect a beam service decision made by the UAV; splice the action vector and the state vector to obtain a spliced vector, and process the spliced vector through a value network to obtain an evaluation value, wherein the value network is used to evaluate the value of the beam service decision made by the UAV; optimize the policy network and the value network according to the evaluation value and a reward function, wherein the reward function is used to quantify the signal quality, effective coverage rate and beam stability of the UAV after adopting a target beam allocation scheme, and the target beam allocation scheme is a pre-allocated beam scheme adjusted according to the association probability matrix and the beam parameter vector.

5. The method of claim 4, wherein, The reward function is determined by the following methods: determine a service probability of a target beam to a target UAV, and determine a signal reception power of the target UAV after accessing the target beam, and determine a first reward function according to the service probability and the signal reception power, wherein the target UAV is any one of the plurality of UAVs, the target beam is an optimal beam corresponding to the target UAV in the target beam allocation scheme, and the first reward function is used to quantify the signal quality received by the target UAV from the target beam; determine a second reward function according to the signal reception power and a preset probability, wherein the second reward function is used to balance the signal strength and effective coverage degree of the target beam to the target UAV; determine a third reward function according to a beam configuration in the beam parameter vector and a historical beam configuration, wherein the third reward function is used to suppress changes in beam parameters; perform weighted processing on the first reward function, the second reward function and the third reward function to obtain the reward function.

6. The method of claim 4, wherein, optimizing the policy network and the value network according to the evaluation value and a reward function includes: determine a target evaluation value according to the reward function, wherein the target evaluation value is used to represent an expected evaluation value under the action vector; determine a second loss function required for optimizing the value network, and determine a second loss between the evaluation value and the target evaluation value through the second loss function; optimize the value network according to the second loss and a preset soft update rate, wherein the preset soft update rate is used to smooth the update process of the value network and the policy network; and determine a third loss function required for optimizing the policy network, and determine a third loss under the action vector through the third loss function; optimize the policy network according to the third loss and the preset soft update rate. 7.A multi-UAV beam adjustment apparatus, characterized in that, includes: an acquisition module, configured to acquire a pre-allocated beam scheme of a UAV and position information of the UAV in a flight process, wherein the pre-allocated beam scheme includes an initial beam allocation scheme corresponding to a plurality of UAVs; The processing module is configured to determine a state vector of the UAV according to the position information, and process the state vector through a target encoder in a policy network to obtain an association probability matrix and a beam parameter vector, wherein the policy network is used to adjust a service decision of a base station beam on the UAV, the target encoder performs semantic enhancement on the state vector through a multi-head attention mechanism, and outputs the association probability matrix and the beam parameter vector through different branches, the association probability matrix is used to reflect a high-order interaction relationship between a mutual spatial distribution mode of the UAVs and the base station beams, and the beam parameter vector is used to reflect a parameter configuration of the base station beams; and determining the state vector of the UAV according to the position information comprises: determining a first vector, a second vector and a third vector respectively according to the position information, wherein the first vector is used to reflect angle coordinate information and velocity change information of the UAV relative to a base station, the second vector is used to reflect historical beam parameter information of the UAV, and the third vector is used to reflect an angle difference between the UAV and the base station beams; and fusing the first vector, the second vector and the third vector to obtain the state vector. The adjusting module is configured to adjust the pre-allocated beam scheme according to the association probability matrix and the beam parameter vector.

8. An electronic device, comprising: The memory and the processor are included, wherein the memory is configured to store program instructions; The processor is connected with the memory and is configured to execute the multi-UAV beam adjustment method in any one of claims 1 to 6. The non-volatile storage medium includes a stored computer program, wherein a device in which the non-volatile storage medium is located executes the multi-UAV beam adjustment method in any one of claims 1 to 6 by running the computer program.

9. A non-volatile storage medium, comprising: The computer instructions are executed by the processor to implement the multi-UAV beam adjustment method in any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that, ​

Images