Satellite hopping beam method based on federated learning

Abstract

The invention relates to the technical field of satellite communication, in particular to a federated learning-based satellite beam hopping method, which comprises the following steps of: acquiring a local model gradient of each satellite; performing quality estimation on the local model gradient of each satellite, and performing asynchronous federated aggregation on a plurality of gradients meeting a preset quality standard condition to obtain a new global model gradient; and calculating the similarity between the local model gradient of each satellite and the new global model gradient, and when the similarity is greater than a preset threshold, determining that the new global model is in a preset convergence state. Therefore, the problems of high user data security risk and too high system synchronous communication complexity in the existing beam hopping technology are solved, the user data security is ensured, the system synchronous communication complexity is reduced, the calculation efficiency is improved, and effective utilization and long-term stable operation of satellite communication system resources are ensured.

Description

Technical Field

[0001] The present application relates to the field of satellite communication technology, and in particular to a satellite beam hopping method based on federated learning. Background Art

[0002] With the advent of the information age, people's demand for efficient and convenient information acquisition is growing, and communication networks have become an important factor affecting economic and social development. In satellite communications, users can only obtain services if they are located in the area illuminated by the satellite beam, otherwise they cannot complete communication. However, due to the uneven distribution of ground users and the rapid movement of low-orbit satellites around the earth, the beam coverage area changes frequently, which often results in a waste of satellite resources. To solve this problem, by reducing the beam illumination range and appropriately increasing the number of beams, limited on-board resources can be concentrated in a smaller area, thereby illuminating different areas at different times, achieving the same coverage effect as a wide beam while avoiding resource waste. Among them, the technology that determines to illuminate a specific smaller area at a specific time is called beam hopping technology.

[0003] Although beam hopping technology improves the flexibility and efficiency of satellite communication systems, it also brings new challenges in how to optimize beam allocation and ensure the long-term efficient operation of the system. After determining the beam coverage area, it is still necessary to further optimize the allocation of limited onboard resources, taking into account performance indicators such as system throughput and latency. In addition, the resource allocation strategy must not only meet immediate user needs and system load, but also estimate the impact of decisions on future system performance to ensure continuous optimization of communication quality.

[0004] As a powerful extension of the ground network, the satellite communication network is an important means of communication for construction in difficult areas and emergency rescue. Among them, low-orbit multi-beam satellites are an important component of the satellite communication network. The research on beam hopping and resource allocation algorithms of low-orbit multi-beam satellites is of great significance.

[0005] However, although beam-hopping technology plays an important role in improving the resource utilization of satellite communication systems, the difficulty in ensuring user data security and the high system complexity caused by synchronous communication are still key challenges in its development. In current research, beam-hopping technology relies on frequent communications between satellites and ground stations to transmit data such as user needs and channel status. This transmission is mostly unprocessed raw data, which is vulnerable to attacks and poses security risks.

[0006] At present, existing technologies mainly focus on adaptive scheduling and resource allocation algorithms, cross-satellite collaborative optimization, and the application of AI large model algorithms, with the goal of improving the resource utilization and communication efficiency of satellite systems. However, in many current studies, resource scheduling is still concentrated in the ground control center. Each strategy update requires a large amount of data exchange between the satellite and the ground station, resulting in a high security vulnerability of the central node of the system. Once the central node is attacked, the key data of the entire system will face the risk of leakage, which needs to be solved urgently. Summary of the invention

[0007] The present application provides a satellite beam hopping method based on federated learning to solve the problems of high user data security risk and high system synchronization communication complexity in existing beam hopping technology. It achieves the goal of reducing the complexity of system synchronization communication and improving computing efficiency while ensuring user data security, thereby ensuring the effective use of satellite communication system resources and long-term stable operation.

[0008] The first aspect of the present application provides a satellite beam hopping method based on federated learning, comprising the following steps: Get the local model gradient for each satellite; Performing quality estimation on the local model gradient of each satellite, and performing asynchronous federation aggregation on multiple gradients that meet preset quality standards to obtain a new global model gradient; The similarity between the local model gradient of each satellite and the new global model gradient is calculated, and when the similarity is greater than a preset threshold, it is determined that the new global model is in a preset convergence state.

[0009] According to one embodiment of the present application, obtaining the local model gradient of each satellite includes: Get the global model parameters of the central node; The global model parameters are initialized, and beam hopping strategy training is performed locally on each satellite based on the initialized global model parameters, and the local model gradient of each satellite is obtained after the training is completed.

[0010] According to one embodiment of the present application, the locally performing beam hopping strategy training on each satellite based on the initialized global model includes: Acquire current system status information of each satellite within a preset time step, wherein the current system status information includes at least one of user demand, interference situation, beam overlap area and channel status; Based on the current system state information of each satellite and the current policy network, determine the target execution action corresponding to each satellite, update the current system state information of each satellite after each satellite executes the corresponding target execution action, and calculate the reward obtained by each satellite; Recording a four-tuple consisting of the current system state information, the target execution action, the reward, and the updated current system state information, and calculating the advantage function based on the four-tuple; According to the value of the advantage function in the new update cycle, the update amplitude of the policy network and / or the parameters of the policy network are determined, and the parameters of the value network are updated based on minimizing the mean square error loss function to obtain updated value network parameters.

[0011] According to an embodiment of the present application, the quality estimation of the local model gradient of each satellite and asynchronous federation aggregation of multiple gradients that meet preset quality standards include: Calculate the comprehensive quality index value of the local model gradient of each satellite based on a preset quality index calculation formula; If the comprehensive quality index value is greater than the preset index value, it is determined that the local model gradient meets the preset quality standard condition, and the number of local model gradients that meet the preset quality standard condition is obtained; If the number of local model gradients that meet the preset quality standard condition reaches a preset number, asynchronous federation aggregation is performed on multiple gradients that meet the preset quality standard condition based on a preset aggregation formula; The preset quality index calculation formula is: ; in, is the comprehensive quality index value of the i-th satellite node, is the time step of this asynchronous federated aggregation; is the staleness of the ith satellite node, is a constant that controls the decay rate of staleness, is the gradient of the policy network parameters of the i-th satellite node and the gradient of the value network parameters the time that has elapsed since the calculation; for and The cosine similarity of for and The cosine similarity of is the gradient of the global policy network parameters, is the gradient of the global value network parameters, is the Euclidean norm of the gradient of the local policy network parameters, is the Euclidean norm of the gradient of the global policy network parameters, is the Euclidean norm of the gradient of the local value network parameters, is the Euclidean norm of the gradient of the global value network parameters.

[0012] According to one embodiment of the present application, after calculating the similarity between the local model gradient of each satellite and the new global model gradient, the method further includes: If the similarity is less than or equal to the preset threshold, broadcasting the new global model gradient to each satellite; The local model parameters of each satellite are gradient updated according to the new global model gradient.

[0013] According to one embodiment of the present application, after determining that the new global model is in a preset convergence state, the method further includes: Calculating the inversion success rate of the new global model; Verifying whether the new global model meets a preset training termination condition according to the inversion success rate; If the new global model satisfies the preset training termination condition, the training of the new global model is terminated; otherwise, the step of broadcasting the new global model gradient to each satellite is performed.

[0014] According to a satellite beam-hopping method based on federated learning in an embodiment of the present application, the quality of the local model gradient of each satellite is estimated, and multiple gradients that meet the preset quality standards are asynchronously federated to obtain a new global model gradient; the similarity between the local model gradient of each satellite and the new global model gradient is calculated, and when the similarity is greater than a preset threshold, the new global model is determined to be in a preset convergence state. In this way, the problems of high user data security risk and high system synchronization communication complexity in the existing beam-hopping technology are solved, and the system synchronization communication complexity is reduced while ensuring user data security, and the computing efficiency is improved, ensuring the effective use of satellite communication system resources and long-term stable operation.

[0015] The second aspect of the present application provides a satellite beam hopping device based on federated learning, including: An acquisition module, used to obtain the local model gradient of each satellite; An aggregation module is used to estimate the quality of the local model gradient of each satellite, and asynchronously aggregate multiple gradients that meet preset quality standards to obtain a new global model gradient; The optimization module is used to calculate the similarity between the local model gradient of each satellite and the new global model gradient, and when the similarity is greater than a preset threshold, determine that the new global model is in a preset convergence state.

[0016] According to one embodiment of the present application, the acquisition module is used to: Get the global model parameters of the central node; The global model parameters are initialized, and beam hopping strategy training is performed locally on each satellite based on the initialized global model parameters, and the local model gradient of each satellite is obtained after the training is completed.

[0017] According to one embodiment of the present application, the acquisition module is used to: Acquire current system status information of each satellite within a preset time step, wherein the current system status information includes at least one of user demand, interference situation, beam overlap area and channel status; Based on the current system state information of each satellite and the current policy network, determine the target execution action corresponding to each satellite, update the current system state information of each satellite after each satellite executes the corresponding target execution action, and calculate the reward obtained by each satellite; Recording a four-tuple consisting of the current system state information, the target execution action, the reward, and the updated current system state information, and calculating the advantage function based on the four-tuple; According to the value of the advantage function in the new update cycle, the update amplitude of the policy network and / or the parameters of the policy network are determined, and the parameters of the value network are updated based on minimizing the mean square error loss function to obtain updated value network parameters.

[0018] According to one embodiment of the present application, the aggregation module is used to: Calculate the comprehensive quality index value of the local model gradient of each satellite based on a preset quality index calculation formula; If the comprehensive quality index value is greater than the preset index value, it is determined that the local model gradient meets the preset quality standard condition, and the number of local model gradients that meet the preset quality standard condition is obtained; If the number of local model gradients that meet the preset quality standard condition reaches a preset number, asynchronous federation aggregation is performed on multiple gradients that meet the preset quality standard condition based on a preset aggregation formula; The preset quality index calculation formula is: ; in, is the comprehensive quality index value of the i-th satellite node, is the time step of this asynchronous federated aggregation; is the staleness of the ith satellite node, is a constant that controls the decay rate of staleness, is the gradient of the policy network parameters of the i-th satellite node and the gradient of the value network parameters the time that has elapsed since the calculation; for and The cosine similarity of for and The cosine similarity of is the gradient of the global policy network parameters, is the gradient of the global value network parameters, is the Euclidean norm of the gradient of the local policy network parameters, is the Euclidean norm of the gradient of the global policy network parameters, is the Euclidean norm of the gradient of the local value network parameters, is the Euclidean norm of the gradient of the global value network parameters.

[0019] According to one embodiment of the present application, after calculating the similarity between the local model gradient of each satellite and the new global model gradient, the optimization module is further used to: If the similarity is less than or equal to the preset threshold, broadcasting the new global model gradient to each satellite; The local model parameters of each satellite are gradient updated according to the new global model gradient.

[0020] According to one embodiment of the present application, after determining that the new global model is in a preset convergence state, the optimization module is further used to: Calculating the inversion success rate of the new global model; Verifying whether the new global model meets a preset training termination condition according to the inversion success rate; If the new global model satisfies the preset training termination condition, the training of the new global model is terminated; otherwise, the step of broadcasting the new global model gradient to each satellite is performed.

[0021] According to a satellite beam-hopping device based on federated learning in an embodiment of the present application, the quality of the local model gradient of each satellite is estimated, and multiple gradients that meet the preset quality standards are asynchronously federated and aggregated to obtain a new global model gradient; the similarity between the local model gradient of each satellite and the new global model gradient is calculated, and when the similarity is greater than a preset threshold, the new global model is determined to be in a preset convergence state. In this way, the problems of high user data security risk and high system synchronization communication complexity in the existing beam-hopping technology are solved, and the system synchronization communication complexity is reduced while ensuring user data security, and the computing efficiency is improved, ensuring the effective use of satellite communication system resources and long-term stable operation.

[0022] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a satellite beam hopping method based on federated learning as described in the above embodiment.

[0023] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement a satellite beam hopping method based on federated learning as described in the above embodiment.

[0024] Beneficial effects of the present invention: (1) Compared with the centralized architecture, the system's synchronous communication complexity is reduced: the traditional centralized method of beam hopping technology requires the central node to communicate frequently with all devices and perform large-scale global optimization. The federated learning adopted by the present invention reduces the need for global synchronization through local training and asynchronous upload of model parameters, avoiding the system complexity caused by synchronous large-scale communication.

[0025] (2) Effectively protect user data security: Since the needs and location information data of ground users are of high importance, if the data is to be uploaded to the central node for processing, the possibility of user data leakage during transmission increases. The characteristics of federated learning adopted by the present invention do not require the original data to be uploaded to the central server.

[0026] (3) Improving computing efficiency: Federated learning allows each device to independently perform local model training, which means that the system can make full use of distributed computing resources and reduce the computing load of the central node. This can effectively improve computing efficiency when the satellite network is large in scale.

[0027] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a satellite beam hopping method based on federated learning provided according to an embodiment of the present application; Figure 2 A flowchart of satellite-local model training according to one embodiment of the present application; Figure 3 A flowchart of asynchronous federated aggregation according to an embodiment of the present application; Figure 4A schematic diagram of a flow chart of a satellite beam hopping method based on federated learning according to an embodiment of the present application; Figure 5 is a block diagram of a satellite beam hopping device based on federated learning according to an embodiment of the present application; Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes a satellite beam hopping method based on federated learning according to an embodiment of the present application with reference to the accompanying drawings. In response to the problems of high user data security risks and high system synchronization communication complexity mentioned in the above background technology, the present application provides a satellite beam hopping method based on federated learning, which solves the problems of high user data security risks and high system synchronization communication complexity in existing beam hopping technologies. Federated learning is used to optimize the algorithm for beam hopping resource allocation to enhance the protection of key data, reduce the complexity of system synchronization operations, adapt to the needs of the actual operation of beam hopping technology, and improve the feasibility of beam hopping resource allocation technology.

[0031] Specifically, Figure 1 A flowchart of a satellite beam hopping method based on federated learning is provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the satellite beam hopping method based on federated learning includes the following steps: In step S101, the local model gradient of each satellite is obtained.

[0033] Furthermore, in some embodiments, obtaining the local model gradient of each satellite includes: obtaining the global model parameters of the central node; initializing the global model parameters, and locally training the beam hopping strategy for each satellite based on the initialized global model parameters, and obtaining the local model gradient of each satellite after the training is completed.

[0034] Specifically, at the beginning of the system, the global model parameters and training data of the central node, that is, the parameters of the policy network, are initialized. and the parameters of the value network , and broadcast the parameters of the initialized policy network and the value network to all participating satellite nodes, so that all satellite nodes can be trained based on the initialized global model parameters.

[0035] Exemplarily, the specific initialization method in the embodiment of the present application may adopt Xavier initialization: (1) in, are the initial parameters of the policy network, is the initial parameter of the value network, is the number of input neurons of the policy network and the value network.

[0036] Furthermore, the initial training dataset can be expressed as ,in, For The system state at the time step, the system state includes Information such as user demand, channel status, beam allocation status, system load, etc. within the time step.

[0037] Further, in some embodiments, beam hopping strategy training is performed locally on each satellite based on the initialized global model, including: obtaining current system status information of each satellite within a preset time step, the current system status information including at least one of user demand, interference situation, beam overlap area and channel status; determining the target execution action corresponding to each satellite based on the current system status information of each satellite and the current policy network, and updating the current system status information of each satellite after each satellite executes the corresponding target execution action, and calculating the reward obtained by each satellite; recording a four-tuple consisting of the current system status information, the target execution action, the reward and the updated current system status information, and calculating the advantage function based on the four-tuple; determining the update amplitude of the policy network and / or the parameters of the policy network according to the value of the advantage function in the new update cycle, and updating the parameters of the value network based on minimizing the mean square error loss function to obtain the updated value network parameters.

[0038] The preset time step may be a time step preset by a person skilled in the art, and is not specifically limited here.

[0039] Specifically, the specific policy training process of the embodiment of the present application adopts the Proximal Policy Optimization (PPO) algorithm. Compared with the deep reinforcement learning algorithm, the PPO algorithm has the advantage of not relying on the experience replay buffer and being more adaptable to the environment of continuous action; at the same time, compared with other policy-based reinforcement learning algorithms, such as Deep Deterministic Policy Gradient and Actor-Critic algorithm, the PPO algorithm has a unique clipping mechanism that can reduce the occurrence of excessive policy updates.

[0040] The process of performing beam hopping strategy training on each satellite locally based on the initialized global model in the embodiment of the present application may specifically include the following steps: First, obtain the time step of each satellite Current system status at , including user needs, interference conditions, beam overlap areas, and channel status.

[0041] Secondly, based on the current policy network , each satellite selects the corresponding target execution action from the current strategy network , that is, determining the beam allocation and hopping decisions.

[0042] Furthermore, the corresponding target execution action is executed on each satellite After that, the system enters a new state , and each satellite performs actions according to the corresponding target to obtain rewards .

[0043] Among them, the design of the reward function is as follows: (2) in, For throughput rewards, To satisfy user needs, For inter-satellite interference penalty, For delay penalty, is the weight of the throughput reward, is the weight of user demand satisfaction, is the weight of the inter-satellite interference penalty, is the weight of the delay penalty.

[0044] Furthermore, each satellite will perform the action and state changes are recorded as a four-tuple , for subsequent local model gradient calculation, thus obtaining a series of state-action-reward-next-state quadruplets to form a decision trajectory , store the decision trajectory obtained from each interaction with the environment into the experience pool .

[0045] The calculation formula of the advantage function is as follows: (3) in, is the advantage function, is the action value function, indicating that the action is executed The expected return after is the output of the value network, representing the state value function.

[0046] It is understandable that, assuming that satellite A selects an action according to the current state in a certain update cycle and executes it, then records the corresponding quadruple and calculates the advantage function. If the value of the advantage function decreases in the new update cycle, it indicates that the newly updated local model parameters may cause performance degradation. Therefore, it is necessary to use a clipping mechanism to suppress the update of the satellite's local model. The clipping mechanism will limit the amplitude of the parameter update to ensure that the satellite's local model will not degrade due to excessive updates.

[0047] Based on this, the embodiment of the present application adds a clipping term to the strategy loss function to limit the range of probability ratio changes, prevent the satellite local model parameter update amplitude from being too large, and thus avoid performance regression. The strategy loss function is as follows: (4) in, is the strategy loss function, For new strategies to be updated, For the old policy to be updated, is the advantage function; the clip function is used to constrain the ratio of the new and old strategies to , to limit the update range of the strategy, is a small constant that controls the update amplitude, usually between 0.1 and 0.2.

[0048] Furthermore, the policy network of each satellite local model uses the gradient ascent method to update the policy network parameters based on the advantage function. The specific formula is as follows: (5) in, are the updated policy network parameters, is the learning rate, which can be a preset constant. The policy network parameters representing the loss function for the satellite local model Find the gradient.

[0049] Finally, the value network of each satellite local model updates the value network parameters by minimizing the mean square error loss function to better predict the state value. The specific formula is as follows: (6) in, is the value loss function, For the experience pool, is the output of the value network, representing the state value function, which is used to estimate the system state The value of; γ is the discount factor, which is used to control the impact of future estimated rewards and is usually between [0,1]; Decision Trajectory The number of time steps in ; is the immediate reward at the new time step, obtained by the reward function, that is, formula (2). is the updated value network parameter, is the learning rate, The network parameters representing the value loss function for the satellite local model Find the gradient.

[0050] In order to facilitate those skilled in the art to more clearly and intuitively understand the process of the embodiment of the present application for each satellite to perform beam hopping strategy training locally, the following is combined with Figure 2 Provide detailed explanation.

[0051] like Figure 2 As shown, the process of training the beam hopping strategy locally for each satellite includes the following steps: S201, obtaining the current system status.

[0052] S202, select and execute an action according to the current strategy network.

[0053] S203, updating the system status.

[0054] S204, calculate the reward.

[0055] S205, record the four-tuple (state, action, reward, new state).

[0056] S206, calculating the advantage function.

[0057] S207, updating the policy network parameters and triggering the clipping mechanism.

[0058] S208, update value network parameters.

[0059] Furthermore, each satellite calculates its own local model gradient, including the gradient of the updated policy network loss function to the policy network parameters. and the gradient of the updated value network loss function with respect to the value network parameters Since the results of each satellite at each time step are different, in order to facilitate the understanding of those skilled in the art, the two gradient results can be recorded as and , and upload the local model gradient of each satellite to the central node.

[0060] In step S102, the quality of the local model gradient of each satellite is estimated, and multiple gradients that meet the preset quality standards are asynchronously federated and aggregated to obtain a new global model gradient.

[0061] Furthermore, in some embodiments, the quality of the local model gradient of each satellite is estimated, and multiple gradients that meet preset quality standards are asynchronously federally aggregated, including: based on a preset quality indicator calculation formula, calculating the comprehensive quality indicator value of the local model gradient of each satellite; if the comprehensive quality indicator value is greater than the preset indicator value, determining that the local model gradient meets the preset quality standards, and obtaining the number of local model gradients that meet the preset quality standards; if the number of local model gradients that meet the preset quality standards reaches a preset number, asynchronously federally aggregate the multiple gradients that meet the preset quality standards based on a preset aggregation formula. Among them, the preset quality compliance condition can be a quality compliance condition pre-set by technical personnel in this field, such as a comprehensive quality index value exceeding a quality threshold; the preset quantity can be a quantity pre-set by technical personnel in this field, and is not specifically limited here.

[0062] Specifically, after receiving the local model gradients of a preset number of satellites that meet the preset quality standards, the central node performs asynchronous federated aggregation and calculates a new global model. The quality estimation of the embodiment of the present application is calculated based on the cosine similarity and staleness of the gradients, and the final comprehensive quality index is represented by the sum of the cosine similarity and staleness of the gradients, so that the local model gradients whose comprehensive quality index values ​​meet the preset quality standards are allowed to participate in the subsequent weighted asynchronous federated aggregation.

[0063] Among them, the preset quality index calculation formula is: ; in, is the comprehensive quality index value of the i-th satellite node, is the time step of this asynchronous federated aggregation; is the staleness of the ith satellite node, is a constant that controls the decay rate of staleness, is the gradient of the policy network parameters of the i-th satellite node and the gradient of the value network parameters the time that has elapsed since the calculation; for and The cosine similarity of for and The cosine similarity of is the gradient of the global policy network parameters, is the gradient of the global value network parameters, is the Euclidean norm of the gradient of the local policy network parameters, is the Euclidean norm of the gradient of the global policy network parameters, is the Euclidean norm of the gradient of the local value network parameters, is the Euclidean norm of the gradient of the global value network parameters.

[0064] Furthermore, the final sign that the local model gradient of each satellite can participate in asynchronous federated aggregation can be the comprehensive quality index value of the i-th satellite node: Exceeds a certain preset quality threshold , where asynchronous federation aggregation is as follows: (8) in, is the gradient of the new global policy network parameters, is the gradient of the new global value network parameters, The weight of the asynchronous federation aggregation.

[0065] In order to help those skilled in the art to more clearly and intuitively understand the process of asynchronous federation aggregation in the embodiment of the present application, Figure 3 Provide detailed explanation.

[0066] like Figure 3 As shown, the asynchronous federated aggregation process includes the following steps: S301, receiving satellite local model gradient information.

[0067] S302, performing quality screening on the local model gradient.

[0068] S303, determine whether K gradients that meet the quality standards are received, if yes, execute S304, otherwise, execute S301.

[0069] S304, calculating new global model parameters after aggregation.

[0070] S305, calculating a new global model gradient.

[0071] In step S103, the similarity between the local model gradient of each satellite and the new global model gradient is calculated, and when the similarity is greater than a preset threshold, it is determined that the new global model is in a preset convergence state.

[0072] The preset threshold may be a threshold pre-set by a person skilled in the art and is not specifically limited here.

[0073] Specifically, the degree of convergence of the model can be evaluated by using the similarity between the gradient of the new global model parameters and the gradient of the local model parameters. The specific judgment is as follows: (10) in, and is the similarity in formula (7), for and The cosine similarity of for and The cosine similarity of is the similarity threshold, that is, the preset threshold, which is usually set between When the average similarity is higher than the preset threshold, it is determined that the new global model is in a preset convergence state.

[0074] Furthermore, in some embodiments, after calculating the similarity between the local model gradient of each satellite and the new global model gradient, it also includes: if the similarity is less than or equal to a preset threshold, broadcasting the new global model gradient to each satellite; and performing gradient updating on the local model parameters of each satellite according to the new global model gradient.

[0075] Specifically, if the average similarity calculated according to formula (10) is less than or equal to the preset threshold, the central node broadcasts the updated new global model to all satellites, and each satellite receives the updated new global model gradient and After that, the local model parameters are updated with gradients.

[0076] (9) in, are the local policy network parameters updated based on the new global model gradient, are the local value network parameters updated based on the new global model gradient, is the learning rate, which is consistent with formula (5) and is used to control the update step size.

[0077] Furthermore, in some embodiments, after determining that the new global model is in a preset convergence state, it also includes: calculating the inversion success rate of the new global model; verifying whether the new global model satisfies the preset training termination condition according to the inversion success rate; if the new global model satisfies the preset training termination condition, terminating the training of the new global model, otherwise, executing the step of broadcasting the new global model gradient to each satellite.

[0078] The preset training termination condition may be that the safety performance of the new global model no longer improves.

[0079] Specifically, the inversion success rate refers to the success rate of an attacker inferring the original user data from a machine learning model through a model inversion attack. The inversion attack is simulated to crack the transmitted user information, and the inversion success rate is used to evaluate whether the optimization method has reached convergence. The calculation formula for the inversion success rate is as follows: (11) in, is the inversion success rate at time step t, is the original data set that the attacker successfully recovers through the inversion attack at time step t. is the set of all original user data involved in the system at time step t, It is the ratio of the number of successfully inverted data entries to the total number of data entries.

[0080] Furthermore, the inversion success rate is used to determine whether the safety performance of the new global model is improved. The conditions for determining whether the safety performance is no longer improved in the embodiment of the present application are: (12) in, For the time step The inversion success rate is For the time step The inversion success rate is is the current time step, is the set inspection period, which means that if the safety performance has not changed in the past n consecutive periods, it is considered that the safety performance of the new global model has reached its peak; It is a small constant used to determine whether the change in safety performance is significant. It can usually be set to a very small value, such as 0.01, and is not specifically limited here.

[0081] In order to facilitate those skilled in the art to more clearly and intuitively understand a satellite beam hopping method based on federated learning in an embodiment of the present application, the following is combined with Figure 4 Provide detailed explanation.

[0082] like Figure 4As shown, the asynchronous federated aggregation process includes the following steps: S401, start.

[0083] S402, initializing model parameters.

[0084] S403, satellite local model training.

[0085] S404, calculate and upload the local gradient.

[0086] S405, performing asynchronous federation aggregation.

[0087] S406, determining whether the similarity between the local model parameters of each satellite and the new global model parameters reaches a preset threshold, if yes, executing S409, otherwise, executing S407.

[0088] S407, broadcast the new global gradient.

[0089] S408, all satellites are updated according to the new global gradient.

[0090] S409, determining whether the inversion success rate is improved, if yes, executing S407, otherwise, executing S410.

[0091] S410, determining that the new global model has converged and terminating the training.

[0092] S411, end.

[0093] Therefore, the present invention adopts the method of federated learning to allow each satellite to train the beam hopping strategy on its onboard computing unit, only transmit the gradient calculated by the model parameters at the appropriate time, and use the computing center node to complete the optimization of the overall model to avoid the transmission of the original data. At the same time, the mechanism of K-asynchronous federated learning is adopted to avoid the central computing node from processing too much data at one time, reducing the complexity of data processing.

[0094] According to a satellite beam-hopping method based on federated learning in an embodiment of the present application, the quality of the local model gradient of each satellite is estimated, and multiple gradients that meet the preset quality standards are asynchronously federated to obtain a new global model gradient; the similarity between the local model gradient of each satellite and the new global model gradient is calculated, and when the similarity is greater than a preset threshold, the new global model is determined to be in a preset convergence state. In this way, the problems of high user data security risk and high system synchronization communication complexity in the existing beam-hopping technology are solved, and the system synchronization communication complexity is reduced while ensuring user data security, and the computing efficiency is improved, ensuring the effective use of satellite communication system resources and long-term stable operation.

[0095] Next, a satellite beam-hopping device based on federated learning proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0096] Figure 5 It is a block diagram of a satellite beam hopping device based on federated learning according to an embodiment of the present application.

[0097] like Figure 5 As shown, the satellite beam hopping device 10 based on federated learning includes: an acquisition module 100, an aggregation module 200 and an optimization module 300.

[0098] Among them, the acquisition module 100 is used to obtain the local model gradient of each satellite; the aggregation module 200 is used to estimate the quality of the local model gradient of each satellite, and asynchronously federate the multiple gradients that meet the preset quality standards to obtain a new global model gradient; the optimization module 300 is used to calculate the similarity between the local model gradient of each satellite and the new global model gradient, and when the similarity is greater than a preset threshold, it is determined that the new global model is in a preset convergence state.

[0099] Furthermore, in some embodiments, the acquisition module 100 is used to: acquire the global model parameters of the central node; initialize the global model parameters, and locally perform beam hopping strategy training on each satellite based on the initialized global model parameters, and obtain the local model gradient of each satellite after the training is completed.

[0100] Further, in some embodiments, the acquisition module 100 is used to: obtain the current system state information of each satellite within a preset time step, the current system state information including at least one of user demand, interference situation, beam overlap area and channel state; determine the target execution action corresponding to each satellite based on the current system state information of each satellite and the current policy network, and update the current system state information of each satellite after each satellite executes the corresponding target execution action, and calculate the reward obtained by each satellite; record a four-tuple consisting of the current system state information, the target execution action, the reward and the updated current system state information, and calculate the advantage function based on the four-tuple; determine the update amplitude of the policy network and / or the parameters of the policy network according to the value of the advantage function in the new update cycle, and update the parameters of the value network based on the minimization of the mean square error loss function to obtain the updated value network parameters.

[0101] Further, in some embodiments, the aggregation module 200 is used to: calculate the comprehensive quality index value of the local model gradient of each satellite based on a preset quality index calculation formula; if the comprehensive quality index value is greater than the preset index value, determine that the local model gradient meets the preset quality standard condition, and obtain the number of local model gradients that meet the preset quality standard condition; if the number of local model gradients that meet the preset quality standard condition reaches a preset number, perform asynchronous federation aggregation on multiple gradients that meet the preset quality standard condition based on a preset aggregation formula; wherein the preset quality index calculation formula is: ; in, is the comprehensive quality index value of the i-th satellite node, is the time step of this asynchronous federated aggregation; is the staleness of the ith satellite node, is a constant that controls the decay rate of staleness, is the gradient of the policy network parameters of the i-th satellite node and the gradient of the value network parameters the time that has elapsed since the calculation; for and The cosine similarity of for and The cosine similarity of is the gradient of the global policy network parameters, is the gradient of the global value network parameters, is the Euclidean norm of the gradient of the local policy network parameters, is the Euclidean norm of the gradient of the global policy network parameters, is the Euclidean norm of the gradient of the local value network parameters, is the Euclidean norm of the gradient of the global value network parameters.

[0102] Furthermore, in some embodiments, after calculating the similarity between the local model gradient of each satellite and the new global model gradient, the optimization module 300 is also used to: if the similarity is less than or equal to a preset threshold, broadcast the new global model gradient to each satellite; and perform gradient update on the local model parameters of each satellite according to the new global model gradient.

[0103] Furthermore, in some embodiments, after determining that the new global model is in a preset convergence state, the optimization module 300 is also used to: calculate the inversion success rate of the new global model; verify whether the new global model satisfies the preset training termination condition according to the inversion success rate; if the new global model satisfies the preset training termination condition, terminate the training of the new global model, otherwise, execute the step of broadcasting the new global model gradient to each satellite.

[0104] It should be noted that the above explanation of an embodiment of a satellite beam hopping method based on federated learning is also applicable to a satellite beam hopping device based on federated learning in this embodiment, which will not be repeated here.

[0105] According to a satellite beam-hopping device based on federated learning in an embodiment of the present application, the quality of the local model gradient of each satellite is estimated, and multiple gradients that meet the preset quality standards are asynchronously federated and aggregated to obtain a new global model gradient; the similarity between the local model gradient of each satellite and the new global model gradient is calculated, and when the similarity is greater than a preset threshold, the new global model is determined to be in a preset convergence state. In this way, the problems of high user data security risk and high system synchronization communication complexity in the existing beam-hopping technology are solved, and the system synchronization communication complexity is reduced while ensuring user data security, and the computing efficiency is improved, ensuring the effective use of satellite communication system resources and long-term stable operation.

[0106] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0107] When the processor 602 executes the program, a satellite beam hopping method based on federated learning provided in the above embodiment is implemented.

[0108] Furthermore, the electronic device further comprises: The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0109] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0110] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0111] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0112] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0113] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0114] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned satellite beam hopping method based on federated learning.

[0115] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0116] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0117] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A satellite beam hopping method based on federated learning, characterized in that: The following steps are involved: Get the local model gradient for each satellite; Performing quality estimation on the local model gradient of each satellite, and performing asynchronous federation aggregation on multiple gradients that meet preset quality standards to obtain a new global model gradient; The similarity between the local model gradient of each satellite and the new global model gradient is calculated, and when the similarity is greater than a preset threshold, it is determined that the new global model is in a preset convergence state.

2. The satellite beam hopping method based on federated learning according to claim 1, characterized in that: The obtaining of the local model gradient of each satellite comprises: Get the global model parameters of the central node; The global model parameters are initialized, and beam hopping strategy training is performed locally on each satellite based on the initialized global model parameters, and the local model gradient of each satellite is obtained after the training is completed.

3. The satellite beam hopping method based on federated learning according to claim 2, characterized in that: The locally performing beam hopping strategy training on each satellite based on the initialized global model includes: Acquire current system status information of each satellite within a preset time step, wherein the current system status information includes at least one of user demand, interference situation, beam overlap area and channel status; Based on the current system state information of each satellite and the current policy network, determine the target execution action corresponding to each satellite, update the current system state information of each satellite after each satellite executes the corresponding target execution action, and calculate the reward obtained by each satellite; Recording a four-tuple consisting of the current system state information, the target execution action, the reward, and the updated current system state information, and calculating the advantage function based on the four-tuple; According to the value of the advantage function in the new update cycle, the update amplitude of the policy network and / or the parameters of the policy network are determined, and the parameters of the value network are updated based on minimizing the mean square error loss function to obtain updated value network parameters.

4. The satellite beam hopping method based on federated learning according to claim 1, characterized in that: The method of estimating the quality of the local model gradient of each satellite and performing asynchronous federated aggregation on multiple gradients that meet preset quality standards includes: Calculate the comprehensive quality index value of the local model gradient of each satellite based on a preset quality index calculation formula; If the comprehensive quality index value is greater than the preset index value, it is determined that the local model gradient meets the preset quality standard condition, and the number of local model gradients that meet the preset quality standard condition is obtained; If the number of local model gradients that meet the preset quality standard condition reaches a preset number, asynchronous federation aggregation is performed on multiple gradients that meet the preset quality standard condition based on a preset aggregation formula; The preset quality index calculation formula is: ； in, is the comprehensive quality index value of the i-th satellite node, is the time step of this asynchronous federated aggregation; is the staleness of the ith satellite node, is a constant that controls the decay rate of staleness, is the gradient of the policy network parameters of the i-th satellite node and the gradient of the value network parameters the time that has elapsed since the calculation; for and The cosine similarity of for and The cosine similarity of is the gradient of the global policy network parameters, is the gradient of the global value network parameters, is the Euclidean norm of the gradient of the local policy network parameters, is the Euclidean norm of the gradient of the global policy network parameters, is the Euclidean norm of the gradient of the local value network parameters, is the Euclidean norm of the gradient of the global value network parameters.

5. The satellite beam hopping method based on federated learning according to claim 1, characterized in that: After calculating the similarity between the local model gradient of each satellite and the new global model gradient, the method further includes: If the similarity is less than or equal to the preset threshold, broadcasting the new global model gradient to each satellite; The local model parameters of each satellite are gradient updated according to the new global model gradient.

6. The satellite beam hopping method based on federated learning according to claim 1, characterized in that: After determining that the new global model is in a preset convergence state, the method further includes: Calculating the inversion success rate of the new global model; Verifying whether the new global model meets a preset training termination condition according to the inversion success rate; If the new global model satisfies the preset training termination condition, the training of the new global model is terminated; otherwise, the step of broadcasting the new global model gradient to each satellite is performed.

7. A satellite beam hopping device based on federated learning, characterized in that: include: An acquisition module, used to obtain the local model gradient of each satellite; An aggregation module is used to estimate the quality of the local model gradient of each satellite, and asynchronously aggregate multiple gradients that meet preset quality standards to obtain a new global model gradient; The optimization module is used to calculate the similarity between the local model gradient of each satellite and the new global model gradient, and when the similarity is greater than a preset threshold, determine that the new global model is in a preset convergence state.

8. The satellite beam hopping device based on federated learning according to claim 7, characterized in that: The acquisition module is used to: Get the global model parameters of the central node; The global model parameters are initialized, and beam hopping strategy training is performed locally on each satellite based on the initialized global model parameters, and the local model gradient of each satellite is obtained after the training is completed.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a satellite beam hopping method based on federated learning as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a satellite beam hopping method based on federated learning as described in any one of claims 1-6.

Images