Satellite-ground convergence network access and caching strategy joint determination method and related equipment

Abstract

The invention provides a satellite-ground convergence network access and cache strategy joint determination method and related equipment, and comprehensively considers the tight coupling relationship between cache resource allocation and satellite-ground connection. Processing through a single agent reinforcement learning algorithm based on the connection state of the target low earth orbit satellite and the target ground station, the user request state, the cache state of the target ground station and the cache state of the target low earth orbit satellite; the determined target connection strategy and target cache strategy can make full use of cache resources in a multi-heterogeneous network, network congestion caused by repeated transmission of hot request content is relieved, and the number of times of interaction between a satellite and a ground station and updating of a cache space can be reduced. Therefore, efficient data transmission between the satellite and the ground can be realized according to the target connection strategy and the target caching strategy.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method for jointly determining satellite-ground integrated network access and cache strategy and related equipment. Background Art

[0002] Traditional ground wireless communication networks have limited coverage and weak disaster resistance, and can no longer meet the growing communication business needs of the power industry. Satellite networks have advantages such as global coverage and strong disaster resistance, which can effectively solve the shortcomings of ground wireless communication networks. Therefore, the satellite-ground fusion network that deeply integrates satellite networks and ground networks is receiving more and more attention. Among them, the ground network achieves high-capacity coverage of key areas on the ground through base stations. The satellite network achieves coverage of remote areas such as the ground, ocean and sky through a three-dimensional network of low-orbit satellites (LEO), medium-orbit satellites (MEO) and geostationary orbits (GEO). The satellite-ground fusion network effectively expands the coverage of the ground network. It provides a more reliable and efficient communication foundation for the increasingly diversified communication application needs in the fields of power, transportation, emergency, etc., and has broad development potential and application prospects.

[0003] However, the reasonable allocation of cache resources and efficient access methods on satellites and ground nodes are the prerequisites for achieving efficient satellite-ground collaboration. The unique characteristics of satellite-ground fusion networks, such as multi-heterogeneity, dynamic, time-varying, and resource-constrained, determine the particularity of their cache strategy design and access methods, which are key factors affecting the performance of satellite-ground fusion networks. Therefore, how to provide effective cache and access methods for satellite-ground fusion networks is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a method for jointly determining satellite-ground integrated network access and cache strategy and related equipment to solve the above technical problems.

[0005] Based on the above purpose, the first aspect of the present application provides a method for jointly determining access and caching strategies of a satellite-ground fusion network, wherein the satellite-ground fusion network includes multiple low-orbit satellites and multiple ground stations, and the method includes:

[0006] Taking each of the plurality of low-orbit satellites as a target low-orbit satellite, and taking each of the plurality of ground stations as a target ground station;

[0007] Acquire the connection status between the target low-orbit satellite and the target ground station at a target time;

[0008] Acquire user request content of the power user terminal, a cache status of the target low-orbit satellite at a target time, and a cache status of the target ground station at a target time, and determine a user request status based on the user request content;

[0009] Determine the cache content of the low-orbit satellite by a low-orbit satellite cache algorithm based on the user request content and the cache state of the target low-orbit satellite, and determine the cache content of the ground station by a ground station cache algorithm based on the user request content and the cache state of the target ground station;

[0010] Based on the connection status, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite, a single-agent reinforcement learning algorithm is used to process, and a target connection strategy of the satellite-ground fusion network is determined from the connection status of the target low-orbit satellite and the target ground station, and a target cache strategy of the satellite-ground fusion network is determined from the cache content of the low-orbit satellite and the cache content of the ground station.

[0011] Based on the same inventive concept, the second aspect of the present application provides a device for jointly determining access and caching strategies of a satellite-ground fusion network, characterized in that the satellite-ground fusion network includes multiple low-orbit satellites and multiple ground stations, and the device includes:

[0012] a target determination module, configured to use each of the plurality of low-orbit satellites as a target low-orbit satellite, and use each of the plurality of ground stations as a target ground station;

[0013] A connection status acquisition module is configured to acquire the connection status between the target low-orbit satellite and the target ground station at a target time;

[0014] a request status determination module, configured to obtain user request content of the power user terminal, a cache status of the target low-orbit satellite at a target time, and a cache status of the target ground station at a target time, and determine a user request status based on the user request content;

[0015] a cache content determination module, configured to determine the cache content of the low-orbit satellite through a low-orbit satellite cache algorithm based on the user request content and the cache state of the target low-orbit satellite, and to determine the cache content of the ground station through a ground station cache algorithm based on the user request content and the cache state of the target ground station;

[0016] The strategy determination module is configured to perform processing through a single-agent reinforcement learning algorithm based on the connection status, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite, determine the target connection strategy of the satellite-ground fusion network from the connection status of the target low-orbit satellite and the target ground station, and determine the target cache strategy of the satellite-ground fusion network from the cache content of the low-orbit satellite and the cache content of the ground station.

[0017] Based on the same inventive concept, the third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0018] From the above, it can be seen that the method and related equipment for jointly determining the satellite-ground integrated network access and cache strategy provided by the present application comprehensively consider the close coupling relationship between cache resource allocation and satellite-ground connection, and process the connection status between the target low-orbit satellite and the target ground station, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite through a single-agent reinforcement learning algorithm, so that the determined target connection strategy and target cache strategy can make full use of the cache resources in multiple heterogeneous networks, alleviate the network congestion caused by repeated transmission of popular request content, and reduce the number of interactions between the satellite and the ground station and the update of the cache space, so that efficient data transmission between the satellite and the ground can be achieved according to the target connection strategy and target cache strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a flow chart of a method for jointly determining satellite-ground fusion network access and cache strategy according to an embodiment of the present application;

[0021] Figure 2A A schematic diagram of a joint determination process of satellite-ground fusion network access and cache strategy according to an embodiment of the present application;

[0022] Figure 2B A schematic diagram of the overall construction process of the satellite-ground integrated network in the power grid scenario of an embodiment of the present application;

[0023] Figure 3 This is a structural block diagram of a device for jointly determining satellite-ground integrated network access and cache strategy according to an embodiment of the present application;

[0024] Figure 4 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] It is understandable that before using the technical solutions of each embodiment of the present application, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0028] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present application according to the prompt message.

[0029] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0030] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of this application. Other methods that meet relevant laws and regulations may also be applied to the implementation method of this application.

[0031] Traditional ground wireless communication networks have limited coverage and weak disaster resistance, and can no longer meet the growing communication business needs of the power industry. Satellite networks have advantages such as global coverage and strong disaster resistance, which can effectively solve the shortcomings of ground wireless communication networks. Therefore, the satellite-ground fusion network that deeply integrates satellite networks and ground networks is receiving more and more attention. Among them, the ground network achieves high-capacity coverage of key areas on the ground through base stations. The satellite network achieves coverage of remote areas such as the ground, ocean and sky through a three-dimensional network of low-orbit satellites (LEO), medium-orbit satellites (MEO) and geostationary orbits (GEO). The satellite-ground fusion network effectively expands the coverage of the ground network. It provides a more reliable and efficient communication foundation for the increasingly diversified communication application needs in the fields of power, transportation, emergency, etc., and has broad development potential and application prospects.

[0032] However, the reasonable allocation of cache resources and efficient access methods on satellites and ground nodes are the prerequisites for achieving efficient satellite-ground collaboration. The unique multi-heterogeneous, dynamic, time-varying, and resource-constrained characteristics of satellite-ground fusion networks determine the particularity of their cache strategy design and access methods. There are still many problems in related technologies that need to be further studied.

[0033] The existing access strategy methods do not consider the satellite cache status, the rational use of cache resources in the satellite-ground fusion network, the design of collaborative cache strategies between nodes at all levels, and the problem of alleviating repeated transmission of hot content. These aspects are key issues that need to be solved in the satellite-ground fusion network.

[0034] An embodiment of the present application provides a method for jointly determining satellite-ground integrated network access and cache strategy, which comprehensively considers the close coupling relationship between cache resource allocation and satellite-ground connection, and processes the connection status between the target low-orbit satellite and the target ground station, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite through a single-agent reinforcement learning algorithm, so that the determined target connection strategy and target cache strategy can make full use of the cache resources in multiple heterogeneous networks, alleviate network congestion caused by repeated transmission of popular requested content, and reduce the number of interactions between the satellite and the ground station and the update of the cache space, so that efficient data transmission between the satellite and the ground can be achieved according to the target connection strategy and target cache strategy.

[0035] like Figure 1 As shown, the satellite-ground fusion network includes multiple low-orbit satellites and multiple ground stations. The method of this embodiment includes:

[0036] Step 101: taking each of the plurality of low-orbit satellites as a target low-orbit satellite, and taking each of the plurality of ground stations as a target ground station.

[0037] In this step, each of the multiple low-orbit satellites is treated as an independent target or object during processing or analysis. This processing method can flexibly respond to different satellite states, requirements and functions.

[0038] Likewise, each of the multiple ground stations is treated as an independent target or object during processing or communication. This processing allows the system to control, monitor or configure each ground station individually to adapt to different network requirements, communication conditions or mission requirements.

[0039] Step 102, obtaining the connection status between the target low-orbit satellite and the target ground station at the target time.

[0040] In this step, at the specified time (target time), it is necessary to obtain or monitor the communication connection status between the target low-orbit satellite and the target ground station. This is usually achieved through satellite measurement and control systems, ground station monitoring systems, etc., to ensure the normal operation of the satellite communication system and the reliable transmission of data.

[0041] The target time is a specific time point or time period used to determine the specific time range to be queried or monitored next. In satellite communication systems, time is a very critical parameter because the position of the satellite (relative to the earth and other communication objects) changes over time, which directly affects the feasibility and quality of communication.

[0042] The connection status refers to whether the communication link between the target low-orbit satellite and the target ground station is established, stable, and of what quality at a certain moment. The quality of the connection status directly affects the transmission efficiency and accuracy of data, and is a key concern in the operation and maintenance of satellite communication systems.

[0043] Step 103, obtaining the user request content of the power user terminal, the cache status of the target low-orbit satellite at the target time, and the cache status of the target ground station at the target time, and determining the user request status based on the user request content.

[0044] In this step, firstly, requests are received from the power user terminal, which may include data upload (such as meter readings, equipment status, etc.), data download (such as software updates, control instructions, etc.) or query requests for specific information.

[0045] The content of user requests is crucial for subsequent steps because it determines how the system needs to respond to those requests.

[0046] Next, we need to query the cache status of the target low-orbit satellite at a specific time (i.e., the "target moment"). The cache status refers to the current situation of the data and resources stored on the satellite, including whether it contains the data requested by the user, the cache capacity, the remaining space, etc. Knowing this information helps decide whether to satisfy the request directly through the satellite or whether to obtain data from the ground station or other sources.

[0047] Similar to querying the satellite cache status, it is also necessary to understand the cache status of the target ground station at the target time. Ground stations are key nodes connecting satellites and ground networks in satellite communication systems, and they usually also have certain data storage capabilities. Understanding the cache status of ground stations can help determine whether user requests can be met faster through the ground network, especially when satellite resources are tight or data latency is high.

[0048] After collecting the user request content, the cache status of the target low-orbit satellite, and the cache status of the target ground station, the status of the user request will be comprehensively judged based on this information.

[0049] This may include deciding whether to transmit data directly via a satellite or ground station, whether data needs to be obtained from another source, or whether the request cannot be fulfilled temporarily due to some limitation (such as insufficient cache, satellite or ground station unavailability, etc.).

[0050] According to the request status, corresponding processing strategies can be further formulated, such as immediate response, delayed response, request redirection, etc., to ensure that the requests of the power user terminals are processed efficiently and reliably.

[0051] Step 104, determining the cache content of the low-orbit satellite through a low-orbit satellite cache algorithm based on the user request content and the cache status of the target low-orbit satellite, and determining the cache content of the ground station through a ground station cache algorithm based on the user request content and the cache status of the target ground station.

[0052] In this step, the content that should be cached on the LEO satellite is determined based on the user's requested content, the satellite's cache status, and other possible factors (such as the popularity of the content, the remaining storage space of the satellite, the size and type of the content, etc.). Such a LEO satellite cache algorithm aims to maximize the cache benefit, that is, to respond to user requests as quickly as possible while avoiding frequent data transmission from ground stations or other sources.

[0053] Similar to low-orbit satellites, ground stations also have certain caching capabilities for storing and distributing content to satellites and users. They decide which content should be cached on the ground station based on user requests for content, the cache status of the ground station, and other related factors (such as the geographical distribution of content, bandwidth limitations of the ground station, frequency of content updates, etc.). The ground station caching algorithm is designed to improve the efficiency of content distribution, reduce network congestion, and optimize user experience.

[0054] By continuously analyzing user request content, the cache status of low-orbit satellites and ground stations, and using cache algorithms to intelligently manage cache content, the performance and service quality of the entire satellite communication network can be improved.

[0055] Step 105, based on the connection status, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite, a single-agent reinforcement learning algorithm is used to process, and a target connection strategy of the satellite-ground fusion network is determined from the connection status of the target low-orbit satellite and the target ground station, and a target cache strategy of the satellite-ground fusion network is determined from the cache content of the low-orbit satellite and the cache content of the ground station.

[0056] In this step, the single-agent reinforcement learning algorithm allows the agent (here, the satellite-ground fusion network system) to learn how to make the best decision by interacting with the environment (i.e., the various states mentioned above). Single-agent means that there is only one learning subject responsible for the entire decision-making process. The reinforcement learning algorithm learns by trial and error, that is, trying different actions (such as choosing different connection strategies or caching strategies) and updating its decision-making strategy based on the results of the actions (such as data transmission efficiency, user satisfaction, etc.).

[0057] Among them, the target connection strategy guides how to effectively utilize the connection resources between satellites and ground stations to meet the communication needs of users.

[0058] The target cache strategy guides how to update and manage the cache contents of satellites and ground stations to improve the efficiency of data transmission and user satisfaction.

[0059] Using single-agent reinforcement learning algorithms to optimize the connection strategy and caching strategy in the satellite-ground fusion network can achieve efficient utilization of limited communication resources and meet users' communication needs.

[0060] Through the above scheme, the close coupling relationship between cache resource allocation and satellite-ground connection is comprehensively considered. The connection status between the target low-orbit satellite and the target ground station, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite are processed through a single-agent reinforcement learning algorithm, so that the determined target connection strategy and target cache strategy can make full use of the cache resources in multiple heterogeneous networks, alleviate network congestion caused by repeated transmission of popular request content, and reduce the number of interactions between the satellite and the ground station and the update of the cache space. Therefore, according to the target connection strategy and target cache strategy, efficient data transmission between the satellite and the ground can be achieved.

[0061] In some embodiments, in step 103, determining the user request status based on the user request content includes:

[0062] Step A1, obtaining the content d requested by the user s The corresponding popularity s, and the user request content d s The corresponding request probability q s .

[0063] Step A2, based on the request probability q s And the popularity s is determined by the following formula:

[0064] Q = {q s |s∈(1,2,…,d,…,D)}.

[0065] In the above scheme, popularity refers to the popularity or prevalence of the content requested by users in the current environment or time range. It may be calculated based on a variety of factors, such as the number of historical requests, the number of mentions in user feedback, etc. Popular content usually means that a large number of users are or will request it.

[0066] Request Probability refers to the probability that a user actually sends a request for the content. This probability may be calculated based on a variety of factors such as the user's personal preferences, historical behavior, current context (such as time, location, device, etc.). Content with a high request probability may mean that the user has a high interest or demand for the content.

[0067] By combining popularity and request probability, the current status of user requests can be accurately determined.

[0068] In some embodiments, in step 104, determining the cache content of the low-orbit satellite by using a low-orbit satellite cache algorithm based on the user request content and the cache state of the target low-orbit satellite includes:

[0069] Based on the user request content d s and the cache status of the target low-orbit satellite The cache content of the low-orbit satellite is determined by the following formula

[0070]

[0071] Among them, D represents the number of user requested contents, t represents the order of low-orbit satellites, T represents the number of low-orbit satellites, and u represents the order of target time.

[0072] In the above scheme, the number of user request contents refers to the total number of requests or the number of requests of a specific type sent by users to the low-orbit satellite network within a certain period of time. This number changes dynamically and reflects the user's demand for data.

[0073] In a network of multiple low-orbit satellites, each satellite has a unique identifier or order, which is used to distinguish different satellites in the algorithm in order to allocate cache content to each satellite.

[0074] In addition, knowing the total number helps the algorithm optimize cache allocation globally to ensure that resources are used reasonably.

[0075] The order of target moments indicates the time points at which the algorithm runs or the data is updated. In a dynamically changing low-orbit satellite network, the user request patterns and satellite positions at different time points will be different, so the caching strategy needs to be able to adjust over time.

[0076] Based on the user's requested content, the satellite's current cache status, the collaboration strategy between satellites, and time factors, it is possible to quickly and dynamically determine what content each satellite should cache.

[0077] In some embodiments, in step 104, determining the cache content of the ground station by using a ground station cache algorithm based on the user request content and the cache status of the target ground station includes:

[0078] Based on the user request content d s and the cache status of the target ground station The cache content of the ground station is determined by the following formula

[0079]

[0080] Where D represents the number of user requested contents, c represents the order of ground stations, ε represents the number of ground stations, and u represents the order of target time.

[0081] In the above scheme, the number of user request contents refers to the total number of requests or the number of requests of a specific type sent by users to the low-orbit satellite network within a certain period of time. This number changes dynamically and reflects the user's demand for data.

[0082] The ground station order indicates an identifier or index of a ground station, and is used to distinguish different ground stations.

[0083] The order of target moments indicates the time points at which the algorithm runs or data is updated. In a ground station with dynamically changing access, the user request patterns and satellite positions at different time points will be different, so the cache strategy of the ground station needs to be able to adjust over time.

[0084] Deciding what content each ground station should cache based on the content requested by the user and the current cache status of the ground station can improve the efficiency of the content distribution network and reduce latency and bandwidth consumption.

[0085] In some embodiments, step 105 includes:

[0086] Step B1: Set the connection status H u , the user request status req u , the cache status of the target ground station and the cache status of the target low-orbit satellite As the state space T u , where the state space T u Specifically:

[0087]

[0088] Among them, u represents the order of the target time, U represents the total time, t represents the order of the low-orbit satellite, T represents the number of low-orbit satellites, and c represents the order of the ground station. Indicates the number of ground stations.

[0089] Step B2: The connection status between the target low-orbit satellite and the target ground station And whether the cache content of the low-orbit satellite caches the cache content of the ground station As the action space b, wherein the action space b is specifically:

[0090]

[0091] Among them, u represents the order of the target time, U represents the total time, t represents the order of the low-orbit satellite, T represents the number of low-orbit satellites, and c represents the order of the ground station. represents the number of ground stations, d represents the user requested content, and D represents the number of user requested content.

[0092] Step B3, from the state space T u Select the current state from the action space b, select the currently executed action from the action space b based on the current state, and obtain the cache delay T of the target ground station c (u), the cache delay T of the target low-orbit satellite t (u) and the transmission delay T of the user request content p (u).

[0093] Step B4, obtaining the total delay T of the satellite-ground fusion network no-cache (u), based on the buffer delay T of the target ground station c (u), the cache delay T of the target low-orbit satellite t (u), transmission delay T for the user request content p (u) and the transmission delay T of the user requested content p (u) The reward function s(u) is obtained by the following formula, and the reward function s(u) is specifically:

[0094]

[0095] Step B5, based on the reward function, determines the reward value corresponding to the currently executed action, and selects the next state of the current state from the state space, stores the current state, the currently executed action, the reward value and the next state as experience tuples in a pre-constructed experience replay pool, and randomly extracts a batch of experience tuples from the experience replay pool.

[0096] Step B6, determining a target value function according to the extracted experience tuple, and determining an update comment network function based on the target value function.

[0097] Step B7, determining an update execution network function based on the comment network function, continuously adjusting the update comment network function based on the update execution network function until the value of the update comment network function is minimized, using the connection strategy corresponding to the minimized value of the update comment network function as the target connection strategy, and using the cache strategy corresponding to the minimized value of the update comment network function as the target cache strategy.

[0098] In the above scheme, the state space is the set of all possible states in the algorithm. In this scenario, the state space contains four main parts: connection state, user request state, cache state of the target ground station, and cache state of the target low-orbit satellite.

[0099] Connection status: describes the connection status between the target low-orbit satellite and the target ground station at each point in time.

[0100] User request status: Indicates the user's request for content, but it is not directly listed as part of the status in this description and may be implicit in the connection or cache status.

[0101] The cache status of the target ground station indicates the cache content of the ground station at each time point.

[0102] The cache status of the target low-orbit satellite indicates the cache content status of the low-orbit satellite at each time point.

[0103] The action space is the set of all possible actions in the algorithm. In this scenario, the action space includes: The connect action determines whether the target low-orbit satellite establishes a connection with the target ground station.

[0104] The cache action determines whether the LEO satellite caches the content of the ground station.

[0105] The algorithm simulates the execution of actions and calculates the cache delay of the target ground station, the cache delay of the target low-orbit satellite, and the transmission delay of the user's requested content. Then, based on these delays, the reward function is calculated, where the reward function aims to minimize the total delay of the network, so it may be a weighted sum of negative delays, or a function inversely proportional to the delay.

[0106] The experience replay pool is used to store experience tuples (including state, action, reward, and next state) so that samples can be randomly drawn during training to improve learning efficiency. A batch of experience tuples are randomly drawn from the experience replay pool. These experience tuples are used to update the target value function (Critic Network) (usually a function approximator based on a neural network) to evaluate the quality of the action.

[0107] Use the comment network function to update the execution network function (ActorNetwork), which is used to select actions.

[0108] Repeat the above process until the value of the comment network function is minimized. At this time, the corresponding connection strategy is the target connection strategy, and the cache strategy is the target cache strategy. The target connection strategy and target cache strategy can be quickly determined through the single-agent reinforcement learning algorithm.

[0109] In some embodiments, in step B6, determining the target value function according to the extracted experience tuple includes:

[0110] Step C1, determining the popularity order s of the current user request content based on the extracted experience tuple j and the expected value of the reward obtained from the current state after taking the action

[0111] Step C2: based on the popularity order of the current user request content j , the expected value of the reward obtained in the current state after taking the action and the preset constant parameter λ are used to determine the target value function z by the following formula k :

[0112]

[0113] In the above scheme, the popularity of the content requested by the current user is analyzed based on the extracted experience tuples. This usually involves calculating some form of "popularity index", such as click-through rate, viewing time, conversion rate, etc. These indicators can help the system understand which content is more popular with users. By sorting these indicators, the system can get the popularity order of the content requested by the current user.

[0114] Next, estimate the expected value of the reward that may be obtained in the current state (i.e., the context of the current user request content) after taking a certain action. The "reward" here can be any quantifiable positive result. By analyzing the user behavior and its corresponding rewards in the historical experience data, we can predict the rewards that may be obtained by taking specific actions in similar situations in the future.

[0115] Finally, using the above popularity order and reward expectation, combined with a preset constant parameter (this parameter may be used to adjust the influence of different factors on the objective value function), Determine the objective value function. The purpose of the objective value function is to quantify the value of taking a specific action in a given state, so that the action most likely to result in a high reward can be selected.

[0116] In some embodiments, in step B6, determining to update the review network function based on the target value function includes:

[0117] Obtain the number of low-orbit satellites T in the satellite-ground fusion network, based on the objective value function z k , the expected value of the reward obtained in the current state after taking the action R′ j (t, b1, L, b O ) and the number of low-orbit satellites T are determined by the following formula to update the review network function M(θ j ):

[0118]

[0119] In the above scheme, the number of low-orbit satellites T in the satellite-ground fusion network is comprehensively considered, based on the objective value function Z k , the expected value of the reward obtained in the current state after taking the action R′ j (t, b1, L, bO ) and the number of low-orbit satellites T can more accurately determine the update review network function M(θ j ).

[0120] In some embodiments, in step B7, determining to update the execution network function based on the comment network function includes:

[0121] Step D1, based on the update comment network function M(θ j ) Determine the number of low-orbit satellites T in the satellite-ground fusion network and the expected value of the reward obtained after taking the action

[0122] Step D2: based on the number of low-orbit satellites T in the satellite-ground fusion network and the expected value of the reward obtained in the current state after taking the action and the preset Jacobian matrix The update execution network function is determined by the following formula

[0123]

[0124] In the above scheme, the updated critic network function (Critic Network) is usually used in reinforcement learning to evaluate the value or expected return of taking an action in a given state. In this context, the updated critic network may predict the expected value of the reward that may be obtained after executing the action based on the current state of the satellite-ground fusion network (such as the number of low-orbit satellites, the state of the ground station, the communication quality, etc.) and the actions taken (such as adjusting the satellite orbit, allocating communication resources, etc.).

[0125] The number of low-orbit satellites is an important component of the status of the satellite-ground integrated network, as it directly affects the network coverage, communication quality and resource allocation.

[0126] The expected value of the reward is a weighted average of the future rewards predicted by the updated critic network based on the current state and action, and is used to evaluate the quality of the action.

[0127] The update execution network function (Actor Network) is responsible for selecting the best action based on the current state in reinforcement learning. Updating the execution network usually involves adjusting its parameters to maximize the reward predicted by the update review network function. The parameters of the execution network function can be updated based on gradient ascent or similar optimization algorithms, using the output of the update review network function (i.e., the expected value of the reward) and a preset Jacobian matrix (which may be used to take into account interactions or constraints between network parameters).

[0128] The Jacobian matrix may be used here to capture the sensitivity of the expected value of the reward to changes in the parameters of the update execution network function. By taking this sensitivity into account, the update process can be more accurate and efficient because it not only considers direct changes in the reward, but also how parameter changes indirectly affect the reward through the network structure.

[0129] In some embodiments, Figure 2A and Figure 2B As shown:

[0130] Step 2011, construct the overall architecture of satellite-ground integrated network in power grid scenario:

[0131] Design modules such as satellite-ground access, user request and cache models, and establish a satellite-ground integrated network architecture that meets the needs of industries such as electricity and transportation.

[0132] Step 2021, constructing a satellite-ground fusion network model based on electricity:

[0133] Establish a satellite-ground fusion network architecture to meet the application of the power industry. The network is set to contain T multi-band low-orbit satellites with a cache capacity of T. All user terminals are within the coverage of the ground station, and the ground station is always in a state of simultaneous coverage of multiple low-orbit satellites. Users are evenly distributed within the coverage of a single ground station. The total number of users in the system is V, V = C × A, and the core network contains the content requested by all users, represented by T = {t j |j=1,2,…,T}, V={v j |j=1, 2, ..., V} represents a satellite set, a ground station set, and a user terminal set. The cache module can store part of the content requested by the user and return the content requested by the user through the user link and the backhaul link, thereby reducing the dependence on the backhaul link.

[0134] During the communication process, if the user cannot obtain the requested content from the local cache resources, it is necessary to connect to the low-orbit satellite through the ground station. If the content requested by the user is not cached on the low-orbit satellite, it is necessary to obtain the relevant content from the core network through the backhaul link.

[0135] Step 2022, constructing a satellite-to-ground access module:

[0136] For the convenience of research, the satellite-to-ground connection model uses the earth's surface as a reference and takes into account the ground coverage changes caused by the orbital motion of low-orbit satellites. As satellites move, ground station areas currently covered by a few satellites may be covered by other satellites in the future. Satellite movement and coverage scenarios involve multiple aspects of information such as satellite orbit height and inclination.

[0137] The process of a ground station sending a communication request and obtaining a low-orbit satellite communication connection is called access. In a satellite-ground fusion network with dynamic coverage of multiple satellites, the switching process is implemented using a channel reservation algorithm. Under the premise of knowing the relative displacement between the satellite network and the ground communication network, the remaining residence time of the ground device in the current beam / low-orbit satellite is calculated, and then the time for the ground device to request switching is calculated. Before switching, some channel resources of the next beam / low-orbit satellite are locked for the current ground device to ensure the success of the switching request, and attention is paid to the access strategy of the ground station in the scenario of dynamic coverage of multiple satellites. Therefore, the switching process in this scenario is set to be automatically executed after successful access.

[0138] A 0-1 matrix is ​​used to represent the satellite's mobility and access model. For a satellite-ground fusion network consisting of T low-orbit satellites and C ground stations, without considering inter-satellite links, the network topology and access status at time u can be represented by a T×C 0-1 matrix H u To express it, the specific calculation formula is as follows:

[0139]

[0140] in, represents the connection status between satellite t and ground station c at time u. When , it means that at time u, ground station c is not connected to satellite c. , indicating that at time u the ground station c is connected to the satellite t.

[0141] Step 2023, constructing an electricity user request module:

[0142] Assume that the content set requested by the power user is D = {d s |s∈(1,2,…,d,…,D)}, where s represents the popularity order of the current request content, and d s The probability of being requested is q s , where 1≤s1≤s2≤D, Therefore, using the set Q = {q s |s∈(1, 2, …, d, …, D)} to represent the user request state. It is assumed that when a single low-orbit satellite provides network services, the user's mobile range will not exceed the coverage of the current ground station.

[0143] Step 2024, build cache module:

[0144] Assuming the average request content size is n, for a low-orbit satellite with a cache capacity of T, its cache state can be represented by the vector express, When its value is 1, it means that at time u, satellite t has cached the content. When its value is 0, it means that at time u, satellite t has not cached the content. The same representation method can be used for the ground station cache status, that is, When its value is 1, it means that at time u, content d s is cached at ground station c. When its value is 0, it means that the content d s Therefore, the cache content of the ground station is set as follows:

[0145]

[0146] The cache content set of low-orbit satellites is as follows:

[0147]

[0148] Step 2012, design the reinforcement learning modeling process for cache and access optimization:

[0149] Assuming that the access status of satellites and ground stations in the model is not fixed, the action space of the agent changes from one-dimensional cache action to two-dimensional cache action and access action. Using separation strategy and experience replay mechanism to solve the multi-agent reinforcement deep learning problem, it can achieve good performance in reinforcement learning tasks with a large number of continuous action spaces. First, define the agent, state space, action space and reward function in the deep reinforcement learning algorithm. The environmental state observed by the model is composed of multiple factors such as the cache state of each node in the system, the file request state, and the connection state of each node in the system.

[0150] In the satellite-ground two-level cache network, each satellite is regarded as an agent. Each agent consists of two networks: an execution network and a critic network. The execution network outputs a probability distribution based on the local observations of each agent, which is similar to the actor-critic algorithm in the single-agent reinforcement learning algorithm. The role of the critic network is to evaluate the value function of the current state-action pair, which is used to calculate the gradient of the execution network. In the proposed method, the critic network calculates the value function based on the global state.

[0151] State Space:

[0152]

[0153] Includes content request status req u , satellite cache status Ground station cache status And the network connection status H u . U is the total simulation time, req u Indicates the request status received by the current low-Earth orbit satellite at time u. It is the cached content stored by the ground station currently served by the low-orbit satellite. is the cache content currently stored in the low-orbit satellite where the agent is currently deployed. u Represents the connection status between nodes in the current satellite-ground fusion network represented by a 0-1 matrix. The agent can identify the ground stations that are within the beam coverage at time u and the ground stations that will be covered at time u+l.

[0154] Action Space:

[0155]

[0156] When the satellite agent receives a request from the ground station at time u, it decides whether to allow the ground station to access and provide it with communication services, and whether to cache and update the content of the current request.

[0157]

[0158] Indicates whether the current satellite agrees to the ground station's access request, where Indicates whether the action at the current moment will affect the system connection status at the next moment. In addition, assuming that in this scenario, a beam only serves one ground station, This means that the number of ground stations accessing the satellite simultaneously does not exceed the total number of its beams.

[0159] This means that at the same time, each ground station has only one satellite beam providing communication services. If the selected ground station is not within the coverage of the current low-orbit satellite, the action is invalid and

[0160]

[0161] Indicates whether the current satellite t has cached the requested content d, where When its value is 1, it indicates that the current request content is updated to the cache space. When its value is 0, it indicates that the current request content is not cached. It should be noted that when the ground station to which the requested content belongs has not established a feeder link with the current satellite, this action is invalid and the content is not cached.

[0162] Reward function: It is usually a numerical value, and the discounted sum of rewards is called return. It is the goal that the satellite agent wants to maximize during the action selection process. Its calculation formula is as follows:

[0163]

[0164] Assume that the agents corresponding to multiple satellite nodes are in a collaborative relationship, and set the reward function of each satellite agent to the same value. Therefore, let all satellite agents work together to minimize the total delay of the requested content. If the total delay of the requested content is used as the reward function, the fluctuation space of the reward value will be large, and the deep reinforcement learning algorithm will be difficult to converge. Therefore, the normalized result of the total delay of the system at time u is used as the reward function.

[0165] Step 2013, construct the basic process of the satellite-ground integrated network access and cache strategy joint optimization method:

[0166] The basic process of the satellite-ground fusion network access and cache strategy joint optimization algorithm is as follows:

[0167] (1) Initialize the actor and critic networks of all agents and the corresponding target networks. Initialize the experience replay pool and related parameters.

[0168] (2) In each training step, all satellite agents observe the current state space, i.e., the content request state req u , satellite cache status Ground station cache status And the network connection status H u , select and execute the corresponding action, that is, whether the current satellite agrees to the access request of the ground station and whether the current satellite has cached the requested content. Continue to observe the network status at the next moment and the reward obtained by executing the action. The comment network calculates the action value function of each satellite agent and stores the obtained experience tuple (t, b, s, t′) in the experience replay pool to form a training data set. During the training process, the satellite agent can randomly sample a batch of value function data corresponding to the previous action space for training. This experience replay method helps to break the time correlation between data and improve efficiency and stability.

[0169] (3) For each satellite agent, a batch of experience tuples are randomly extracted from the experience replay buffer, which can alleviate data correlation, improve sample utilization, and reduce training variance.

[0170] (4) Calculate as follows The objective value function shown in the figure is then updated as The evaluation network shown.

[0171] (5) Update the actor network according to the value function of the comment network, as shown in the following formula:

[0172]

[0173] (6) Repeat steps (2) to (5) until the model reaches the predetermined number of training times or training goals. Finally, the satellite-ground fusion network cache and access optimization based on deep reinforcement learning and power grid is realized.

[0174] In the formula, z k represents the target value function, s j Indicates the popularity order of the current request content. Indicate action Then calculate the expected value of the future reward obtained by the current state t. M(θ j ) represents the definition of critical network loss, Represents the optimization formula of policy gradient. is the Jacobian matrix, Its d-th column represents the gradient θ of the d-th dimension of the policy with respect to the policy parameters.

[0175] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0176] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a device for jointly determining satellite-ground fusion network access and cache strategy.

[0178] refer to Figure 3 The satellite-ground fusion network access and cache strategy joint determination device includes:

[0179] The target determination module 301 is configured to use each of the plurality of low-orbit satellites as a target low-orbit satellite and each of the plurality of ground stations as a target ground station;

[0180] A connection status acquisition module 302 is configured to acquire a connection status between the target low-orbit satellite and the target ground station at a target time;

[0181] The request status determination module 303 is configured to obtain the user request content of the power user terminal, the cache status of the target low-orbit satellite at the target time, and the cache status of the target ground station at the target time, and determine the user request status based on the user request content;

[0182] The cache content determination module 304 is configured to determine the cache content of the low-orbit satellite through a low-orbit satellite cache algorithm based on the user request content and the cache status of the target low-orbit satellite, and determine the cache content of the ground station through a ground station cache algorithm based on the user request content and the cache status of the target ground station;

[0183] The strategy determination module 305 is configured to process through a single-agent reinforcement learning algorithm based on the connection status, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite, determine the target connection strategy of the satellite-ground fusion network from the connection status of the target low-orbit satellite and the target ground station, and determine the target cache strategy of the satellite-ground fusion network from the cache content of the low-orbit satellite and the cache content of the ground station.

[0184] In some embodiments, the request status determination module 303 is specifically configured to:

[0185] Get the content requested by the user s The corresponding popularity s, and the user request content d s The corresponding request probability q s ;

[0186] Based on the request probability q s And the popularity s is determined by the following formula:

[0187] Q = {q s |s∈(1,2,…,d,…,D)}.

[0188] In some embodiments, the cache content determination module 304 is specifically configured to:

[0189] Based on the user request content d s and the cache status of the target low-orbit satellite The cache content of the low-orbit satellite is determined by the following formula

[0190]

[0191] Among them, D represents the number of user requested contents, t represents the order of low-orbit satellites, T represents the number of low-orbit satellites, and u represents the order of target time.

[0192] In some embodiments, the cache content determination module 304 is specifically configured to:

[0193] Based on the user request content d s and the cache status of the target ground station The cache content of the ground station is determined by the following formula

[0194]

[0195] Where D represents the number of user requested contents, c represents the order of ground stations, ε represents the number of ground stations, and u represents the order of target time.

[0196] In some embodiments, the strategy determination module 305 includes:

[0197] A state space determining unit is configured to convert the connection state H u , the user request status req u , the cache status of the target ground station and the cache status of the target low-orbit satellite As the state space T u , where the state space T u Specifically:

[0198]

[0199] Among them, u represents the order of the target time, U represents the total time, t represents the order of the low-orbit satellite, T represents the number of low-orbit satellites, and c represents the order of the ground station. Indicates the number of ground stations;

[0200] An action space determination unit is configured to determine the connection state between the target low-orbit satellite and the target ground station. And whether the cache content of the low-orbit satellite caches the cache content of the ground station As the action space b, wherein the action space b is specifically:

[0201]

[0202] Among them, u represents the order of the target time, U represents the total time, t represents the order of the low-orbit satellite, T represents the number of low-orbit satellites, and c represents the order of the ground station. represents the number of ground stations, d represents the user requested content, and D represents the number of user requested content;

[0203] The transmission delay determination unit is configured to determine the transmission delay from the state space T uSelect the current state from the action space b, select the currently executed action from the action space b based on the current state, and obtain the cache delay T of the target ground station c (u), the cache delay T of the target low-orbit satellite t (u) and the transmission delay T of the user request content p (u);

[0204] A reward function determination unit is configured to obtain the total delay T of the satellite-ground fusion network. no-cache (u), based on the buffer delay T of the target ground station c (u), the cache delay T of the target low-orbit satellite t (u), transmission delay T for the user request content p (u) and the transmission delay T of the user requested content p (u) The reward function s(u) is obtained by the following formula, and the reward function s(u) is specifically:

[0205]

[0206] a replay extraction unit configured to determine a reward value corresponding to the currently executed action based on the reward function, select a next state of the current state from the state space, store the current state, the currently executed action, the reward value and the next state as experience tuples in a pre-constructed experience replay pool, and randomly extract a batch of experience tuples from the experience replay pool;

[0207] a determination unit configured to determine a target value function according to the extracted experience tuple, and to determine an update comment network function based on the target value function;

[0208] An adjustment unit is configured to determine an update execution network function based on the comment network function, continuously adjust the update comment network function based on the update execution network function until the value of the update comment network function is minimized, use the connection strategy corresponding to the minimized value of the update comment network function as the target connection strategy, and use the cache strategy corresponding to the minimized value of the update comment network function as the target cache strategy.

[0209] In some embodiments, the determining unit is specifically configured to:

[0210] Determine the popularity order s of the current user request content based on the extracted experience tuple j and the expected value of the reward obtained from the current state after taking the action

[0211] Based on the popularity order of the current user request content sj , the expected value of the reward obtained in the current state after taking the action and the preset constant parameter λ are used to determine the target value function z by the following formula k :

[0212]

[0213] In some embodiments, the determining unit is specifically configured to:

[0214] Obtain the number of low-orbit satellites T in the satellite-ground fusion network, based on the objective value function z k , the expected value of the reward obtained in the current state after taking the action R′ j (t, b1, L, b O ) and the number of low-orbit satellites T are determined by the following formula to update the review network function M(θ j ):

[0215]

[0216] In some embodiments, the adjustment unit is specifically configured to:

[0217] Based on the updated review network function M(θ j ) Determine the number of low-orbit satellites T in the satellite-ground fusion network and the expected value of the reward obtained after taking the action

[0218] Based on the number of low-orbit satellites T in the satellite-ground fusion network and the expected value of the reward obtained after taking action, and the preset Jacobian matrix The update execution network function is determined by the following formula

[0219]

[0220] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0221] The device of the above embodiment is used to implement the corresponding satellite-ground integrated network access and cache strategy joint determination method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0222] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for jointly determining satellite-ground fusion network access and cache strategy as described in any of the above embodiments is implemented.

[0223] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 401, a memory 402, an input / output interface 403, a communication interface 404, and a bus 405. The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are connected to each other in communication within the device through the bus 405.

[0224] The processor 401 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0225] The memory 402 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 402 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 402 and called and executed by the processor 401.

[0226] The input / output interface 403 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0227] The communication interface 404 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0228] The bus 405 comprises a pathway for transmitting information between the various components of the device (eg, the processor 401 , the memory 402 , the input / output interface 403 , and the communication interface 404 ).

[0229] It should be noted that, although the above device only shows the processor 401, the memory 402, the input / output interface 403, the communication interface 404 and the bus 405, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0230] The electronic device of the above embodiment is used to implement the corresponding satellite-ground integrated network access and cache strategy joint determination method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0231] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for jointly determining the satellite-ground integrated network access and cache strategy as described in any of the above embodiments.

[0232] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0233] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for jointly determining satellite-ground integrated network access and cache strategy as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0234] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0235] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0236] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0237] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A method for jointly determining satellite-ground fusion network access and cache strategy, characterized in that: The satellite-ground fusion network includes a plurality of low-orbit satellites and a plurality of ground stations, and the method includes: Taking each of the plurality of low-orbit satellites as a target low-orbit satellite, and taking each of the plurality of ground stations as a target ground station; Acquire the connection status between the target low-orbit satellite and the target ground station at a target time; Acquire user request content of the power user terminal, a cache status of the target low-orbit satellite at a target time, and a cache status of the target ground station at a target time, and determine a user request status based on the user request content; Determine the cache content of the low-orbit satellite by a low-orbit satellite cache algorithm based on the user request content and the cache state of the target low-orbit satellite, and determine the cache content of the ground station by a ground station cache algorithm based on the user request content and the cache state of the target ground station; Based on the connection status, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite, a single-agent reinforcement learning algorithm is used to process, and a target connection strategy of the satellite-ground fusion network is determined from the connection status of the target low-orbit satellite and the target ground station, and a target cache strategy of the satellite-ground fusion network is determined from the cache content of the low-orbit satellite and the cache content of the ground station.

2. The method according to claim 1, characterized in that The determining the user request status based on the user request content includes: Get the content requested by the user s The corresponding popularity s, and the user request content d s The corresponding request probability q s ; Based on the request probability q s And the popularity s is determined by the following formula: Q={q s |s∈(1,2,…,d,…,D)}。 3. The method according to claim 1, characterized in that The determining the cache content of the low-orbit satellite by a low-orbit satellite cache algorithm based on the user request content and the cache state of the target low-orbit satellite includes: Based on the user request content d s and the cache status of the target low-orbit satellite The cache content of the low-orbit satellite is determined by the following formula Among them, D represents the number of user requested contents, t represents the order of low-orbit satellites, T represents the number of low-orbit satellites, and u represents the order of target time.

4. The method according to claim 1, characterized in that: The determining the cache content of the ground station by using a ground station cache algorithm based on the user request content and the cache state of the target ground station includes: Based on the user request content d s and the cache status of the target ground station The cache content of the ground station is determined by the following formula Where D represents the number of user requested contents, C represents the order of ground stations, ε represents the number of ground stations, and u represents the order of target time.

5. The method according to claim 1, characterized in that The method comprises: processing the connection state, the user request state, the cache state of the target ground station and the cache state of the target low-orbit satellite through a single-agent reinforcement learning algorithm, determining the target connection strategy of the satellite-ground fusion network from the connection state of the target low-orbit satellite and the target ground station, and determining the target cache strategy of the satellite-ground fusion network from the cache content of the low-orbit satellite and the cache content of the ground station, including: The connection state H u , the user request status req u , the cache status of the target ground station and the cache status of the target low-orbit satellite As the state space T u , where the state space T u Specifically: Among them, u represents the order of the target time, U represents the total time, t represents the order of the low-orbit satellite, T represents the number of low-orbit satellites, and c represents the order of the ground station. Indicates the number of ground stations; The connection status between the target low-orbit satellite and the target ground station And whether the cache content of the low-orbit satellite caches the cache content of the ground station As the action space b, wherein the action space b is specifically: Among them, u represents the order of the target time, U represents the total time, t represents the order of the low-orbit satellite, T represents the number of low-orbit satellites, and C represents the order of the ground station. represents the number of ground stations, d represents the user requested content, and D represents the number of user requested content; From the state space T u Select the current state from the action space b, select the currently executed action from the action space b based on the current state, and obtain the cache delay T of the target ground station c (u), the cache delay T of the target low-orbit satellite t (u) and the transmission delay T of the user request content p (u); Get the total delay T of the satellite-ground fusion network no-cache (u), based on the buffer delay T of the target ground station c (u), the cache delay T of the target low-orbit satellite t (u), transmission delay T for the user request content p (u) and the transmission delay T of the user requested content p (u) The reward function s(u) is obtained by the following formula, and the reward function s(u) is specifically: Determine a reward value corresponding to the currently executed action based on the reward function, select a next state of the current state from the state space, store the current state, the currently executed action, the reward value and the next state as experience tuples in a pre-constructed experience replay pool, and randomly extract a batch of experience tuples from the experience replay pool; Determine a target value function according to the extracted experience tuple, and determine an update comment network function based on the target value function; An update execution network function is determined based on the comment network function, and the update comment network function is continuously adjusted based on the update execution network function until the value of the update comment network function is minimized, and a connection strategy corresponding to the value that minimizes the update comment network function is used as a target connection strategy, and a cache strategy corresponding to the value that minimizes the update comment network function is used as a target cache strategy.

6. The method according to claim 5, characterized in that Determining the target value function according to the extracted experience tuple includes: Determine the popularity order S of the current user request content based on the extracted experience tuple j and the expected value of the reward obtained from the current state after taking the action Based on the popularity order of the current user request content s j , the expected value of the reward obtained in the current state after taking the action and the preset constant parameter λ are used to determine the target value function z by the following formula k :

7. The method according to claim 6, characterized in that The step of determining and updating the comment network function based on the target value function includes: Obtain the number of low-orbit satellites T in the satellite-ground fusion network, based on the objective value function z k , the expected value of the reward obtained in the current state after taking the action R′ j (t, b1, L, b o ) and the number of low-orbit satellites T are determined by the following formula to update the review network function M(θ j ):

8. The method according to claim 1, characterized in that The step of determining an update execution network function based on the comment network function includes: Based on the updated review network function M(θ j ) Determine the number of low-orbit satellites T in the satellite-ground fusion network and the expected value of the reward obtained after taking the action Based on the number of low-orbit satellites T in the satellite-ground fusion network and the expected value of the reward obtained after taking action, and the preset Jacobian matrix The update execution network function is determined by the following formula 9. A satellite-ground fusion network access and cache strategy joint determination device, characterized in that: The satellite-ground fusion network includes a plurality of low-orbit satellites and a plurality of ground stations, and the device includes: a target determination module, configured to use each of the plurality of low-orbit satellites as a target low-orbit satellite, and use each of the plurality of ground stations as a target ground station; A connection status acquisition module, configured to acquire the connection status between the target low-orbit satellite and the target ground station at a target time; a request status determination module, configured to obtain user request content of the power user terminal, a cache status of the target low-orbit satellite at a target time, and a cache status of the target ground station at a target time, and determine a user request status based on the user request content; a cache content determination module, configured to determine the cache content of the low-orbit satellite through a low-orbit satellite cache algorithm based on the user request content and the cache state of the target low-orbit satellite, and to determine the cache content of the ground station through a ground station cache algorithm based on the user request content and the cache state of the target ground station; The strategy determination module is configured to perform processing through a single-agent reinforcement learning algorithm based on the connection status, the user request status, the cache status of the target ground station and the cache status of the target low-orbit satellite, determine the target connection strategy of the satellite-ground fusion network from the connection status of the target low-orbit satellite and the target ground station, and determine the target cache strategy of the satellite-ground fusion network from the cache content of the low-orbit satellite and the cache content of the ground station.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images