Cognitive edge computing node parameter optimization method and device based on reinforcement learning

Abstract

The present invention provides a method and device for optimizing cognitive edge computing node parameters based on reinforcement learning. The method includes: determining a partially observable Markov decision model based on the frequency band usage status of the current time slot of the main user end, using the partially observable Markov decision model to determine the belief probability corresponding to the main user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot; constructing a Bellman optimization model based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end, and maximizing the average reward within the preset future time slot range under the corresponding target state based on the Bellman optimization model, and determining the corresponding cognitive edge computing node parameter optimization strategy. The method disclosed in the present invention can maximize the long-term benefits of the secondary user end in cognitive radio scenarios, improve performance optimization efficiency, and save costs.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and more particularly to a method and device for optimizing cognitive edge computing node parameters based on reinforcement learning. In addition, the present invention also relates to an electronic device and a processor-readable storage medium. Background Art

[0002] In recent years, with the rapid development of mobile Internet technology, edge computing services realized with the help of mobile communication technology have been increasingly widely used. Edge computing technology is one of the key technologies of 5G. It sinks the server to the edge of the network, close to the user, allowing users to offload the computing business to the edge computing server for fast computing and then return the results, providing users with low-latency computing services. Its application process requires mobile users to have available frequency band resources. However, with the development of communication technology, the continuous growth of mobile users has made frequency band resources increasingly tight, making it difficult to allocate dedicated frequency bands to a large number of wireless user terminals, making it impossible for them to have available frequency bands for edge computing to improve their computing performance. Such a large number of mobile users will inevitably put tremendous pressure on the already very tight frequency band resources. As a very valuable natural resource, the increasing tension and even exhaustion of frequency band resources has become an indisputable fact. To truly solve this contradiction, how to improve the frequency band management method has become the focus of current research.

[0003] Cognitive radio is a dynamic spectrum sharing technology. When there are users with different priorities in the edge computing network, cognitive radio can allow users with low priority in using the frequency band in the network (secondary users or secondary users) to access the frequency band for data transmission when high-priority users (primary users or primary users) are not using the frequency band, which can effectively improve the spectrum utilization rate.

[0004] In the prior art, most of the research on edge computing resource allocation does not take user priority into consideration. They all consider that users can access the spectrum anytime and anywhere to offload tasks, which lacks consideration of the actual situation. Cognitive radio technology allows users to dynamically share spectrum resources according to priority, allowing low-priority users to access the spectrum while ensuring the service quality of high-priority users, which can improve spectrum utilization efficiency and provide spectrum access opportunities for users who lack spectrum resources. However, existing research that considers the priority of user spectrum resource use focuses on optimizing the performance of users within a single time slot. For edge computing networks, the long-term benefits of the network and users, that is, the long-term average performance indicators, are more important. Only considering instantaneous or short-term performance optimization may lead to a decline in long-term performance. Therefore, how to design a performance optimization solution that can consider the long term has become an important research topic in this field. Summary of the invention

[0005] To this end, the present invention provides a cognitive edge computing node parameter optimization method and device based on reinforcement learning to solve the problem of poor energy efficiency of the prior art performance optimization scheme that only considers instantaneous or short-term performance optimization.

[0006] In a first aspect, the present invention provides a method for optimizing cognitive edge computing node parameters based on reinforcement learning, comprising: determining a corresponding partially observable Markov decision model based on a frequency band usage state of a primary user terminal in a current time slot;

[0007] Using the partially observable Markov decision model, determining the belief probability corresponding to the primary user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot; wherein the frequency band usage priority of the primary user end is higher than the frequency band usage priority of the secondary user end;

[0008] A Bellman optimization model including a discount factor is constructed based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end, and the average reward within the preset future time slot range under the corresponding target state is maximized based on the Bellman optimization model to determine the corresponding cognitive edge computing node parameter optimization strategy.

[0009] In one embodiment, determining the corresponding partially observable Markov decision model based on the frequency band usage state of the primary user terminal in the current time slot specifically includes:

[0010] For each time slot, a preset energy detection model is used to detect the frequency band usage status of the main user terminal in the current time slot to determine the corresponding frequency band usage status; wherein the frequency band usage status includes a frequency band working status and a frequency band idle status;

[0011] The partially observable Markov decision model is determined according to the transition of the primary user terminal between the frequency band working state and the frequency band idle state in the current time slot.

[0012] In one embodiment, the partially observable Markov decision model is used to determine the belief probability corresponding to the primary user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot, specifically including:

[0013] The partially observable Markov decision model is used to observe the frequency band usage status of the current time slot, and the belief probability corresponding to the main user end in each future time slot is determined based on the obtained observation value, and the different observation probabilities and rewards corresponding to different decision operations performed by the secondary user end in each future time slot are determined based on the observation value.

[0014] In one embodiment, the method of maximizing the average reward within a preset future time slot range under the corresponding target state based on the Bellman optimization model and determining the corresponding cognitive edge computing node parameter optimization strategy specifically includes:

[0015] Based on the Bellman optimization model, the average reward within the preset future time slot range corresponding to the target state is maximized and the corresponding reward of the current time slot is determined to determine the corresponding cognitive edge computing node parameter optimization strategy.

[0016] In one embodiment, the reinforcement learning-based cognitive edge computing node parameter optimization method further includes: sending the cognitive edge computing node parameter optimization strategy to the secondary user terminal.

[0017] In a second aspect, the present invention also provides a cognitive edge computing node parameter optimization method based on reinforcement learning, comprising: receiving a cognitive edge computing node parameter optimization strategy issued by an edge computing node;

[0018] According to the target states in different time slots, the corresponding target cognitive edge computing node parameter optimization strategies are selected from the cognitive edge computing node parameter optimization strategies for optimization processing operations.

[0019] In a third aspect, the present invention also provides a cognitive edge computing node parameter optimization device based on reinforcement learning, comprising:

[0020] A Markov decision model determination unit, configured to determine a corresponding partially observable Markov decision model based on a frequency band usage state of a primary user terminal in a current time slot;

[0021] A future time slot state analysis unit, used to determine the belief probability corresponding to the primary user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot by using the partially observable Markov decision model; wherein the frequency band usage priority of the primary user end is higher than the frequency band usage priority of the secondary user end;

[0022] A parameter optimization strategy determination unit is used to construct a Bellman optimization model including a discount factor based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end, and maximize the average reward within a preset future time slot range under the corresponding target state based on the Bellman optimization model to determine the corresponding cognitive edge computing node parameter optimization strategy.

[0023] In one embodiment, the Markov decision model determination unit is specifically used to:

[0024] For each time slot, a preset energy detection model is used to detect the frequency band usage status of the main user terminal in the current time slot to determine the corresponding frequency band usage status; wherein the frequency band usage status includes a frequency band working status and a frequency band idle status;

[0025] The partially observable Markov decision model is determined according to the transition of the primary user terminal between the frequency band working state and the frequency band idle state in the current time slot.

[0026] In one embodiment, the future time slot state analysis unit is specifically used to:

[0027] The partially observable Markov decision model is used to observe the frequency band usage status of the current time slot, and the belief probability corresponding to the main user end in each future time slot is determined based on the obtained observation value, and the different observation probabilities and rewards corresponding to different decision operations performed by the secondary user end in each future time slot are determined based on the observation value.

[0028] In one embodiment, the parameter optimization strategy determination unit is specifically used to:

[0029] Based on the Bellman optimization model, the average reward within the preset future time slot range corresponding to the target state is maximized and the corresponding reward of the current time slot is determined to determine the corresponding cognitive edge computing node parameter optimization strategy.

[0030] In one embodiment, the cognitive edge computing node parameter optimization device based on reinforcement learning further includes: a parameter optimization strategy sending unit, which is used to send the cognitive edge computing node parameter optimization strategy to the secondary user terminal.

[0031] In a fourth aspect, the present invention also provides a cognitive edge computing node parameter optimization device based on reinforcement learning, comprising:

[0032] A parameter optimization strategy receiving unit, used to receive a cognitive edge computing node parameter optimization strategy issued by an edge computing node;

[0033] The optimization processing unit is used to select corresponding target cognitive edge computing node parameter optimization strategies from the cognitive edge computing node parameter optimization strategies according to the target states in different time slots to perform optimization processing operations.

[0034] In a fifth aspect, the present invention also provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for optimizing cognitive edge computing node parameters based on reinforcement learning as described in any one of the above are implemented.

[0035] In a sixth aspect, the present invention also provides a processor-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the cognitive edge computing node parameter optimization method based on reinforcement learning as described in any one of the above are implemented.

[0036] The reinforcement learning-based cognitive edge computing node parameter optimization method described in the present invention combines cognitive radio technology with edge computing technology, and utilizes partially observable Markov decision processes in the field of reinforcement learning to optimize the decisions of secondary users and base stations equipped with edge computing servers. That is, cognitive edge computing network parameter optimization can be performed in cognitive radio scenarios to maximize the long-term benefits of secondary users, improve performance optimization efficiency, and save costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 One of the flow charts of the method for optimizing cognitive edge computing node parameters based on reinforcement learning provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a cognitive edge computing system corresponding to the cognitive edge computing node parameter optimization method based on reinforcement learning provided in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a partially observable Markov decision model corresponding to the cognitive edge computing node parameter optimization method based on reinforcement learning provided in an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of changes in the expected rewards of secondary users corresponding to different discount factors as the time slot increases, provided by an embodiment of the present invention;

[0042] Figure 5 One of the structural schematic diagrams of the cognitive edge computing node parameter optimization device based on reinforcement learning provided by an embodiment of the present invention;

[0043] Figure 6 The second structural diagram of the device for optimizing cognitive edge computing node parameters based on reinforcement learning provided by an embodiment of the present invention;

[0044] Figure 7The second structural diagram of the device for optimizing cognitive edge computing node parameters based on reinforcement learning provided by an embodiment of the present invention;

[0045] Figure 8 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The following is a detailed description of an embodiment of the cognitive edge computing node parameter optimization method based on reinforcement learning according to the present invention. Figure 1 As shown, it is one of the flow charts of the cognitive edge computing node parameter optimization method based on reinforcement learning provided by an embodiment of the present invention. The specific implementation process includes the following steps:

[0048] Step 101: Determine a corresponding partially observable Markov decision model based on the frequency band usage status of the primary user terminal in the current time slot, wherein the primary user terminal may refer to a user terminal with a high frequency band usage priority in the network.

[0049] Before executing this step, it is necessary to predetermine a cognitive edge computing system consisting of a primary user terminal, a secondary user terminal, and a base station equipped with an edge computing server (i.e., edge computing node). Each edge computing node has a synchronized working time and the same time slot structure, and the length of a single time slot is T. Let g s represents the channel power gain between the base station and the secondary user end, and the channel coherence time is greater than or equal to θT. The primary user end has two states: working state A and idle state S. t ∈{A,S} represents the real state of the primary user in time slot t.

[0050] In this step, the frequency band usage status of the main user terminal in the current time slot is detected using a preset energy detection model for each time slot to determine the corresponding frequency band usage status. Specifically, the frequency band usage status includes a frequency band working state (i.e., working state A) and a frequency band idle state (i.e., idle state B). According to the transition between the frequency band working state and the frequency band idle state of the main user terminal in the current time slot, the partially observable Markov decision model is determined. That is, the state transition can constitute a two-state Markov chain model, and the transition probability is as follows Figure 2As shown, the main user is the main user terminal, and the secondary user is the secondary user terminal.

[0051] In the specific implementation process, the base station side can detect the frequency band usage status of the main user end through a preset energy detection model for each time slot. When the detection probability (i.e., when the main user end is in a working state, the probability of detecting that the main user end is also in a working state) is given as p d , perception duration τ0 and sampling rate f s In the case of , the false alarm probability (i.e., the probability of detecting that the main user terminal is in the working state when the main user terminal is in the idle state) is:

[0052]

[0053] Where θ is the received signal-to-noise ratio, make Represents the sensing result of the base station in the current time slot t.

[0054] Step 102: Determine the belief probability corresponding to the primary user terminal in each future time slot and the observation probability and reward corresponding to the secondary user terminal in each future time slot using the partially observable Markov decision model. The primary user terminal may refer to a user terminal with a low frequency band usage priority in the network. Specifically, the frequency band usage priority of the primary user terminal is higher than the frequency band usage priority of the secondary user terminal.

[0055] The partially observable Markov decision model is used to observe the frequency band usage status of the current time slot, and the belief probability corresponding to the main user end in each future time slot is determined based on the obtained observation value, and the different observation probabilities and rewards corresponding to different decision operations performed by the secondary user end in each future time slot are determined based on the observation value.

[0056] In the specific implementation process, the secondary user terminal needs to pre-select the corresponding operation parameters at the beginning of each time slot. t =(e a,t ,β t ) represents the operation parameters of the secondary user end in the tth time slot, where e a,t represents the computing energy value that the secondary user intends to invest, β t Indicates the secondary user-side computing mode, β t =1: The secondary client only performs local calculations; β t =2: The secondary user terminal offloads part of the service to the base station for calculation, and calculates part of the service locally. If the offloading is successful, the base station returns ACK (Acknowledge character; that is, confirmation character) information, that is, ACK=1, otherwise ACK≠1.

[0057] In the partially observable Markov decision model, the corresponding reward that the secondary user terminal can obtain after executing a decision or action in each time slot, that is, the amount of calculation, is also related to the CPU frequency and unloading power of the secondary user terminal. t =(e a,t ,β t =1), the reward that the secondary user can obtain is in:

[0058]

[0059] Where η is the power consumption coefficient of each CPU cycle of the secondary user. t =(e a,t ,β t =2), the maximum reward that the secondary user terminal can obtain, that is, the maximum reward of the current time slot, is the objective function value of the following optimization problem.

[0060]

[0061] stp(T-τ0)+ηf(T-τ0)≤e a,t

[0062] 0≤p≤p max

[0063] 0≤f≤f max

[0064] Where B is the channel bandwidth; p is the unloading power; σ 2 is the noise power at the small base station; C is the number of CPU cycles required for the secondary user end to calculate 1 bit of data; st (i.e., subject to) means that the following conditions are met; f max With p max are the maximum computing frequency and maximum unloading power of the secondary user CPU respectively. a,t ≤p max (T-τ0)+ηf max When (T-τ0), the optimal solution of the optimization problem is:

[0065]

[0066]

[0067] In the formula, When e a,t ≥p max (T-τ0)+ηf max When (T-τ0), the optimal solution is

[0068] Let b t =[bt (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) The probability that the base station considers the primary user end to be in state S at the beginning of time slot t.

[0069] In the partially observable Markov decision model, the secondary user executes a strategy (action) a t =(e a,t ,β t ) can be observed after the corresponding six observation values: k , k = 1, 2, 3, 4, 5, 6. The observation value is determined by the perception result of the frequency usage state of the current time slot detected by the energy detection model and the ACK returned by the base station after the secondary user terminal executes the corresponding strategy. The following analyzes the six existing observation values ​​one by one.

[0070] (1) First, t =(e a,t ,β t =1) or e a,t =0,e a,t ≤e s,t The observed values ​​at time , where e s,t is the available energy of the secondary user end at the beginning of time slot t:

[0071]

[0072] (2) When a t =(e a,t ,β t =2) or e a,t =0,e a,t ≤e s,t Analyze the observed values ​​at:

[0073]

[0074]

[0075] In the partially observable Markov decision model, the secondary user executes strategy a t =(e a,t ,β t ) can be used to iteratively update the above belief probability according to the observed value in each future time slot, and the corresponding specific expression is:

[0076]

[0077] In the formula, T(θ t+1|θ t ) is the main user band and the slave band usage status θ t Transfer to the frequency band usage state θ t+1 The state transition probability.

[0078] Step 103: A Bellman optimization model including a discount factor is constructed based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user terminal, and the average reward within the preset future time slot range under the corresponding target state is maximized based on the Bellman optimization model, and the corresponding cognitive edge computing node parameter optimization strategy is determined. The cognitive edge computing node parameter optimization strategy is sent to the secondary user terminal.

[0079] Specifically, the equation corresponding to the Bellman optimization model can be solved using a value iteration or policy iteration algorithm to obtain the secondary user end in different target states, that is, (e s,t ,b t ) is the optimal decision under the condition t , that is, the cognitive edge computing node parameter optimization strategy. The optimal decision includes the input energy value and computing mode corresponding to each future time slot.

[0080] Specifically, the corresponding cognitive edge computing node parameter optimization strategy can be determined based on the Bellman optimization model to maximize the average reward within the preset future time slot range under the corresponding target state and the corresponding reward of the current time slot. Among them, the reward of the current time slot refers to the reward obtained in the traditional optimization scheme for a single time slot optimization scheme, that is, when γ=0, the optimization decision of the secondary user end only considers the reward of the current time slot. That is, in the specific implementation process, in addition to determining the cognitive edge computing node parameter optimization strategy according to the average reward within the preset future time slot range under the target state obtained by the present invention, further, it is also possible to combine the average reward within the preset future time slot range under the target state obtained by the present invention and the reward of the current time slot obtained in the traditional optimization scheme for comprehensive consideration to determine the cognitive edge computing node parameter optimization strategy, that is, the optimal decision.

[0081] In the present invention, the algorithm expression corresponding to the Bellman optimization model is as follows:

[0082]

[0083] Among them, γ∈[0,1] is the discount factor;

[0084] The average reward within the preset future time slot range in the target state is expressed as:

[0085]

[0086] The reward for the current time slot is expressed as:

[0087]

[0088] The target state probability of the secondary user terminal is expressed as:

[0089]

[0090] When simulating the strip, the matlab simulation tool is used for simulation verification. The simulation results are as follows Figure 4 As shown in , the ordinate represents the expected reward of the secondary user end in each time slot in the future, and the abscissa represents the range of future time slots. When γ = 0, the optimization decision of the secondary user end only considers the reward of the current time slot, which is the traditional optimization scheme. Figure 4 It can be seen that under the premise of a given channel coherence time, the average reward that the secondary user can obtain per time slot in the future increases with the increase of the discount factor. Compared with the traditional solution optimized for a single time slot, the present invention has better performance benefits. Figure 3 It can also be seen that when the discount factor γ is small, the expected reward of the secondary user decreases as the time slot Θ increases. When the discount factor γ is large, the expected reward of the secondary user increases as the time slot Θ increases. This is because the discount factor corresponds to the degree to which the base station considers future benefits when making decisions. The larger the discount factor, the more important future benefits are. The long-term expected benefit of the secondary user increases as the time slot Θ increases, and vice versa.

[0091] The cognitive edge computing node parameter optimization method based on reinforcement learning described in the embodiment of the present invention is adopted. By combining cognitive radio technology with edge computing technology, the partially observable Markov decision process in the field of reinforcement learning is used to optimize the decision-making of secondary users and base stations equipped with edge computing servers. That is, cognitive edge computing network parameter optimization can be performed in cognitive radio scenarios to maximize the long-term benefits of secondary users, improve performance optimization efficiency, and save costs.

[0092] Corresponding to the above-mentioned method for optimizing cognitive edge computing node parameters based on reinforcement learning located on the base station side, the present invention also provides a device for optimizing cognitive edge computing node parameters based on reinforcement learning located on the base station side. Since the embodiment of the device is similar to the above-mentioned method embodiment, the description is relatively simple. For relevant matters, please refer to the description of the above-mentioned method embodiment. The embodiment of the device for optimizing cognitive edge computing node parameters based on reinforcement learning described below is only illustrative. Please refer to Figure 5 As shown, it is one of the structural schematic diagrams of a cognitive edge computing node parameter optimization device based on reinforcement learning provided by an embodiment of the present invention.

[0093] The cognitive edge computing node parameter optimization device based on reinforcement learning described in the present invention specifically includes the following parts:

[0094] A Markov decision model determining unit 501, configured to determine a corresponding partially observable Markov decision model based on the frequency band usage status of the primary user terminal in the current time slot;

[0095] The future time slot state analysis unit 502 is used to determine the belief probability corresponding to the primary user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot by using the partially observable Markov decision model; wherein the frequency band usage priority of the primary user end is higher than the frequency band usage priority of the secondary user end;

[0096] The parameter optimization strategy determination unit 503 is used to construct a Bellman optimization model including a discount factor based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end, and maximize the average reward within the preset future time slot range under the corresponding target state based on the Bellman optimization model to determine the corresponding cognitive edge computing node parameter optimization strategy.

[0097] The cognitive edge computing node parameter optimization device based on reinforcement learning described in the embodiment of the present invention is adopted. By combining cognitive radio technology with edge computing technology, the partially observable Markov decision process in the field of reinforcement learning is used to optimize the decision-making of secondary users and base stations equipped with edge computing servers. That is, cognitive edge computing network parameter optimization can be performed in cognitive radio scenarios to maximize the long-term benefits of secondary users, improve performance optimization efficiency, and save costs.

[0098] Corresponding to the above-mentioned method and device for optimizing cognitive edge computing node parameters based on reinforcement learning located on the base station side, the present invention also provides a method and device for optimizing cognitive edge computing node parameters based on reinforcement learning located on the secondary user side. The embodiment of the device for optimizing cognitive edge computing node parameters based on reinforcement learning described below is only illustrative. Please refer to Figure 6 As shown, it is the second structural diagram of a cognitive edge computing node parameter optimization method based on reinforcement learning provided by an embodiment of the present invention.

[0099] Step 601: Receive a cognitive edge computing node parameter optimization strategy issued by an edge computing node.

[0100] Step 602: Select corresponding target cognitive edge computing node parameter optimization strategies from the cognitive edge computing node parameter optimization strategies according to the target states in different time slots to perform optimization processing operations. Figure 2As shown, the optimization processing operation includes: parameter configuration operation and / or computing task offloading operation.

[0101] Please refer to Figure 7 As shown, it is the second structural schematic diagram of a cognitive edge computing node parameter optimization device based on reinforcement learning provided by an embodiment of the present invention.

[0102] The parameter optimization strategy receiving unit 701 is used to receive the cognitive edge computing node parameter optimization strategy sent by the edge computing node;

[0103] The optimization processing unit 702 is used to select corresponding target cognitive edge computing node parameter optimization strategies from the cognitive edge computing node parameter optimization strategies according to the target states in different time slots to perform optimization processing operations.

[0104] The method and device for cognitive edge computing node parameter optimization based on reinforcement learning described in the embodiments of the present invention are adopted. By combining cognitive radio technology with edge computing technology, the partially observable Markov decision process in the field of reinforcement learning is used to optimize the decision-making of secondary users and base stations equipped with edge computing servers. That is, cognitive edge computing network parameter optimization can be performed in cognitive radio scenarios, thereby maximizing the long-term benefits of secondary users, improving performance optimization efficiency, and saving costs.

[0105] Corresponding to the above-mentioned method for optimizing cognitive edge computing node parameters based on reinforcement learning, the present invention also provides an electronic device. Since the embodiment of the electronic device is similar to the above-mentioned method embodiment, the description is relatively simple. For relevant details, please refer to the description of the above-mentioned method embodiment. The electronic device described below is only exemplary. Figure 8 As shown, it is a schematic diagram of the physical structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may include: a processor (processor) 801, a memory (memory) 802 and a communication bus 803, wherein the processor 801 and the memory 802 communicate with each other through the communication bus 803 and communicate with the outside through the communication interface 804. The processor 801 can call the logic instructions in the memory 802 to execute the cognitive edge computing node parameter optimization method based on reinforcement learning. The method includes:

[0106] Determine a corresponding partially observable Markov decision model based on the frequency band usage status of the main user terminal in the current time slot; use the partially observable Markov decision model to determine the belief probability corresponding to the main user terminal in each future time slot and the observation probability and reward corresponding to the secondary user terminal in each future time slot; wherein the frequency band usage priority of the main user terminal is higher than the frequency band usage priority of the secondary user terminal; construct a Bellman optimization model containing a discount factor based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user terminal, and maximize the average reward within the preset future time slot range under the corresponding target state based on the Bellman optimization model, and determine the corresponding cognitive edge computing node parameter optimization strategy;

[0107] Or it includes: receiving a cognitive edge computing node parameter optimization strategy issued by an edge computing node; selecting corresponding target cognitive edge computing node parameter optimization strategies from the cognitive edge computing node parameter optimization strategies according to target states in different time slots for optimization processing operations.

[0108] In addition, the logic instructions in the above-mentioned memory 802 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a memory chip, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0109] On the other hand, an embodiment of the present invention further provides a computer program product, the computer program product comprising a computer program stored on a processor-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the cognitive edge computing node parameter optimization method based on reinforcement learning provided by the above-mentioned method embodiments. The method comprises:

[0110] Determine a corresponding partially observable Markov decision model based on the frequency band usage status of the main user terminal in the current time slot; use the partially observable Markov decision model to determine the belief probability corresponding to the main user terminal in each future time slot and the observation probability and reward corresponding to the secondary user terminal in each future time slot; wherein the frequency band usage priority of the main user terminal is higher than the frequency band usage priority of the secondary user terminal; construct a Bellman optimization model containing a discount factor based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user terminal, and maximize the average reward within the preset future time slot range under the corresponding target state based on the Bellman optimization model, and determine the corresponding cognitive edge computing node parameter optimization strategy;

[0111] Or it includes: receiving a cognitive edge computing node parameter optimization strategy issued by an edge computing node; selecting corresponding target cognitive edge computing node parameter optimization strategies from the cognitive edge computing node parameter optimization strategies according to target states in different time slots for optimization processing operations.

[0112] On the other hand, an embodiment of the present invention further provides a processor-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the method for optimizing cognitive edge computing node parameters based on reinforcement learning provided in the above embodiments is implemented. The method includes:

[0113] Determine a corresponding partially observable Markov decision model based on the frequency band usage status of the main user terminal in the current time slot; use the partially observable Markov decision model to determine the belief probability corresponding to the main user terminal in each future time slot and the observation probability and reward corresponding to the secondary user terminal in each future time slot; wherein the frequency band usage priority of the main user terminal is higher than the frequency band usage priority of the secondary user terminal; construct a Bellman optimization model containing a discount factor based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user terminal, and maximize the average reward within the preset future time slot range under the corresponding target state based on the Bellman optimization model, and determine the corresponding cognitive edge computing node parameter optimization strategy;

[0114] Or it includes: receiving a cognitive edge computing node parameter optimization strategy issued by an edge computing node; selecting corresponding target cognitive edge computing node parameter optimization strategies from the cognitive edge computing node parameter optimization strategies according to target states in different time slots for optimization processing operations.

[0115] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cognitive edge computing node parameter optimization method based on reinforcement learning, characterized in that: include: Determine a corresponding partially observable Markov decision model based on the frequency band usage status of the primary user terminal in the current time slot; Using the partially observable Markov decision model, determining the belief probability corresponding to the primary user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot; wherein the frequency band usage priority of the primary user end is higher than the frequency band usage priority of the secondary user end; A Bellman optimization model including a discount factor is constructed based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end, and the average reward within a preset future time slot range corresponding to the target state is maximized based on the Bellman optimization model, and a corresponding cognitive edge computing node parameter optimization strategy is determined; The algorithm expression corresponding to the Bellman optimization model is as follows: Where γ∈[0,1] is the discount factor; Θ is the time slot; Among them, T(θt +1 θt) is the main user band and the slave band usage status θ t Transfer to the frequency band usage state θ t+1 The state transition probability of θ t ∈{A,S} represents the real state of the primary user in time slot t, A represents the working state of the primary user, and S represents the idle state of the primary user; a t =(e a,t ,β t ) represents the execution decision of the secondary user at the tth time slot, where e a,t represents the computing energy value that the secondary user intends to invest, β t Indicates the secondary user-side computing mode, β t =1: The secondary client only performs local calculations; β t =2: The secondary user terminal offloads part of the service to the base station for calculation; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) The probability that the base station considers the primary user terminal to be in state S at the beginning of time slot t; O(O i |θ t+1 , a t ) represents the observation probability; O i represents the observed value; The reward for the current time slot is expressed as: Among them, a t represents the execution decision of the user end at the tth time slot; e s,t is the available energy of the secondary user end at the beginning of time slot t; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) is the probability that the base station considers the primary user end to be in state S at the beginning of time slot t; R(e s,t ,b t |a t , O i ) represents the reward, where O i represents the observed value; The target state probability of the secondary user terminal is expressed as: Among them, e s,t is the available energy of the secondary user end at the beginning of time slot t; e a,t Indicates the computing energy value that the secondary user intends to invest.

2. The method for optimizing cognitive edge computing nodes based on reinforcement learning according to claim 1, characterized in that: The determining of the corresponding partially observable Markov decision model based on the frequency band usage state of the primary user terminal in the current time slot specifically includes: For each time slot, a preset energy detection model is used to detect the frequency band usage status of the main user terminal in the current time slot to determine the corresponding frequency band usage status; wherein the frequency band usage status includes a frequency band working status and a frequency band idle status; The partially observable Markov decision model is determined according to the transition of the primary user terminal between the frequency band working state and the frequency band idle state in the current time slot.

3. The method for optimizing cognitive edge computing node parameters based on reinforcement learning according to claim 1, characterized in that: Using the partially observable Markov decision model, determining the belief probability corresponding to the primary user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot specifically includes: The partially observable Markov decision model is used to observe the frequency band usage status of the current time slot, and the belief probability corresponding to the main user end in each future time slot is determined based on the obtained observation value, and the different observation probabilities and rewards corresponding to different decision operations performed by the secondary user end in each future time slot are determined based on the observation value.

4. The method for optimizing cognitive edge computing nodes based on reinforcement learning according to claim 1, characterized in that: The method of maximizing the average reward within a preset future time slot range under the corresponding target state based on the Bellman optimization model and determining the corresponding cognitive edge computing node parameter optimization strategy specifically includes: Based on the Bellman optimization model, the average reward within the preset future time slot range corresponding to the target state is maximized and the corresponding reward of the current time slot is determined to determine the corresponding cognitive edge computing node parameter optimization strategy.

5. The method for optimizing cognitive edge computing nodes based on reinforcement learning according to claim 1, characterized in that: Also includes: The cognitive edge computing node parameter optimization strategy is sent to the secondary user terminal.

6. A method for optimizing cognitive edge computing node parameters based on reinforcement learning, characterized in that: include: Receive the cognitive edge computing node parameter optimization strategy issued by the edge computing node; The cognitive edge computing node parameter optimization strategy is to construct a Bellman optimization model including a discount factor based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end, and maximize the average reward within the preset future time slot range under the corresponding target state based on the Bellman optimization model; The algorithm expression corresponding to the Bellman optimization model is as follows: Where γ∈[0,1] is the discount factor; Θ is the time slot; Among them, T(θ t+1 |θ t ) is the main user band and the slave band usage status θ t Transfer to the frequency band usage state θ t+1 The state transition probability of θ t ∈{A,S} represents the real state of the primary user in time slot t, A represents the working state of the primary user, and S represents the idle state of the primary user; a t =(e a,t ,β t ) represents the execution decision of the secondary user at the tth time slot, where e a,t represents the computing energy value that the secondary user intends to invest, β t Indicates the secondary user-side computing mode, β t =1: The secondary client only performs local calculations; β t =2: The secondary user terminal offloads part of the service to the base station for calculation; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) The probability that the base station considers the primary user terminal to be in state S at the beginning of time slot t; O(O i |θ t+1 , a t ) represents the observation probability; O i represents the observed value; The reward for the current time slot is expressed as: Among them, a t represents the execution decision of the user end at the tth time slot; e s,t is the available energy of the secondary user end at the beginning of time slot t; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) is the probability that the base station considers the primary user end to be in state S at the beginning of time slot t; R(e s,t ,b t |a t , O i ) represents the reward, where O i represents the observed value; The target state probability of the secondary user terminal is expressed as: Among them, e s,t is the available energy of the secondary user end at the beginning of time slot t; e a,t Indicates the calculated energy value that the secondary user terminal intends to invest; According to the target states in different time slots, the corresponding target cognitive edge computing node parameter optimization strategies are selected from the cognitive edge computing node parameter optimization strategies for optimization processing operations.

7. A cognitive edge computing node parameter optimization device based on reinforcement learning, characterized in that: include: A Markov decision model determination unit, configured to determine a corresponding partially observable Markov decision model based on a frequency band usage state of a primary user terminal in a current time slot; A future time slot state analysis unit, used to determine the belief probability corresponding to the primary user end in each future time slot and the observation probability and reward corresponding to the secondary user end in each future time slot by using the partially observable Markov decision model; wherein the frequency band usage priority of the primary user end is higher than the frequency band usage priority of the secondary user end; A parameter optimization strategy determination unit is used to construct a Bellman optimization model including a discount factor based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end, and maximize the average reward within a preset future time slot range under the corresponding target state based on the Bellman optimization model to determine the corresponding cognitive edge computing node parameter optimization strategy; The algorithm expression corresponding to the Bellman optimization model is as follows: Where γ∈[0,1] is the discount factor; Θ is the time slot; Among them, T(θ t+1 |θ t ) is the main user band and the slave band usage status θ t Transfer to the frequency band usage state θ t+1 The state transition probability of θ t ∈{A,S} represents the real state of the primary user in time slot t, A represents the working state of the primary user, and S represents the idle state of the primary user; a t =(e a,t ,β t ) represents the execution decision of the secondary user at the tth time slot, where e a,t represents the computing energy value that the secondary user intends to invest, β t Indicates the secondary user-side computing mode, β t =1: The secondary client only performs local calculations; β t =2: The secondary user terminal offloads part of the service to the base station for calculation; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) The probability that the base station considers the primary user terminal to be in state S at the beginning of time slot t; O(O i |θ t+1 , a t ) represents the observation probability; O i represents the observed value; The reward for the current time slot is expressed as: Among them, a t represents the execution decision of the user end at the tth time slot; e s,t is the available energy of the secondary user end at the beginning of time slot t; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) is the probability that the base station considers the primary user end to be in state S at the beginning of time slot t; R(e s,t ,b t |a t , O i ) represents the reward, where O i represents the observed value; The target state probability of the secondary user terminal is expressed as: Among them, e s,t is the available energy of the secondary user end at the beginning of time slot t; e a,t Indicates the computing energy value that the secondary user intends to invest.

8. A cognitive edge computing node parameter optimization device based on reinforcement learning, characterized in that: include: A parameter optimization strategy receiving unit, used to receive a cognitive edge computing node parameter optimization strategy issued by an edge computing node; The cognitive edge computing node parameter optimization strategy is based on the belief probability, the observation probability and reward corresponding to each future time slot, and the target state probability of the secondary user end to build a Bellman optimization model including a discount factor, and based on the Bellman optimization model, the average reward within the preset future time slot range under the corresponding target state is maximized; the algorithm expression corresponding to the Bellman optimization model is as follows: Where γ∈[0,1] is the discount factor; Θ is the time slot; Among them, T(θ t+1 |θ t ) is the main user band and the slave band usage status θ t Transfer to the frequency band usage state θ t+1 The state transition probability of θ t ∈{A,S} represents the real state of the primary user in time slot t, A represents the working state of the primary user, and S represents the idle state of the primary user; a t =(e a,t ,β t ) represents the execution decision of the secondary user at the tth time slot, where e a,t represents the computing energy value that the secondary user intends to invest, β t Indicates the secondary user-side computing mode, β t =1: The secondary client only performs local calculations; β t =2: The secondary user terminal offloads part of the service to the base station for calculation; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) The probability that the base station considers the primary user terminal to be in state S at the beginning of time slot t; O(O i |θ t+1 , a t ) represents the observation probability; O i represents the observed value; The reward for the current time slot is expressed as: Among them, a t represents the execution decision of the user end at the tth time slot; e s,t is the available energy of the secondary user end at the beginning of time slot t; b t =[b t (A),b t (S)] represents the belief probability, where b t (A) is the probability that the small base station believes that the primary user terminal is in state A at the beginning of time slot t, b t (S) is the probability that the base station considers the primary user end to be in state S at the beginning of time slot t; R(e s,t ,b t |a t , O i ) represents the reward, where O i represents the observed value; The target state probability of the secondary user terminal is expressed as: Among them, e s,t is the available energy of the secondary user end at the beginning of time slot t; e a,t Indicates the calculated energy value that the secondary user terminal intends to invest; The optimization processing unit is used to select the corresponding target cognitive edge computing node parameter optimization strategy from the cognitive edge computing node parameter optimization strategy according to the target state in different time slots to perform optimization processing operations.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the cognitive edge computing node parameter optimization method based on reinforcement learning as described in any one of claims 1-6 are implemented.

10. A processor-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cognitive edge computing node parameter optimization method based on reinforcement learning as described in any one of claims 1 to 6 are implemented.

Images