Multi-constraint spectrum allocation method based on LSTM optimized DQN network
Through the multi-constrained spectrum allocation method based on LSTM optimization DQN network, the interference problem of electromagnetic spectrum resource allocation in complex battlefield environments is solved, and efficient, fast and low-interference spectrum allocation is achieved, which is suitable for multi-task scenarios.
Patent Information
- Application Number
- CN202210271164.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-18
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-18
AI Technical Summary
In complex multi-dimensional battlefield environments, improper allocation of electromagnetic spectrum resources may lead to serious interference between equipment and frequency band paralysis, affecting the effectiveness of joint combat, and the existing technology is difficult to effectively solve this problem.
Using the multi-constrained spectrum allocation method based on LSTM to optimize the DQN network, the available and interference matrix are calculated by obtaining device parameters and frequency band resource data, and the LSTM network is used to learn long-term dependencies, and the frequency band selection is combined with the DQN network to achieve the optimal spectrum allocation scheme. .
It increases the normal access number of equipment, reduces interference in the environment, generates a broad and adaptable optimized spectrum allocation scheme, and realizes efficient, fast and low-interference spectrum allocation, which is suitable for different task scenarios.
Smart Images

Figure CN114554497B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to electromagnetic spectrum related communication technology, and specifically relates to a multi-constraint spectrum allocation method based on LSTM optimized DQN network. Background Art
[0002] Joint operations are the basic combat form of future information warfare. Through scientific and reasonable organization, various forces can form complementary functions and exert the greatest combat effectiveness. With the rapid development of science and technology, the combat area of future wars has expanded from the three-dimensional battlefield of land, sea and air to the multi-dimensional battlefield space of land, sea, air, space and electromagnetic. A large number of communications, radars, electronic warfare equipment and even electromagnetic pulse equipment will be put into the battlefield in large quantities in a short period of time, which will inevitably form an extremely complex electromagnetic environment in the regional battlefield, which is very likely to cause serious interference problems between equipment or a large number of equipment cannot be allocated available frequency bands and thus paralyzed. Therefore, joint operations require the command system to be able to coordinate, plan and dispatch battlefield resources spread across various spaces in a very short time according to the changes in the entire battlefield situation, so that battlefield electromagnetic and other resources can operate efficiently and finally achieve the goal of joint operations. From this point of view, the reasonable allocation of electromagnetic spectrum resources has become an important reference indicator for our electromagnetic and information rights on the information battlefield, which will greatly affect the final outcome of our war. However, with the continuous improvement of technology in the field of artificial intelligence, some research results have been achieved in solving the problem of spectrum resource allocation using knowledge in this field. Summary of the invention
[0003] In view of this, the main purpose of the present invention is to provide a multi-constraint spectrum allocation method based on LSTM optimization DQN network.
[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:
[0005] An embodiment of the present invention provides a multi-constraint spectrum allocation method based on LSTM optimization DQN network, the method comprising:
[0006] In the pre-processing stage, the parameter indicators and frequency band resource data of various devices in the system are obtained and various function matrices are calculated;
[0007] Initialize the DQN network and complete the mapping between the pre-processing results and the network elements;
[0008] Set the number of simulation phases to episode and record it as e, and divide the network training process into multiple episodes e;
[0009] Update search probability p t ;
[0010] Based on the centralized distribution, the local distribution method is used to distribute the devices D in the environment in turn.i Perform actions, that is, the selection of frequency bands, and finally summarize them into overall actions a t ;
[0011] Iterative learning of the entire network;
[0012] Collect and update the historical optimal overall action a t , and output it as the optimal spectrum allocation plan, which will be allocated to specific frequency band resources for each device in turn.
[0013] In the above scheme, the parameter indicators and frequency band resource data in the system are obtained in the pre-processing stage, specifically: the total length of the current available frequency band is F, F∈[f min ,f max ], all devices in the area D i The internal information of the frequency band, such as working frequency band, bandwidth, time, deployment location, etc., i = 1, 2, 3, ..., m, the protection frequency band f restricted for use within the total frequency band F i and its position L i With time t i Case, i = 1, 2, 3, ..., r, frequency band usage priority between different devices P i , i=1,2,3,...,p, current frequency band occupancy O i , where O i is an array, which records the occupancy of the observed frequency points in the frequency band F with 0 and 1;
[0014] The calculation of each function matrix is specifically as follows: each function matrix includes an available frequency band matrix M a and the interference matrix M i , where the free space loss formula involved in interference discrimination is as follows: LS = 32.45 + 20lnf + 20lnd;
[0015] Calculate the interference matrix M i , according to the device D i The information of the available frequency band and working bandwidth is obtained, so that the available frequency band F is divided into k sub-bands and each sub-band is given a specific frequency band number k. i , and then generate an m-row, k-column matrix M of available frequency bands for all devices a , the matrix elements are all binary, that is, 0 represents the device D i Not available in this frequency band, and 1 represents D i Available in this frequency band, where the M a When checking each element, determine whether the available frequency band overlaps with the restricted time and area in the environment. If so, it will be judged as unavailable. Based on the occupancy of the overall frequency band, if the frequency band is already occupied, it will also be judged as unavailable.
[0016] Calculate the interference matrix M i , according to the device D i The information of the transmission power and receiver sensitivity is obtained according to the available frequency band matrix M a Generate an m×m×k three-dimensional interference matrix M i , where the elements are also binary. When a certain frequency band is fixed, if the device D i , satisfying the free space loss formula below, and the working time of the two devices overlaps, then the element is set to 1 at this time, that is, interference occurs, otherwise it is set to 0, no interference, the free space loss formula is as follows: LS = 32.45 + 20lnf + 20lnd, where the unit of space loss LS is dB, f is the operating frequency of the device, the unit is MHz, and d is the distance between the devices, the unit is km. When the attenuated value of the transmission power of the device is still greater than the receiving sensitivity of the other device, it is determined that the two devices are interfering.
[0017] In the above scheme, the mapping between the pre-processing results and the network elements is completed as follows: Action a t It refers to the specific frequency band selected by the device at a certain moment, and serves as an integral part of the overall solution for subsequent steps; state s t is a row vector composed of the action selected by the user, the reuse of the entire frequency band, and the 0 / 1 signal indicating whether the allocation is successful or not; the reward r t It is a function related to the ratio of failed devices to overall devices in the overall solution obtained after all devices select actions, and is defined as follows: Among them, α and p represent the weight coefficient and priority of device i respectively.
[0018] In the above scheme, the update search probability p t , specifically: Among them, p t is the search probability at a certain moment, p start With p end are the search probabilities of starting and ending, r decay Its decay speed.
[0019] In the above scheme, the local distribution method is used to sequentially distribute the devices D in the environment on the basis of centralized distribution. i Perform actions, that is, the selection of frequency bands, and finally summarize them into overall actions a t , specifically: each device D to be allocated in the communication environment i According to the search probability p t Conduct a t The judgment choice; the historical status data s tThe LSTM network is sent to learn long-distance dependencies, that is, the input historical data is selectively retained or forgotten; the Q network obtains the corresponding value Q according to the output data t , so as to select action a t , and combine the environment to obtain the corresponding reward r t , then the environment state is transferred to s t+1 , and at the same time, the elements of the current moment are (s t ,a t ,r t ,s t+1 ) and stored in the experience replay pool.
[0020] In the above scheme, each device D to be allocated in the communication environment i According to the search probability p t Conduct a t The judgment selection is as follows: each device D to be allocated i According to the search probability p t Perform an action t , that is, the judgment and selection of specific available frequency bands, introducing the random quantity r const , if p t ≥r const , in M a If the above conditions are not met, it is necessary to select according to the maximum Q value corresponding to the state observation value, and the formula is as follows: t = argmax a Q(φ(s t ),a i ;θ).
[0021] In the above scheme, the Q network obtains the corresponding value Q according to the output data t , so as to select action a t , and combine the environment to obtain the corresponding reward r t , then the environment state is transferred to s t+1 , and at the same time, the elements of the current moment are (s t ,a t ,r t ,s t+1 ) in the form of a combination and storage in the experience replay pool. Specifically, the phased data output by the LSTM network is used as the input data of the subsequent fully connected part of the overall network and the Q value is calculated accordingly. Device D i Each optional frequency band has a Q value corresponding to it; the action at that moment is selected according to the rule, that is, the frequency band scheme a i ; After all devices have completed action selection at this moment, all actions a i Composition of overall action t, according to the formula and status shown below, the final reward r is obtained t : Equipment in the environment D i According to the action, the overall frequency band usage and the signal identification of whether the allocation is successful or not caused by the action selection are updated, and its own state s at the future time is further updated. t+1 ; All devices at the current time and future time D i The states are combined into the overall s t and t+1 , and compare these two data with the a of the overall action plan t and reward r t Merge and store them in the experience replay pool.
[0022] In the above scheme, the overall network iterative learning is specifically as follows: a small sample batch of batch_size is randomly extracted from the historical data of the experience replay pool and recorded as b i , each sample b i are all data bodies composed of four items of data in step 7.5; for each sample b i , extract the internal etc.; multiple samples b i Status in Combine and use as network input, select the frequency band action selected by the device from the many Q values obtained Corresponding value For sample b i , the Q network will calculate the future state The maximum value of Q that may be involved, and using the formula Calculate the target Q value; use the formula based on the target Q value and the current Q value Calculate the loss value of the network and update the network parameters; where N b is batch_size, which means the number of samples extracted in each batch; repeat the above steps until the learning process is completed and the results are generated.
[0023] Compared with the prior art, the multi-constraint spectrum allocation technology based on LSTM optimized DQN network provided by the present invention utilizes artificial intelligence methods to collect and organize various data such as model-free random environment and random equipment in the area, enriching the feasibility of complex practical scenarios. Under the constraints of satisfying the priority of different devices, a comprehensive analysis is performed on various information of all devices in the model-free random environment that allows resource reuse. On the premise of increasing the number of normal access devices, the interference generated in the environment is gradually reduced, and finally an optimized spectrum allocation scheme with wide adaptability is produced. The method proposed by the present invention has good and wide applicability for different mission scenarios, and at the same time conforms to the actual parameter architecture of battlefield equipment. It is a practical, efficient, fast, and low-interference spectrum allocation method. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings described herein are used to disclose a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0025] Figure 1 It is a schematic flow chart of the whole process of the present invention;
[0026] Figure 2 It is the basic configuration diagram of the overall DQRN network;
[0027] Figure 3 It is a diagram of the internal data of each (simulated) device in the embodiment of the present invention;
[0028] Figure 4 is a diagram of external restriction conditions in an embodiment of the present invention;
[0029] Figure 5 is the result graph generated by the algorithm of the present invention;
[0030] Figure 6 This is a comparison chart of the improvement of the present invention compared to the basic DQN network. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0032] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, article or device. In the absence of more restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, article or device including the element.
[0033] The present invention provides a multi-constraint spectrum allocation method based on LSTM optimization DQN network, and the present invention is described in detail below with reference to the accompanying drawings.
[0034] like Figure 1 As shown, the steps of allocating frequency bands in a regional battlefield environment of the present invention are as follows:
[0035] Step 1, use Figure 3 , Figure 4 The data shown in the figure obtains various data in the system and calculates various function matrices in the pre-processing stage:
[0036] Step 1.1: Obtain various parameter indicators. Among them, the total length of the current available frequency band is F, F∈[f min ,f max ] All equipment in the area D i The internal information of the total frequency band F, such as the working frequency band, bandwidth, time, deployment location, etc., i = 1, 2, 3, ..., m. The protected frequency band f within the total frequency band F is restricted for use. i and its position L i With time t i Case, i = 1, 2, 3, ..., r. Frequency band usage priority between different devices P i ,i=1,2,3,...,p。 Current frequency band occupancy O i , where O i is an array, which records the occupancy of the observed frequency points in the frequency band F with 0 and 1;
[0037] Step 1.2: Calculate the available frequency band matrix M a The device D obtained according to step 1 i The information of the available frequency band and working bandwidth is obtained, so that the available frequency band F can be divided into k sub-bands and given a specific frequency band number k. i , and then generate an m-row, k-column matrix M of available frequency bands for all devices a , the matrix elements are all binary, that is, 0 represents the device D i Not available in this frequency band, and 1 represents D i Available in this frequency band.
[0038] Among them, in generating M a When selecting each element, it is necessary to consider whether the available frequency band overlaps with the restricted time and area in the environment. If so, it will also be judged as unavailable. The occupancy of the overall frequency band. If the frequency band is occupied, it will also be judged as unavailable;
[0039] Step 1.3: Calculate the interference matrix M i . According to the device D i The information of the transmission power and receiver sensitivity can be obtained, so that the available frequency band matrix M a Generate an m×m×k three-dimensional interference matrix M i , where the elements are also binary. When a certain frequency band is fixed, if the device D i , satisfying the free space loss formula below, and the working time of the two devices overlaps, then the element is set to 1, which means interference occurs, otherwise it is set to 0, which means no interference. The free space loss formula is as follows:
[0040] LS=32.45+20lnf+20lnd
[0041] The unit of space loss LS is dB, f is the operating frequency of the device, in MHz, and d is the distance between the devices, in km. When the attenuated transmit power of a device is still greater than the receiving sensitivity of another device, it is determined that there is interference between the two devices.
[0042] Step 2: Initialize the DRQN network in the structure and complete the mapping between the preprocessing results and the network elements:
[0043] Step 2.1: Initialize the Experience Replay pool and assign a value to its capacity;
[0044] Step 2.2: Affecting action a t The selected Q-value network Q-Network is initialized and given parameters θ;
[0045] Step 2.3: If Figure 2 As shown, the LSTM network is embedded into the overall network to form a DRQN network, so that the LSTM network can be used to reasonably solve the performance consumption and dependency problems caused by long-term information;
[0046] Among them, the LSTM network designs the concept of three types of "gates", and realizes the reasonable processing of information at different times in the network through the coordinated work of each gate. Among them, the input of the gate is a vector, and the output is a real number vector between 0 and 1, which can be expressed as:
[0047] g(x)=σ(Wx+b)
[0048] Where W is the weight vector of the gate, b is the bias term, and σ is the Sigmoid function. The three types of internal gate structures of LSTM networks are:
[0049] (1) Input gate: controls the current state that needs to be saved to the cell state c t The input data x t The scale of its action function is as follows:
[0050]
[0051] i t =σ(W i [h t-1 ,x t ]+b i )
[0052]
[0053] (2) Forget gate: controls the cell state c at time t-1 t-1 At time t, the cell state is c t The degree of retention determines whether a part of the data can be discarded by the LSTM system. Its function is as follows;
[0054] f t =σ(W f [h t-1 ,x t ]+b f )
[0055] (3) Output gate: controls the cell state c at time t t The output value h at the current moment can be successfully t The scale is saved in the function as follows;
[0056] O t =σ(W o [h t-1 ,x t ]+b o )
[0057] h t =O t *tanh(C t )
[0058] Step 2.5: Map the pre-processing results and other spectrum allocation related concepts to network elements, where action a tIt refers to the specific frequency band selected by the device at a certain moment, and serves as an integral part of the overall plan for subsequent steps; state s t The row vector is composed of the action selected by the user, the reuse of the entire frequency band, and the 0 / 1 signal indicating whether the allocation is successful or not. The reward r t It is a function related to the ratio of failed devices to overall devices in the overall solution obtained after all devices select actions, and is defined as follows:
[0059]
[0060]
[0061] Among them, α and p represent the weight coefficient and priority of device i respectively.
[0062] Step 3: Set the number of simulation stages, episode, and record it as e. In order to fully reflect the process of continuous network optimization, the DQN algorithm generally divides the network training process into multiple episodes, and the following steps are all limited to e;
[0063] Step 4: Update the search probability p t . The search probability p t is defined as follows:
[0064]
[0065] Among them, p t is the search probability at a certain moment, p start With p end are the search probabilities of starting and ending, r decay is its decay rate. From the above formula, we can see that the search probability gradually decreases with the increase of training times.
[0066] From step 5 to step 7, the following steps will use local distribution to distribute the devices D in the environment in turn based on centralized distribution. i Perform actions, that is, the selection of frequency bands, and finally summarize them into overall actions a t .
[0067] Step 5: Each device D to be allocated in the frequency environment i According to the search probability p t Choices to make judgments:
[0068] Step 5.1: Each device D to be allocated i According to the search probability p t Perform an action t , that is, the judgment and selection of specific available frequency bands. In order to ensure flexibility, the random quantity r is introduced const If p t ≥rconst , then you can a Randomly select from the corresponding available frequency bands;
[0069] Step 5.2: If the above conditions are not met, it is necessary to select the maximum Q value corresponding to the state observation value, and the formula is as follows:
[0070] a t = argmax a Q(φ(s t ),a i ;θ)
[0071] Step 6: Get historical data and pass it to the LSTM network for processing:
[0072] Step 6.1, set the variable step_size and record it as N step , replace t,t-1,...,tN step +1 A historical state s at the same time t The data is used as the input of the LSTM network built in the overall network in step 2 and the stage result output is obtained. The output output represents the h in the LSTM structure. t Data, which determines whether to selectively retain or forget certain long-term data, thereby improving algorithm processing performance to a certain extent;
[0073] Step 7, get the Q value and transfer the state:
[0074] Step 7.1: Use the phased data output by the LSTM network in step 6.1 as the input data for the subsequent fully connected part of the overall network and calculate the Q value accordingly. Device D i Each optional frequency band has a corresponding Q value;
[0075] Step 7.2: Select the action at this moment according to the rule shown in step 5, that is, the frequency band scheme a i ;
[0076] Step 7.3: After all devices have completed action selection at this moment, all actions a i Composition of overall action t , according to the formula and status shown below, the final reward r is obtained t :
[0077]
[0078]
[0079] Step 7.4: Each device in the environment D iAccording to the action obtained in step 7.3, the overall frequency band usage and the signal identification of whether the allocation is successful or not caused by the action selection are updated, and its own state s at the future time is further updated. t+1 ;
[0080] Step 7.5: D i The states are combined into the overall s t and t+1 , and compare these two data with a of the overall action plan in 7.3 t and reward r t Merge and store them in the experience replay pool;
[0081] Starting from step 8, the overall network iteratively learns:
[0082] Step 8.1: Randomly extract a small sample batch of batch_size from the historical data in the experience replay pool and record it as b i , each sample b i They are all data bodies composed of four items of data in step 7.5;
[0083] Step 8.2: For each sample b i , extract the internal and other information;
[0084] Step 8.3: Multiple samples b obtained in the previous step i Status in Combine and use as network input, select the frequency band action selected by the device from the many Q values obtained Corresponding value
[0085] Step 9, calculate the target Q value. For sample b in step 8.2 i , the Q network will calculate the future state The maximum value of Q that may be involved, and using the formula Calculate the target Q value;
[0086] Step 10: The target Q value and current Q value obtained through the above steps are calculated using the formula Calculate the network loss value and update the network parameters;
[0087] Among them, N b is batch_size, which means the number of samples drawn in each batch.
[0088] Step 11, repeat the above steps until the learning process is completed and generate Figure 5 Results shown;
[0089] Depend on Figure 5As shown, in the preprocessing stage of this method, all frequency band resources are processed according to Figure 3 The maximum working bandwidth between devices is used as the cutting scale, and the overall frequency band resource of 40MHz is allocated to 8 segments. Since the implementation example data sets the extreme state that the devices interfere with each other and can compete for resources with each other, the expected optimal result must be that each device exclusively occupies a frequency band number, that is, a 5MHz frequency band resource. Finally, multiple results generated by the method of the present invention are consistent with the expected results. Under this result, all devices complete the access and minimize the interference of the overall environment.
[0090] Step 12: Collect and update the historical optimal overall action a t , and output it as the optimal spectrum allocation plan, which will be allocated to specific frequency band resources for each device in turn.
[0091] Since an overall action a is generated at each moment in the learning process of step 11 t , that is, the overall allocation plan, and does not exclude a at multiple different times t It can become the optimal solution. Therefore, in step 12, it is necessary to collect and update the historical overall action with the highest comprehensive reward value, that is, the best solution output to the decision-making layer. Finally, the decision-making department will evaluate it comprehensively and then t The frequency band corresponding to a device in the middle position is allocated to the device.
[0092] For the performance of DQN network optimized by LSTM, such as Figure 6 The performance and time comparison curve reflects the results obtained with the basic DQN network.
[0093] Simulation conditions
[0094] like Figure 3 , Figure 4 As shown, it is assumed that the total number of devices that need to be allocated frequencies in the entire spectrum environment is D i The frequency bands are set to 8, and each device can select 8 frequency bands together to increase the conflict effect within the system. At the same time, the other parameters are as follows:
[0095] (1) The number of iterations is 100,000, each 5,000 iterations is one episode, and one optimization curve will be displayed;
[0096] (2) The hidden layer of the neural network in the basic DQN is set to 128, and the ReLu function is used as the overall activation function;
[0097] (3) batch_size is set to 16;
[0098] (4) The hidden layer of the LSTM network is set to 96 and the step_size is set to 15;
[0099] (5) The capacity of the Experience Replay pool is set to 3000;
[0100] (6) The value p of the exploration probability explore_p t As the number of iterations t increases, it gradually decays from 0.5 to 0.1;
[0101] (7) The learning rate is 0.01;
[0102] (8) The discount factor γ is 0.02;
[0103] Simulation Results
[0104] like Figure 5 As shown in the figure, this method allocates a unique frequency band to each device when the simulation data specifies mutual interference between devices, which minimizes the overall interference and increases the number of successfully allocated devices. It basically solves the allocation problem of priority and multi-constraint scenarios in complex scenarios and achieves the expected results. Figure 6 In the LSTM_DQN and basic DQN algorithms, the overall cumulative number of allocation failures is decreasing. After multiple learnings, the two algorithms reduce the number of user cumulative allocation failures to about 4100 and 5000 respectively. In terms of the overall learning effect, both algorithms can successfully solve the spectrum allocation problem of the resource environment model constructed in the simulated battlefield environment set in this paper, but the network based on LSTM optimization processes data better overall, and the final overall efficiency improvement is about 18%. In the problem of time consumption, the DQN architecture based on LSTM optimization has a convergence time that is basically in the case of episode 10. Compared with the basic DQN architecture, the results shown in the present invention are ahead of the convergence time by 3 episodes, and in the optimization process, the former also basically maintains a leading advantage of 2 to 3 episodes, indicating that the rapidity function of LSTM in the theoretical design is basically realized.
[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.
Claims
1. A multi-constrained spectrum allocation method based on LSTM optimized DQN network, It is characterized in that The method includes: In the pre-processing stage, the parameter indicators and frequency band resource data of various devices in the system are obtained and various function matrices are calculated; Initialize the DQN network and complete the mapping between the pre-processing results and the network elements; Set the number of simulation phases to episode and record it as e, and divide the network training process into multiple episodes e; Update search probability p t ; Based on the centralized distribution, the local distribution method is used to distribute the devices D in the environment in turn. i Perform actions, that is, the selection of frequency bands, and finally summarize them into overall actions a t ; Iterative learning of the entire network; Collect and update the historical optimal overall action a t , and output it as the optimal spectrum allocation plan, and allocate specific frequency band resources to each device in turn; The parameter indicators and frequency band resource data in the system are obtained in the pre-processing stage, specifically: the total length of the current available frequency band is F, F∈[f min ,f max ], all devices in the area D i The internal information of the frequency band, such as working frequency band, bandwidth, time, deployment location, etc., i = 1, 2, 3, ..., m, the protection frequency band f restricted for use within the total frequency band F i and its position L i With time t i Case, i = 1, 2, 3, ..., r, frequency band usage priority between different devices P i , i=1,2,3,...,p, current frequency band occupancy O i , where O i is an array, which records the occupancy of the observed frequency points in the frequency band F with 0 and 1; The calculation of each function matrix is specifically as follows: each function matrix includes an available frequency band matrix M a and the interference matrix M i , where the free space loss formula involved in interference discrimination is as follows: LS = 32.45 + 20lnf + 20lnd; Calculate the interference matrix M i , according to the device D i The information of the available frequency band and working bandwidth is obtained, so that the available frequency band F is divided into k sub-bands and each sub-band is given a specific frequency band number k. i , and then generate an m-row, k-column matrix M of available frequency bands for all devices a , the matrix elements are all binary, that is, 0 represents the device D i Not available in this frequency band, and 1 represents D i Available in this frequency band, where the M a When checking each element, determine whether the available frequency band overlaps with the restricted time and area in the environment. If so, it will be judged as unavailable. Based on the occupancy of the overall frequency band, if the frequency band is already occupied, it will also be judged as unavailable. Calculate the interference matrix M i , according to the device D i The information of the transmission power and receiver sensitivity is obtained according to the available frequency band matrix M a Generate an m×m×k three-dimensional interference matrix M i , where the elements are also binary. When a certain frequency band is fixed, if the device D i , satisfying the free space loss formula below, and the working time of the two devices overlaps, then the element is set to 1 at this time, that is, interference occurs, otherwise it is set to 0, no interference, the free space loss formula is as follows: LS = 32.45 + 20lnf + 20lnd, where the unit of space loss LS is dB, f is the operating frequency of the device, the unit is MHz, and d is the distance between the devices, the unit is km. When the attenuated value of the transmission power of a device is still greater than the receiving sensitivity of the other device, it is determined that the two devices are interfering; The mapping between the pre-processing results and the network elements is completed as follows: Action a t It refers to the specific frequency band selected by the device at a certain moment, and serves as an integral part of the overall plan for subsequent steps; state s t The row vector is composed of the action selected by the user, the reuse of the entire frequency band, and the 0 / 1 signal indicating whether the allocation is successful or not. The reward r t It is a function related to the ratio of failed devices to overall devices in the overall solution obtained after all devices select actions, and is defined as follows: Among them, α and p represent the weight coefficient and priority of device i respectively; The updated search probability p t , specifically: Among them, p t is the search probability at a certain moment, p start With p end are the search probabilities of starting and ending, r decay is its decay rate; The centralized distribution is based on the local distribution method to sequentially distribute the devices D in the environment. i Perform actions, that is, the selection of frequency bands, and finally summarize them into overall actions a t , specifically: each device D to be allocated in the communication environment i According to the search probability p t Conduct a t The judgment and selection of historical status data s t The LSTM network is sent to learn long-distance dependencies, that is, the input historical data is selectively retained or forgotten; the Q network obtains the corresponding value Q according to the output data t , so as to select action a t , and combine the environment to obtain the corresponding reward r t , then the environment state is transferred to s t+1 , and at the same time, the elements of the current moment are (s t ,a t ,r t ,s t+1 ) in the form of a combination and stored in the experience replay pool; Each device D to be allocated in the communication environment i According to the search probability p t Conduct a t The judgment selection is as follows: each device D to be allocated i According to the search probability p t Perform an action t , that is, the judgment and selection of specific available frequency bands, introducing the random quantity r const , if p t ≥r const , in M a Randomly select from the available frequency bands corresponding to ; if the above conditions are not met, it is necessary to select according to the maximum Q value corresponding to the state observation value, and the formula is as follows: a t =argmax a Q(φ(s t ),a i ;i); The Q network obtains the corresponding value Q according to the output data t , so as to select action a t , and combine the environment to obtain the corresponding reward r t , then the environment state is transferred to s t+1 , and at the same time, the elements of the current moment are (s t ,a t ,r t ,s t+1 ) in the form of a combination and storage in the experience replay pool. Specifically, the phased data output by the LSTM network is used as the input data of the subsequent fully connected part of the overall network and the Q value is calculated accordingly. Device D i Each optional frequency band has a Q value corresponding to it; the action at that moment is selected according to the rule, that is, the frequency band scheme a i ; After all devices have completed action selection at this moment, all actions a i Composition of overall action t , according to the formula and status shown below, the final reward r is obtained t : Equipment in the environment D i According to the action, the overall frequency band usage and the signal identification of whether the allocation is successful or not caused by the action selection are updated, and its own state s at the future time is further updated. t+1 ; All devices at the current time and future time D i The states are combined into the overall s t and t+1 , and compare these two data with the a of the overall action plan t and reward r t Merge and store them in the experience replay pool; The overall network iterative learning is specifically as follows: a small sample batch of batch_size is randomly extracted from the historical data of the experience replay pool and recorded as b i , each sample b i are all data bodies composed of four items of data in step 7.5; for each sample b i , extract the internal etc.; multiple samples b i Status in Combine and use as network input, select the frequency band action selected by the device from the many Q values obtained Corresponding value For sample b i , the Q network will calculate the future state The maximum value of Q that may be involved, and using the formula Calculate the target Q value; use the formula based on the target Q value and the current Q value Calculate the loss value of the network and update the network parameters; where N b is batch_size, which means the number of samples extracted in each batch; repeat the above steps until the learning process is completed and the results are generated.
