Electromagnetic space and geographic space combined navigation optimization method for single mobile user

Abstract

The invention provides an electromagnetic space and geographic space combined navigation optimization method for a single mobile user, which comprises the following steps: receiving the current state and terminal point information of a mobile user agent, the current state comprising the current position of the agent, a path obstacle, measurement electromagnetic information and a walkable path; the current state containing the measured electromagnetic information is input to the deep neural network such that the output predicted direction of movement tends to move toward a direction in which the electromagnetic intensity is sufficient to maintain communication. The method not only considers the navigation of the geographic space, but also further considers the combined navigation of the electromagnetism and the geographic space, so that the communication quality is ensured while the mobile user in the space can arrive at the destination more quickly. According to the method, the characteristics of reinforcement learning are combined during navigation model training based on a deep network, and a real electromagnetic map after intelligent agent movement is compared with a prediction map for supervised learning, so that deep reinforcement learning and machine learning are organically combined together.

Description

Technical Field

[0001] The present invention relates to navigation technology for mobile users, and particularly to the combined navigation technology of electromagnetic space and geographical space. Background Art

[0002] Due to the rapid development of wireless communication technology, wireless networks have been widely deployed and become extremely large in scale. For wireless networks, the main function is to provide stable and reliable services for mobile users. In the scenario of simultaneous communication and movement, it is necessary to jointly plan the communication behavior and movement behavior of users in both electromagnetic space and geographical space. Considering the electromagnetic space, if a user is always in an area with a relatively high signal strength during movement, it will help to obtain a more stable network connection and reduce the risk of signal loss and communication interruption; considering the geographical space, it is necessary to ensure that during path planning, the user can avoid obstacles in the environment and find the shortest path as the optimal path.

[0003] The concurrent intertwining of movement and communication complicates the combined navigation problem. For example, in addition to geographical obstacles, users should also avoid areas with weak signals because in these areas, the communication quality of users will seriously decline. Therefore, when navigating, we should consider both the geographical map and the electromagnetic map. Otherwise, if only path planning is considered, encountering areas with weak signals will cause communication interruption for mobile users. On the other hand, if only moving in areas with strong signals is considered without considering path planning, it is difficult for mobile users to reach the destination quickly. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a navigation solution that can enable mobile users in space to reach the destination faster while ensuring communication quality.

[0005] The technical solution adopted by the present invention to solve the above technical problem is to provide an optimization method for combined navigation of electromagnetic space and geographical space for a single mobile user, which receives the current state and end point information of the mobile user agent. The current state includes the current position of the agent, path obstacles, measured electromagnetic information, and available paths; the current state containing the measured electromagnetic information is input into a deep neural network, so that the output predicted movement direction tends to move towards the direction where the electromagnetic intensity is sufficient to maintain communication; during the training process of the deep neural network, the measured electromagnetic information after the agent executes an action is compared with the predicted electromagnetic information for supervised learning.

[0006] The prediction process of the deep neural network includes the following steps:

[0007] Output the predicted electromagnetic information of the next state according to the measured electromagnetic information in the current state;

[0008] The predicted electromagnetic information of the next state is fused with the current state to obtain the joint information of the electromagnetic space and the geographical space;

[0009] Then, the joint coding information is output according to the joint information of the electromagnetic space and the geographical space;

[0010] The joint coding information is input into the Q-network module, and the action corresponding to the maximum value of the Q function output by the Q-network module is used as the predicted moving direction.

[0011] The present invention also provides a joint navigation model of the electromagnetic space and the geographical space for a single moving user based on a deep neural network. The deep neural network realizes the above steps through a prediction network block, a fusion unit, an observation encoder and a Q-network module.

[0012] The Q-network module adopts the architecture of the Dueling DQN dueling network; the Q-value calculation is decomposed into two branches of the state value and the advantage function, which are calculated by two independent fully connected layers FC respectively, and finally the results of the two fully connected layers FC are combined to output the Q-value.

[0013] The training of the deep neural network adopts a double-network structure, and a target deep neural network with the same structure as the deep neural network but with asynchronous parameter updates is introduced.

[0014] The present invention not only considers the navigation of the geographical space, but also further considers the joint navigation of the electromagnetic and geographical spaces, enabling the moving user in the space to reach the destination faster while ensuring the communication quality. The joint navigation problem of the electromagnetic and geographical spaces is modeled as a challenging non-convex problem and solved by algorithms of deep reinforcement learning and supervised learning.

[0015] The beneficial effect of the present invention is that, according to the characteristics of the electromagnetic map (electromagnetic information), a prediction network block is added, enabling the intelligent agent to predict the change of the electromagnetic intensity, so that the intelligent agent can move in the direction of high electromagnetic intensity on the premise of finding the destination. When training the navigation model based on the deep network, combined with the characteristics of reinforcement learning, the real electromagnetic map after the movement of the intelligent agent is compared with the predicted map for supervised learning, so that deep reinforcement learning and machine learning are organically combined. Compared with the existing method that only considers the geographical space navigation, the user can consume less energy in the electromagnetic and geographical joint space and can quickly plan a path for the user in a complex and changing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the navigation model based on the deep network used in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] Consider a wireless communication network where a base station is trying to serve a user and there are two interference sources in the network. We assume that the network area is a square, defined as [x L ,x U ]×[y L ,y U ], where the subscripts L and U represent the lower and upper bounds of the square’s two-dimensional coordinates, respectively. M represents the total number of cells in the network area, and the cell side length is Δs, n=1,..., N represents the time step. Since the mobile user moves one cell in each time step, we define b n ∈{m=1,...,M} is the cell where the user moves in time step n, and m is the cell number. n represents the location of the mobile user at time step n, Represents the trajectory of mobile users.

[0018]

[0019] in is the initial position, x I ,y I is the two-dimensional coordinate of the initial position, is the end position. Indicates the lower boundary of the region and Indicates the upper right boundary of the region, ≤ is the element-by-element inequality sign, Indicates any.

[0020] set up Represents the time step n, from the base station to the unit b n The channel loss, g m represents the channel loss from the interference source in cell m to the base station. The channel loss depends on the large-scale path loss and shadowing of the local environment as well as small-scale fading. Considering the uplink, if the signal to interference plus noise ratio (SINR) transmitted by the mobile user is lower than the threshold γ th , that is, when SINR n <γ th When , the mobile user is considered to be interrupted at time step n:

[0021]

[0022] in Indicates that at time step n, the user arrives at unit b n The power transmitted by the user, |σ n | 2 Represents the noise interference at time step n, usually white noise. m is the power emitted by the interference source in cell m. We define the communication energy consumption as E c ,but

[0023]

[0024] During the movement of the mobile user, we not only need to minimize the energy consumption E of task communication c , but also minimize the weighted distance sum from the user's current position to the end point at each moment, so that it moves towards the end point as much as possible. The navigation modeling optimization problem P0 is as follows:

[0025]

[0026] p bn ≤ P max (4c)

[0027] q0 = q I , q N = q F (4d)

[0028]

[0029] where γ is the discount factor, P max is the maximum transmission power of the user, |||| represents the 2-norm, the side length of the cell is Δs, V n represents the set of feasible movement directions of the mobile user, V n = {1, -1, i, -i}, 1 and -1 represent moving right and left respectively, and i and -i represent moving up and down; represents the feasible movement direction of the mobile user at time step n. is the geographical map, taking values of 0 or 1. 0 means passable. 1 means there is an obstacle. satisfy cond. means that the conditions clearly written in the subsequent cond need to be satisfied. cond∶ represents the condition definition.

[0030] If there is an optimal solution for P0, then the QoS constraint condition equation (4b) in the above formula must satisfy the equality at the optimal solution. Therefore, P0 can be rewritten as P1.

[0031]

[0032] Solve the navigation modeling optimization problem P1:

[0033] Solve P1 using reinforcement learning. First, regard P1 as a Markov decision process MDP with partial observability. MDP is a mathematical framework for modeling sequential decision-making problems, which describes a process in which an agent interacts with the environment through actions in an environment with Markov properties to maximize the cumulative reward. The MDP of the embodiment is represented by a quadruple

[0034] State set Each element of the state set is a state is the information within the perspective range of the agent, including the current position b of the agent n , path obstacles, measured electromagnetic information, and traversable paths.

[0035] The current position of the agent can be displayed using a binary local map, which can display the positions of other agents within the surrounding radius in addition to the current position of the agent.

[0036] The measured electromagnetic information is used to display the spectral information within the visible range around, and can be a continuous normalized electromagnetic map. The path obstacles are used to show the geographical obstacle information within the visible range around, and can be a binary obstacle map. The traversable paths include the information of the paths already traveled and the probabilities of reaching the end point by marking the four moving directions. Specifically, a heuristic algorithm can be used to mark the probabilities of reaching the end point by the four moving directions.

[0037] During the actual training process, it is observed that the agent sways severely, so the paths traveled within the view are provided at the same time. The function is to give a negative reward, that is, a penalty, once it is determined that the agent passes by again.

[0038] Action set The action space corresponds to the user s moving direction and is divided into four directions: forward, backward, left, and right.

[0039] Among them, 1 and -1 represent moving right and left respectively, and i and -i represent moving up and down

[0040] State transition probability The state transition matrix is deterministic because the next position is directly determined by the current position of the agent and the currently selected action.

[0041] Reward set Each reward r in the reward set is determined according to the situation after the agent executes the action:

[0042] 1) If the agent executes the action and can move, a negative reward value R m (q n ) is given;

[0043] 2) If the agent executes the action and the communication is normal, a negative reward value R d (b n ) is given;

[0044] 3) If the agent executes the action and encounters an obstacle, a negative reward value R o is given;

[0045] 4) If the agent executes an action and then the communication is interrupted, a negative reward value R is given n ;

[0046] 5) If the agent executes an action and then it is on a repeated path, a negative reward value R is given c ;

[0047] 6) If the agent executes an action and then reaches the end point, a positive reward value R is given f ;

[0048] During training, in order to enable the agent to reach the destination with fewer steps, in the path planning problem, a positive reward is only given when the destination is reached, and a negative reward is given in all other cases. To ensure that the agent's path is movable and the communication is normal, |R m (q n )| < |R o |, |R d (b n )| < |R n |. The ratio between movability and normal communication can be adjusted according to actual needs. In this embodiment, it is considered that movability and sufficient electromagnetic intensity to maintain communication are equally important. Therefore, R o = R n . After the agent executes an action and does not reach the end point, and is in a state where it can move and the communication is normal but on a repeated path, then r = R m (q n ) + R d (b n ) + R c .

[0049] Specifically, the reward value R m (q n ) = -γ||q n - q F || 2 .

[0050] Specifically, the method for determining whether the communication is interrupted is: According to (5b), obtain the power transmitted by the agent in unit b n . If it is considered that the communication is interrupted, if it is considered that the communication is normal;

[0051] Specifically, the reward value Specifically, for the reward value R c for taking a repeated path, its magnitude can be set within the reward value R m (q n) and the reward value R for encountering an obstacle o Between, |R m (q n )| < |R c | < |R o |, and is proportional to the number of times the agent has passed through the current point.

[0052] By establishing and training a deep neural network for joint electromagnetic space and geospatial navigation for a single mobile user, including a prediction network block, a fusion unit, an observation encoder, and a Q-network module, as Figure 1 shown.

[0053] The prediction network block is used to output the predicted electromagnetic information p of the next state according to the currently measured electromagnetic information d.

[0054] The prediction network block adopted in the embodiment includes two independent convolutional layers, namely the first convolutional layer and the fourth convolutional layer, and a residual block. The first convolutional layer, the residual block, and the fourth convolutional layer are connected in sequence. The input of the residual block is the output of the first convolutional layer. The output of the second convolutional layer and the third convolutional layer is added to the original input of the residual block, and the added result is then activated by ReLU to form the output of the residual block. The first, second, and third convolutional layers are all structures in which a 3*3 convolutional kernel, a batch normalization layer BN, and a ReLU activation are connected in sequence. The fourth convolutional layer is a structure in which a 3*3 convolutional kernel, a batch normalization layer BN, and a Sigmod activation are connected in sequence.

[0055] The measured electromagnetic information d predicts the predicted electromagnetic information p of the next state corresponding to the four actions of up, down, left, and right respectively within the current time step through the prediction network block so that the agent can know the electromagnetic map at the next moment and has the ability to speculate, and it is easier to move in the direction with higher electromagnetic intensity and sufficient communication.

[0056] The fusion unit is used to fuse the predicted electromagnetic information p of the next state with the current state s and then output the joint information c of the electromagnetic space and the geospatial space. The fusion unit in the embodiment is an adder. The current state is the information within the perspective range of the agent, including the current position b n of the agent, path obstacles, measured electromagnetic information d, and passable paths.

[0057] The observation encoder is used to output the joint encoded information e according to the joint information c of the electromagnetic space and the geospatial space.

[0058] The observation encoder used in the embodiment includes two independent convolutional layers, namely the first convolutional layer and the eighth convolutional layer, and three residual blocks. The input of the first residual block is the output of the first convolutional layer. The input of the second residual block is the output of the first residual block. The input of the third residual block is the output of the first residual block. Each residual block includes 2 convolutional layers. The first residual block includes the second convolutional layer and the third convolutional layer. The second residual block includes the fourth convolutional layer and the fifth convolutional layer. The third residual block includes the sixth convolutional layer and the seventh convolutional layer. In each residual block, the output of the input passing through 2 convolutional layers is added to the original input of the residual block, and the added result is then activated by ReLU to form the output of the residual block. Each of the 8 convolutional layers has a structure in which a 3×3 convolutional kernel, a batch normalization layer BN, and a ReLU activation are connected in sequence.

[0059] The Q-network module is used to output the q value according to the joint encoding information e to predict the user's moving direction. The Q-network module is a neural network structure used to approximate the Q function in deep reinforcement learning. The embodiment adopts the architecture of the Dueling DQN dueling network, which decomposes the Q value into a state value and advantages, calculates them through two independent fully connected layers FC, and finally combines and outputs the Q value. The optimal action is selected through the maximum q value.

[0060] To stabilize the training process and avoid the oscillation and divergence of Q value estimation, a double-network structure is adopted for the training of the deep neural network, that is, a target deep neural network with the same structure as the deep neural network but with asynchronous parameter updates is introduced. The deep neural network used after training is called the online deep neural network. The model parameters of the target deep neural network are periodically copied or delayed synchronized from the online deep neural network for calculating the target Q value.

[0061] The embodiment gives the training process of a double-network structure as follows:

[0062] Step 1: Input the state s into the online deep neural network with initial weights.

[0063] First, collect the historical experience of the interaction between the agent to be planned and the environment and store it in the experience replay memory. The historical experience is quadruple information. s is the current state. is the executed action, r is the reward corresponding to the action. s ′ is the executed action. The next state after the action;

[0064] The acquisition process of the historical experience is as follows: The agent has a probability of ∈ to randomly select an action from the action set. There is a 1 - ∈ possibility according to Select the optimal action where ω is the weight of the target deep neural network. The target deep neural network outputs the maximum Q - function value q based on the input s and d in s, and thus selects the optimal execution action of the agent The agent evaluates the action to obtain the reward r, and then samples the next state s obtained after executing the action ′ , continuously samples the historical experience of the agent to be planned interacting with the environment, and stores it in the experience replay memory to obtain the training sample set D.

[0065] Step 2: Randomly sample quadruple information to update the deep neural network

[0066] First, randomly sample quadruple information from the training sample set D in the experience replay memory and input it into the deep neural network and the target deep neural network, and perform the following processing:

[0067] where initially, the parameters of the deep neural network and the target deep neural network are the same;

[0068] Substitute the state s into the deep neural network for a feed - forward operation to obtain the predicted estimated Q - values corresponding to all available actions

[0069] Substitute the state s corresponding to the state s in its quadruple information ′ into the target deep neural network for a feed - forward operation to calculate the maximum Q - function value output by the network where is the action corresponding to the maximum Q - value when the state s ′ is substituted into the target deep neural network, and ω - is the weight of the observation encoder and the Q - network module; obtain the target value where γ is the discount factor.

[0070] Construct the loss function according to the outputs of the deep neural network and the target deep neural network as:

[0071] Compare the electromagnetic information p with the electromagnetic information s after the agent executes the action p for supervised learning, and the loss function is: where ω p is the weight of the prediction network block.

[0072] The random gradient descent method is applied to iteratively update the weights of the deep neural network and the target deep neural network, where the weights in the deep neural network are updated in real time, while the weights in the target deep neural network are updated every set time step; after reaching the iteration steps, the trained deep neural network is obtained.

Claims

1. An electromagnetic space and geospatial joint navigation optimization method for a single mobile user, characterized in that Receive the current state and destination information of the mobile user agent. The current state includes the current position of the agent, path obstacles, measured electromagnetic information, and available paths. Input the current state containing the measured electromagnetic information into the deep neural network, so that the output predicted moving direction tends to move towards the direction where the electromagnetic intensity is sufficient to maintain communication. During the training process of the deep neural network, the measured electromagnetic information after the agent executes an action is compared with the predicted electromagnetic information for supervised learning. The prediction process of the deep neural network includes the following steps: Output the predicted electromagnetic information of the next state according to the measured electromagnetic information in the current state. Fuse the predicted electromagnetic information of the next state with the current state to obtain the joint information of the electromagnetic space and the geographical space. Then output the joint coding information according to the joint information of the electromagnetic space and the geographical space. Input the joint coding information into the Q-network module, and take the action corresponding to the maximum value of the Q function output by the Q-network module as the predicted moving direction.

2. The method according to claim 1, wherein The current position where the agent is located is displayed using a binary local map. The measured electromagnetic information is used to display the spectrum information within the visible range around, and is a continuous normalized electromagnetic map. The path obstacles are used to show the geographical obstacle information within the visible range around, and are a binary obstacle map. The available paths include the information of the paths already traveled and the possibilities of reaching the destination in the four moving directions of up, down, left, and right.

3. The method according to claim 1, characterized in that, During the training process of the deep neural network, the reward r is obtained by evaluating the actions executed by the agent, and the training process is constrained by making the reward r as large as possible. The reward r is determined according to the specific situation after the agent executes an action. The rules are as follows: 1) The agent executes an action If it can move after that, a negative reward value R is given m (q n ); 2) The agent executes an action If the communication is normal after that, a negative reward value R is given d (b n ) 3) The agent executes an action If an obstacle is encountered afterwards, a negative reward value R is given o ; 4) The agent executes an action If the communication is interrupted afterwards, a negative reward value R is given n ; 5) The agent executes an action If it is a repeated path that has been walked, then a negative reward value R is given c ; 6) The agent executes an action and reaches the end point, then a positive reward value R is given f .

4. The method according to claim 3, wherein Reward r rule satisfaction: |R(b n )| < |R n |; |R m (q n )| < |R c | < |R o |; R o = R n .

5. The method according to claim 3, wherein The method for determining whether communication is interrupted is: obtain the transmission power of the agent at the current location When it is considered that communication is interrupted. When it is considered that communication is normal; P max is the maximum transmission power of the user, γ th is a preset threshold, M represents the total number of cells in the network area, m is the cell serial number, b n is the cell where the user is located at time step n, represents the channel loss from the base station to cell b at time step n n The channel loss of the interference source in cell m to the base station, g m represents the channel loss from the interference source in cell m to the base station, P m is the transmission power of the interference source in cell m, |σ n | 2 represents the noise interference at time step n.

6. The method according to claim 5, wherein Reward value R m (q n )=-γ||q n -q F || 2 ; Reward value where γ is the discount factor, q n represents the location of the mobile user at time step n, and q F represents the termination location.

7. Electromagnetic space and geospatial joint navigation model for a single mobile user, characterized in that, The model is based on a deep neural network, receives the current state and destination information of the mobile user agent. The current state includes the current position of the agent, path obstacles, measured electromagnetic information, and available paths. Input the current state containing the measured electromagnetic information into the deep neural network, so that the output predicted moving direction tends to move towards the direction where the electromagnetic intensity is sufficient to maintain communication. During the training process of the deep neural network, the measured electromagnetic information after the agent executes an action is compared with the predicted electromagnetic information for supervised learning. The deep neural network includes a prediction network block, a fusion unit, an observation encoder, and a Q-network module. The prediction network block is used to receive the measured electromagnetic information in the input current state and output the predicted electromagnetic information of the next state. The fusion unit is used to fuse the predicted electromagnetic information of the next state with the current state and output the joint information of the electromagnetic space and the geographical space. The observation encoder is used to receive the input joint information of the electromagnetic space and the geographical space and output the joint coding information. The Q-network module is used to receive the input joint coding information, output the action corresponding to the maximum value of the Q function, and take this action as the predicted moving direction.

8. The navigation model according to claim 7, wherein The Q-network module adopts the architecture of the Dueling DQN duel network; the Q-value calculation is decomposed into two branches of the state value and the advantage function, which are calculated by two independent fully connected layers FC respectively, and finally the results of the two fully connected layers FC are merged to output the Q-value.

9. The navigation model according to claim 8, wherein, The training of the deep neural network adopts a dual-network structure, and a target deep neural network with the same structure as the deep neural network but with asynchronous parameter updates is introduced.

10. The navigation model according to claim 9, wherein, During the training process of the deep neural network, the reward r is obtained by evaluating the actions executed by the agent, and the training process is constrained by maximizing the reward r as much as possible; The reward r is determined according to the specific situation after the agent executes the action, and the rules are as follows: 1) The agent executes an action If it can move after that, a negative reward value R is given m (q n ); 2) The agent executes an action If the communication is normal after that, a negative reward value R is given d (b n ) 3) The agent executes an action If an obstacle is encountered afterwards, a negative reward value R is given o ; 4) The agent executes an action If the communication is interrupted afterwards, a negative reward value R is given n ; 5) The agent executes an action If it is a repeated path of walking afterwards, a negative reward value R is given c ; 6) The agent executes an action and reaches the end point, then a positive reward value R is given f .

Images