Sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization

Abstract

The invention discloses a sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization. The method comprises the following steps: constructing a target function by taking the maximum sensor coverage rate, the minimum sensor radius and the minimum sensor number as targets; and solving the target function by adopting an improved multi-task algorithm to obtain the optimal coverage rate, radius and number of the sensors. According to the method, an appropriate task is selected as a source task to perform information exchange with a target task through deep reinforcement learning, so that the possibility of negative migration in the process is reduced; when knowledge migration occurs, individuals in a population are adjusted through a domain self-adaption mechanism, so that a source task is converted into a target task, and the domain self-adaption method effectively reduces the distribution difference between tasks, reduces the negative migration risk and improves the optimization efficiency. Therefore, efficient optimization of sensor network coverage is realized through dynamic task selection, feature distribution alignment and shape similarity adjustment, and the coverage efficiency of the sensor network is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of sensor network optimization methods, and particularly relates to a sensor coverage optimization method based on deep reinforcement learning and domain adaptation multi-task optimization. Background Art

[0002] The sensor coverage problem (SCP) is one of the core problems in sensor network optimization, aiming to maximize the coverage efficiency of the target area by reasonably deploying sensor nodes. Traditional methods usually adopt heuristic algorithms or single-task optimization strategies, but face the following challenges in complex scenarios: 1) Task differences: The feature distributions of sensor deployment tasks in different regions may vary, resulting in low knowledge transfer efficiency; 2) Dynamic environment: The sensor network environment changes dynamically, and traditional static optimization methods are difficult to adapt to; 3) Negative transfer risk: The mismatched feature distributions between tasks may lead to negative transfer, reducing the optimization effect.

[0003] In the same area, different tasks can be divided according to the number of sensors. It can be found that there is a certain similarity between tasks. Tasks can learn each other's sensor coverage positions and sensor coverage radii to enhance coverage and optimize resources. Multi-task optimization algorithms study the use of populations for optimization search, and achieve the purpose of simultaneous processing of multiple tasks by migrating genetic information between tasks. Then, by using the shared knowledge between multiple optimization tasks, the convergence of each task is promoted. Therefore, it has natural advantages in solving multi-task optimization problems composed of problems such as different numbers of sensors or different regions as described above. However, for traditional multi-task optimization algorithms when solving problems with low similarity and large differences in dimensions after coding, the excessive knowledge transfer between tasks will lead to negative transfer (that is, the transfer causes negative impacts and slows down the time for the algorithm to find the optimal solution). Summary of the Invention

[0004] The purpose of the present invention is to provide a sensor coverage optimization method based on deep reinforcement learning (DRL) and domain adaptation (DA) multi-task optimization (MTO), which solves the problem of low coverage efficiency of the target area in the existing sensor node deployment.

[0005] The technical solution adopted by the present invention is: A sensor coverage optimization method based on deep reinforcement learning and domain adaptation multi-task optimization, comprising the following steps:

[0006] Step 1: Construct an objective function with the goals of maximizing the sensor coverage rate, minimizing the sensor radius, and minimizing the number of sensors;

[0007] Step 2: Solve the objective function constructed in Step 1 using an improved multi-task algorithm to obtain the optimal coverage rate, radius, and number of sensors.

[0008] The features of the present invention also lie in that

[0009] The objective function constructed in Step 1 is:

[0010]

[0011] where x i , y i , and r i respectively represent the abscissa, ordinate, and coverage radius of the i-th sensor; A is the given coverage area; a is the penalty coefficient for insufficient coverage, representing the cost per unit area not covered; c is the cost coefficient, representing the basic cost of each sensor; b is the variable cost coefficient related to the coverage radius; and K is the number of sensors.

[0012] The constraint conditions of the objective function in Step 1 are:

[0013] -1 < x i < 1, i = 1, …, K

[0014] -1 < y i < 1, i = 1, …, K

[0015] 0.1 < r i < 0.25, i = 1, ..., K.

[0016] Step 2 specifically includes the following steps:

[0017] Step 2.1: Set that each individual in the multi-task algorithm represents a set of sensor deployment schemes, optimize the coverage tasks with different numbers of sensors through knowledge sharing and migration, multiple individuals form a population, perform population initialization in the decision space, and the individual encoding is a sequence of sensor coordinates and radii;

[0018] Step 2.2: Set the number of population iterations, based on the source task selection strategy of reinforcement learning, input the sensor parameter encoding sequence of the individuals in the population into the deep Q network, output the value of each action, and select the action as the source task matched by the current individual through the ε-greedy strategy;

[0019] Step 2.3: Generate a corresponding adjusted population between the target task and the remaining tasks through the method of domain adaptation, and the adjusted population is used to optimize the distribution of sensor coordinates and radii to reduce the difference in sensor deployment between different tasks;

[0020] Step 2.4: The individuals of the population start iterative evolution. For the source tasks obtained through Step 2.2, if they belong to the same task as the target task, randomly select an individual from the target task population; if they belong to different tasks from the target task, randomly select an individual from the corresponding adjusted population generated in Step 2.3. After an individual is selected, use the mutation strategy DE / current-to-best / 1 to mutate the individual to obtain a new offspring individual containing sensor coverage information, and then perform a crossover operation on the offspring individual to obtain an offspring population.

[0021] Step 2.5: Calculate the fitness difference between the parent and offspring as the immediate reward. The fitness is jointly calculated by the sensor coverage rate, the total radius, and the quantity. Use the reward to update the network parameters of the deep Q-network and optimize the task selection strategy.

[0022] Step 2.6: Merge the parent and offspring populations, sort them by fitness, and select excellent individuals to form a new population. After the end, if the iteration times are reached, enter Step 2.7; if the iteration times are not reached, enter Step 2.2.

[0023] Step 2.7: Decode the individuals in the final excellent population, linearly map the encoded values to the actual coordinate range and radius range, and output the sensor deployment positions and radii.

[0024] Step 2.2 specifically includes the following steps:

[0025] Step 2.2.1: Linearly normalize all the dimensional values of each individual in the population, that is, the sensor coordinate and radius encoding sequence, and map it to the interval [0, 1].

[0026] Step 2.2.2: Set the number of nodes in the input layer of the deep Q-network to be equal to the total number of sensor parameters, that is, 3 times the number of sensors, and set the number of nodes in the output layer of the deep Q-network to be equal to the number of optional source tasks.

[0027] Step 2.2.3: Input the normalized individual encodings into the deep Q-network in batches, and calculate the Q-values of each individual corresponding to each source task.

[0028] Step 2.2.4: For each individual, select the column index corresponding to the maximum value from the Q-value matrix output by the deep Q-network through the ε-greedy strategy as the source task matched by the corresponding individual. The source task of each individual corresponds to a deployment optimization strategy for a certain number of sensors.

[0029] Step 2.3 specifically includes the following steps:

[0030] Step 2.3.1: Use the CORAL method to align the covariance matrices of the source task and the target task to make the sensor coordinate distribution characteristics consistent among different tasks:

[0031]

[0032] Among them, * represents the source and the target, n is the number of individuals in the population, and X c is the centered matrix, and Σ is the covariance matrix;

[0033] Then, perform singular value decomposition on the covariance matrices of the source and the target respectively to obtain the eigenvector matrix U s and U t and the eigenvalue diagonal matrix Λ s and Λ t , and then use the source diagonal matrix Λ s to calculate the inverse square root of the covariance matrix, and use the target diagonal matrix Λ t to calculate the square root of the covariance matrix:

[0034]

[0035] Transform the matrix of the source task through the inverse square root and square root matrices to obtain the new intermediate matrix Y s :

[0036]

[0037] Step 2.3.2. Adjust the distribution shape between tasks through Procrustes analysis to optimize the global layout of sensor coordinates:

[0038] First, through the formula to equalize the source data matrix X S and the target data matrix X T to obtain the centered matrices and

[0039] Then, through SVD decomposition to obtain UDV T , where U and V are orthogonal matrices:

[0040]

[0041] Calculate the rotation matrix R:

[0042] R = VU T

[0043] Use R to perform rotation and scaling transformation on to obtain the optimally aligned matrix X new :

[0044]

[0045] Step 2.3.3. Integrate the results of CORAL and Procrustes, and use R to perform operations on Ys Perform rotation and scaling transformations on Y s and X new Perform weighted averaging to generate the final adjusted population.

[0046] The mutation strategy DE / current-to-best / 1 expression adopted in Step 2.4 is:

[0047]

[0048] where is to randomly select an individual among the top p*Np individuals in terms of fitness in the target population, x i,g is the i-th individual in the g-th generation population, F i is the mutation factor of the i-th individual, Np is the population size, p is a given ratio, is an individual randomly selected from the target population, is an individual randomly selected from the target population and its external archive set.

[0049] In Step 2.5, calculating the fitness difference between the parent and offspring as the immediate reward r, which is the reward immediately obtained after taking action a in state s:

[0050]

[0051] r = sign(r′)·log(1 + abs(r′))

[0052] where r′ is the fitness difference between the parent individual and the offspring individual, sk parent and sk child are the skill factors of the parent and offspring respectively, Fit(*) represents calculating the fitness value of this individual, and p is the p%-th individual in the population with the same offspring skill factor.

[0053] The beneficial effects of the present invention are as follows: The sensor coverage optimization method based on deep reinforcement learning and domain adaptation multi-task optimization of the present invention selects appropriate tasks as source tasks to exchange information with target tasks through deep reinforcement learning, reducing the possibility of negative transfer occurring in this process; when knowledge transfer occurs, the individuals in the population are adjusted through the domain adaptation mechanism to transform the source task into the target task. The domain adaptation method effectively reduces the distribution difference between tasks, reduces the negative transfer risk, and improves the optimization efficiency. Thus, through dynamic task selection, feature distribution alignment, and shape similarity adjustment, the efficient optimization of sensor network coverage is achieved, significantly improving the coverage efficiency of the sensor network. Brief Description of the Drawings

[0054] Figure 1It is a schematic flowchart of the sensor coverage optimization method based on deep reinforcement learning and domain - adaptive multi - task optimization of the present invention;

[0055] Figure 2 It is the coverage map of the sensor coverage problem of MTO - DQNHD in the present invention with n1 = 27 and n2 = 28;

[0056] Figure 3 It is the coverage map of the sensor coverage problem of MFEA in the present invention with n1 = 27 and n2 = 28;

[0057] Figure 4 It is the coverage map of the sensor coverage problem of MFDE in the present invention with n1 = 27 and n2 = 28;

[0058] Figure 5 It is the coverage map of the sensor coverage problem of MFPSO in the present invention with n1 = 27 and n2 = 28;

[0059] Figure 6 It is the coverage map of the sensor coverage problem of ASCMFDE in the present invention with n1 = 27 and n2 = 28;

[0060] Figure 7 It is the coverage map of the sensor coverage problem of BLKT - DE in the present invention with n1 = 27 and n2 = 28;

[0061] Figure 8 It is the schematic diagram of decoding the solution to the instance problem in the unified decision space of the present invention. Detailed implementation manners

[0062] The present invention will be described in detail below with reference to the accompanying drawings and specific implementation manners.

[0063] Example 1

[0064] The present invention provides a sensor coverage optimization method based on deep reinforcement learning and domain - adaptive multi - task optimization, including constructing an objective function with the maximum sensor coverage rate, the minimum sensor radius, and the minimum number of sensors as the goals; using an improved multi - task algorithm to solve the objective function to obtain the optimal sensor coverage rate, radius, and number. The present invention selects appropriate tasks as source tasks through deep reinforcement learning to exchange information with target tasks, reducing the possibility of negative transfer occurring in this process; when knowledge transfer occurs, the individuals in the population are adjusted through the domain - adaptive mechanism, enabling the source tasks to transform into target tasks. The domain - adaptive method effectively reduces the distribution differences between tasks, reduces the risk of negative transfer, and improves the optimization efficiency. Thus, through dynamic task selection, feature distribution alignment, and shape similarity adjustment, the efficient optimization of sensor network coverage is achieved, significantly improving the coverage efficiency of the sensor network.

[0065] Example 2

[0066] The present invention provides a sensor coverage optimization method based on deep reinforcement learning and domain - adaptive multi - task optimization, which is applicable to coverage enhancement and resource optimization in large - scale sensor networks. The specific steps are as follows;

[0067] S1: Construct an objective function with the goals of maximizing the sensor coverage rate, minimizing the sensor radius, and minimizing the number of sensors:

[0068]

[0069] Subject to - 1 < x i < 1, i = 1, …, K

[0070] - 1 < y i < 1, i = 1, …, K

[0071] 0.1 < r i < 0.25, i = 1, ..., K

[0072] In formula (1), where (x i , y i , r i ) represent the abscissa, ordinate, and coverage radius of the i - th sensor respectively. A is the given coverage area [-1, 1]×[-1, 1]. a is the penalty coefficient for insufficient coverage, representing the cost per unit area not covered, emphasizing the importance of the covered area. When the coverage ratio is low, the value of the objective function will increase significantly. c is the cost coefficient, representing the basic cost of each sensor. As the number of sensors increases, this fixed cost will also increase linearly. b is the variable cost coefficient related to the coverage radius, punishing larger coverage radii because a larger radius may mean higher costs or higher energy consumption. K is the number of sensors, and different tasks can be formed according to different K values. In the present invention, a = 1000, c = 1, and b = 10 are respectively set.

[0073] S2: Use an improved multi - task algorithm to solve the objective function to obtain the optimal coverage rate, radius, and number of sensors.

[0074] Example 3

[0075] The present invention provides a sensor coverage optimization method based on deep reinforcement learning and domain - adaptive multi - task optimization. On the basis of Example 2, S2 is preferably:

[0076] S201: In the improved multi - task algorithm, an individual carries two or more policy solutions. Taking two tasks as an example, they respectively correspond to the solution of the sensor coverage optimization problem with n1 sensors and the solution of the sensor coverage optimization problem with n2 sensors. Multiple individuals form a population, and population initialization is performed in the decision space.

[0077] S202: Set the number of population iterations. Based on the source task selection strategy of reinforcement learning, input each dimension value of the individual into the neural network to obtain the source task that the current individual best matches;

[0078] The goal of reinforcement learning is to find a policy π that selects an action a in a given state s to maximize the cumulative expected return. Define the optimal action - value function Q * (s,a) as the maximum expected return that can be achieved by following any policy, seeing a certain sequence s and then taking some action a. Where π is a sequence of policy mappings for actions (or action distributions).

[0079] The optimal action - value function follows the Bellman equation. According to the Bellman equation, Q * (s,a) can be rewritten in the following form:

[0080]

[0081] Select the source task through deep reinforcement learning. The neural network function approximator with weight θ is called the Q - network. Input the individual into the Q - network to obtain the action value of each action, and the action represents the selection of the source task. So select the action with the highest action value as the output. The Q - network can be trained by minimizing the loss function L i (θ i ) sequence:

[0082] L i (θ i ) = E s,a～ρ(·) [(y i - Q(s,a;θ i )) 2 (3)

[0083] Where y i = E s′～S [r + γmax a′ Q(s′,a′;θ i-1 )|s,a], ρ(s,a) is the probability distribution of the sequence s and the action a, which is called the behavior distribution. Differentiate the loss function with respect to the weight, and the present invention obtains the following gradient to adjust the network. Through continuous iteration, a gradually stable Q - network is finally obtained.

[0084]

[0085] S203: Generate corresponding adjusted populations between pairwise tasks through domain adaptation methods to produce offspring during knowledge transfer;

[0086] The domain adaptation strategy includes the CORAL (Correlation Alignment) algorithm and Procrustes analysis. The CORAL (Correlation Alignment) method is used to align the covariance matrices of the source task and the target task to reduce the difference in feature distributions. Combining Procrustes analysis to adjust the distribution shapes between tasks further reduces the risk of negative transfer.

[0087] CORAL is expressed as:

[0088]

[0089] where * represents source and target, n is the number of individuals in the population, and X c is the centered matrix, and Σ is the covariance matrix. Perform singular value decomposition on the covariance matrices of the source and target respectively to obtain the eigenvector matrices U s and U t as well as the eigenvalue diagonal matrices Λ s and Λ t . Then use the source diagonal matrix Λ s to calculate the inverse square root of the covariance matrix, and use the target diagonal matrix Λ t to calculate the square root of the covariance matrix:

[0090]

[0091] Transform the matrix of the source task through the inverse square root and square root matrices to obtain a new intermediate matrix Y s , making the transformed source data closer to the target data in terms of feature distribution.

[0092]

[0093] The adjusted new matrix not only retains part of the distribution information of the source task as knowledge but also makes the distribution more similar to the target task, reducing the probability of negative transfer.

[0094] Procrustes analysis includes three steps: translation, rotation, and scaling, and is expressed as:

[0095] Translation: Mean center the source data matrix X S and the target data matrix X T through formula 5 to obtain the centered matrices and

[0096] Rotation and Scaling: Decomposition via SVD Get UDV T , where U and V are orthogonal matrices:

[0097]

[0098] Calculate the rotation matrix R:

[0099] R=VU T (11)

[0100] Use R to Perform rotation and scaling transformations to obtain the optimal alignment matrix X new :

[0101]

[0102] S204: The population individuals begin iterative evolution. The source task is obtained through S202. If it is the same task as the target task, an individual is randomly selected from the target task population. If it is a different task from the target task, an individual is randomly selected from the adjusted population generated in S203. After the individual is selected, the mutation strategy DE / current-to-best / 1 is used to mutate the population to obtain the offspring individuals. Then the offspring population is crossovered to obtain the offspring population;

[0103] The mutation strategy DE / current-to-best / 1 in S204 is expressed as:

[0104]

[0105] in, To randomly select an individual with the top p*Np (Np is the population size) in fitness ranking in a target population, where p is a given proportion (set P to [5%, 25%]). is a randomly selected individual from the target population, The target population and the individuals randomly selected from its external archive set are stored in the external archive A, which stores the individuals with poor fitness after mutation and crossover in each generation of evolution. Its size is fixed at NP, and individuals are randomly deleted when the size exceeds the limit.

[0106] S205: Generate rewards based on the changes in the fitness of the parent and offspring individuals to quantify the degree of improvement, and use them to guide the selection of reinforcement learning tasks, guiding DQN to prioritize positive transfer actions;

[0107] The immediate reward r in S205 is the reward obtained immediately after taking action a in state s.

[0108]

[0109] r = sign(r′)·log(1 + abs(r′)) (15)

[0110] sk parent and sk child are the skill factors of the parent and offspring respectively. Fit(*) represents calculating the fitness value of this individual, and p is the p% - th individual in the population with the same offspring skill factor.

[0111] S206: Offspring selection. The offspring population and the initial population are merged, and an excellent offspring population is selected. After that, if the iteration number is reached, go to S207; if the iteration number is not reached, go to S202.

[0112] S207: Decode the individuals in the excellent offspring population to obtain the sensor coverage information in the individuals.

[0113] Example 4

[0114] The present invention provides a sensor coverage optimization method based on deep reinforcement learning and domain - adaptive multi - task optimization. As Figure 1 shown, for the method of the sensor coverage problem of the multi - task algorithm, the specific steps include:

[0115] S1: Construct an objective function with the maximum sensor coverage rate, the minimum sensor radius, and the minimum number of sensors as the goals:

[0116]

[0117] Subject to - 1 < x i < 1, i = 1, …, K

[0118] - 1 < y i < 1, i = 1, …, K

[0119] 0.1 < r i < 0.25, i = 1,..., K

[0120] In formula (1), where (x i , y i , r i ) represent the abscissa, ordinate, and coverage radius of the i - th sensor respectively. A is the given coverage area [-1, 1]×[-1, 1]. a is the penalty coefficient for coverage shortage, representing the cost per unit area not covered. c is the cost coefficient, representing the basic cost of each sensor. b is the variable cost coefficient related to the coverage radius, penalizing a larger coverage radius. K is the number of sensors, and different tasks can be formed according to different K values.

[0121] S2: Solve the objective function using an improved multi-task algorithm to obtain the optimal coverage rate, radius, and number of sensors.

[0122] Example 5

[0123] The present invention provides a sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization. On the basis of Example 4, S2 is preferably:

[0124] S201: In the improved multi-task algorithm, each individual carries two or more policy solutions. Taking two tasks as an example, they respectively correspond to the solution of n1 sensor coverage optimization problems and the solution of n2 sensor coverage optimization problems. Multiple individuals form a population, and population initialization is performed in the decision space.

[0125] S202: Set the number of population iterations. Based on the source task selection strategy of reinforcement learning, input each dimension value of the individual into the neural network to obtain the source task most matching the current individual;

[0126] The goal of reinforcement learning is to find a policy π that selects an action a in a given state s to maximize the cumulative expected return. The standard assumption made by the present invention is that future rewards are discounted by γ at each time step, and the future discounted return at time t is defined as Define the optimal action value function Q * (s,a) as the maximum expected return that can be achieved by following any policy, seeing a certain sequence s and then taking certain actions a where π is a sequence of policy mappings for actions (or action distributions).

[0127] The optimal action value function follows the Bellman equation. For the Bellman equation of the action value function Q, it describes the expected return of taking a certain action in a given state, and this return consists of two parts:

[0128] The immediate reward r in S205, which is the reward immediately obtained after taking action a in state s.

[0129]

[0130] r = sign(r′)·log(1 + abs(r′)) (3)

[0131] sk parent and sk child are the skill factors of the parent and offspring respectively. Fit(*) represents calculating the fitness value of this individual. p is the p%th individual in the population with the same skill factor of the offspring.

[0132] According to the Bellman equation, Q * (s,a) can be rewritten in the following form:

[0133]

[0134] In S202, the source task is selected through deep reinforcement learning. In the present invention, the neural network function approximator with weight θ is called the Q-network. The individual is input into the Q-network to obtain the action value of each action, and the action represents the selection of the source task. Therefore, the action with the highest action value is selected as the output. The Q-network can train by minimizing the loss function L i (θ i ) sequence:

[0135] L i (θ i ) = E s,a～ρ(·) [(y i - Q(s,a;θ i )) 2 (5)

[0136] where y i = E s′～S [r + γmax a′ Q(s′,a′;θ i-1 )|s,a], ρ(s,a) is the probability distribution of the sequence s and the action a, which is called the behavior distribution. Differentiate the loss function with respect to the weight to obtain the following gradient to adjust the network, and finally obtain a gradually stable Q-network through continuous iteration.

[0137]

[0138] S203: Generate corresponding adjustment populations between tasks pairwise through the method of domain adaptation to generate offspring during knowledge transfer;

[0139] The domain adaptation strategy includes the CORAL (Correlation Alignment) algorithm and Procrustes analysis. The CORAL (Correlation Alignment) method is used to align the covariance matrices of the source task and the target task to reduce the difference in feature distributions. Combining Procrustes analysis to adjust the distribution shape between tasks further reduces the risk of negative transfer.

[0140] CORAL is expressed as:

[0141]

[0142] where * represents the source and the target, n is the number of individuals in the population, X cis the centralized matrix, and Σ is the covariance matrix. Perform singular value decomposition on the covariance matrices of the source and the target respectively to obtain the eigenvector matrices U s and U t as well as the eigenvalue diagonal matrices Λ s and Λ t . Then use the source diagonal matrix Λ s to calculate the inverse square root of the covariance matrix, and use the target diagonal matrix Λ t to calculate the square root of the covariance matrix:

[0143]

[0144] Transform the matrix of the source task through the inverse square root and square root matrices to obtain the new intermediate matrix Y s , making the transformed source data closer to the target data in terms of feature distribution.

[0145]

[0146] The adjusted new matrix not only retains part of the distribution information of the source task as knowledge but also makes the distribution more similar to the target task, reducing the probability of negative transfer.

[0147] Procrustes analysis consists of three steps: translation, rotation, and scaling, expressed as:

[0148] Translation: Mean-center the source data matrix X S and the target data matrix X T through formula 7 to obtain the centralized matrices and

[0149] Rotation and Scaling: Obtain UDV through SVD decomposition T , where U and V are orthogonal matrices:

[0150]

[0151] Calculate the rotation matrix R

[0152] R = VU T (13) Rotate and scale using R to obtain the optimally aligned matrix X new

[0153]

[0154] S204: The individuals of the population start iterative evolution. The same mutation strategy as JADE is adopted to mutate the population to obtain the offspring population, and then a crossover operation is performed on the offspring population to obtain the offspring population.

[0155] S205: Rewards are generated according to the changes in the fitness of the parent and offspring individuals to quantify the improvement degree, and are used to guide the selection of reinforcement learning tasks, guiding DQN to preferentially select positive migration actions.

[0156] S206: Offspring selection. The offspring population and the initial population are combined, and the excellent offspring population is selected. After the end, if the iteration times are reached, enter S207; if the iteration times are not reached, enter S202.

[0157] S207: Decode the individuals in the excellent offspring population to obtain the sensor coverage information in the individuals. Among them, when decoding the individuals in the excellent offspring population, since the solutions are generated in a unified search space, and their range is [0, 1], they may not directly conform to the standard form of a specific sensor coverage problem, so a decoding operation needs to be performed on them to obtain the sensor coverage information in the individuals.

[0158] Since the algorithm uses a unified representation method and encodes each solution with a random key between 0 and 1, and since sensor coverage requires three variables: the sensor position coordinates and the sensor coverage radius. Therefore, the solution should be a group of numbers that is 3 times the number of sensors. Figure 8 , showing a simple example of the arrangement rule. For the solution x = (0.42, 0.85, 0.61, 0.53, 0.36, 0.48). The decoded solution x = (-0.16, 0.70, 0.19, 0.06, -0.28, 0.17), which represents the coverage of two sensors. Among them, the first sensor is deployed at (-0.16, 0.7), and its radius is 0.19, while the second sensor is deployed at (0.06, -0.28), with a radius of 0.17. The above solution only takes two sensors as an example.

[0159] Example 6

[0160] In the simulation experiment of the present invention, (MTO-DQNHD) is selected for comparison with MFEA and MFDE. 25 - 35 sensor problems are selected from the SCP problem, and a total of 11 groups of tasks are processed simultaneously.

[0161] As Figures 2 to 7 shown, 11 sensor problems are selected from the SCP problem to compare MTO-DQNHD and the comparative algorithms MFEA and MFDE, and the respective sensor coverage situations are obtained.

[0162] Table 1 shows 11 test problems in the SCP problem, which are used to test the comparative experiment of the present invention in the n-sensor coverage problem and two other algorithms.

[0163] Table 1 11 test problems in the SCP problem

[0164]

[0165]

[0166] Through experiments, the coverage information of MTO-DQNHD in the sensor coverage problem SCP with n = 27 is as follows:

[0167] {(-0.05, -0.85, 0.24), (-0.83, -0.10, 0.24), (-0.55, 0.47, 0.24),

[0168] (0.81, 0.83, 0.24), (0.75, -0.94, 0.24), (0.42, -0.27, 0.24),

[0169] (-0.56, 0.07, 0.24), (0.45, 0.76, 0.24), (0.79, -0.65, 0.24),

[0170] (-0.43, 0.94, 0.24), (-0.78, -0.47, 0.24), (0.83, 0.43, 0.25),

[0171] (0.10, -0.00, 0.24), (0.06, 0.93, 0.24), (-0.18, 0.17, 0.25),

[0172] (0.00, -0.40, 0.24), (-0.77, -0.86, 0.24), (-0.37, -0.71, 0.24),

[0173] (0.79, -0.39, 0.24), (0.16, 0.47, 0.25), (-0.21, 0.61, 0.24),

[0174] (-0.36, -0.25, 0.24), (-0.92, 0.32, 0.24), (0.32, -0.75, 0.24),

[0175] (-0.85, 0.78, 0.24), (0.84, 0.00, 0.24), (0.49, 0.21, 0.24)}

[0176] When the coverage area is [-1, 1] × [-1, 1], the sensor radius is (0, 0.25], and the coordinate positions and coverage radii of 27 sensors are as shown in the above sensor coverage information, such coverage minimizes the total cost finally obtained.

[0177] The following is the coverage information of MTO-DQNHD for the problem of n = 28 in the sensor coverage problem SCP:

[0178] {(-0.37, 0.50, 0.25), (-0.80, -0.77, 0.24), (-0.89, -0.29, 0.25),

[0179] (0.19, 0.96, 0.25), (0.78, -0.81, 0.24), (0.50, -0.47, 0.25),

[0180] (-0.58, -0.31, 0.25), (0.84, 0.80, 0.25), (-0.22, -0.41, 0.25),

[0181] (-0.57, 0.95, 0.25), (0.81, 0.40, 0.24), (-0.23, 0.04, 0.24),

[0182] (-1.00, 0.86, 0.21), (0.45, 0.29, 0.24), (1.00, 0.45, 0.24),

[0183] (0.79, -0.05, 0.25), (-0.43, -0.79, 0.24), (0.03, 0.21, 0.25),

[0184] (-0.63, 0.13, 0.25), (0.42, -0.07, 0.25), (0.49, 0.75, 0.24),

[0185] (0.09, 0.55, 0.24), (-0.99, 0.16, 0.25), (0.36, -0.87, 0.25),

[0186] (0.03, -0.75, 0.24), (-0.21, 0.79, 0.25), (0.12, -0.27, 0.25,

[0187] (-0.76, 0.56, 0.25)}

[0188] In the coverage area of [-1, 1] × [-1, 1], the sensor radius is in the range of (0, 0.25], and the coordinate positions and coverage radii of 28 sensors are as shown in the above sensor coverage information, so that such coverage minimizes the total cost and maximizes the coverage rate.

Claims

1. A sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization, characterized in that It includes the following steps: Step 1: Construct an objective function with the goals of maximizing sensor coverage rate, minimizing sensor radius, and minimizing the number of sensors; Step 2: Use an improved multi-task algorithm to solve the objective function constructed in Step 1 to obtain the optimal sensor coverage rate, radius, and number.

2. The sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization according to claim 1, characterized in that The objective function constructed in Step 1 is: Among them, x i , y i , r i respectively represent the abscissa, ordinate and coverage radius of the i-th sensor; A is the given coverage area; a is the penalty coefficient for insufficient coverage, representing the cost per unit area not covered; c is the cost coefficient, representing the basic cost of each sensor; b is the variable cost coefficient related to the coverage radius; K is the number of sensors.

3. The sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization according to claim 2, wherein The constraint conditions of the objective function in Step 1 are: -1 < x i <1, i = 1, …, K -1 < y i <1, i = 1, …, K 0.1 < r i <0.25, i = 1, ..., K.

4. The sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Set that each individual in the multi-task algorithm represents a group of sensor deployment schemes, and optimize the coverage tasks with different numbers of sensors through knowledge sharing and migration. Multiple individuals form a population, and population initialization is performed in the decision space. The individual is encoded as a sequence of sensor coordinates and radius; Step 2.2: Set the number of population iterations. Based on the source task selection strategy of reinforcement learning, input the sensor parameter encoding sequence of the individuals in the population into the deep Q network, output the value of each action, and select an action as the source task matched by the current individual through the ε-greedy strategy; Step 2.3: Generate a corresponding adjusted population between the target task and the remaining tasks through the method of domain adaptation. The adjusted population is used to optimize the distribution of sensor coordinates and radius, and reduce the differences in sensor deployment between different tasks; Step 2.4: The individuals in the population start iterative evolution. Obtain the source task through Step 2.

2. If it belongs to the same task as the target task, randomly select an individual in the target task population; if it belongs to a different task from the target task, randomly select an individual from the corresponding adjusted population generated in Step 2.

3. After the individual is selected, use the mutation strategy DE / current-to-best / 1 to mutate the individual to obtain a new offspring individual containing sensor coverage information, and then perform crossover operation on the offspring individual to obtain an offspring population; Step 2.5: Calculate the difference between the fitness of the parent and the offspring as the immediate reward. The fitness is jointly calculated by the sensor coverage rate, the total radius, and the quantity. Update the network parameters of the deep Q network with the reward to optimize the task selection strategy; Step 2.6: Merge the parent and offspring populations, sort them by fitness, select excellent individuals to form a new population. After the iteration number is reached, enter Step 2.7; if the iteration number is not reached, enter Step 2.2; Step 2.7: Decode the individuals in the final excellent population, linearly map the encoded values to the actual coordinate range and radius range, and output the sensor deployment positions and radius.

5. The sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization according to claim 4, characterized in that Step 2.2 specifically includes the following steps: Step 2.2.1: Linearly normalize all dimension values of each individual in the population, that is, the sensor coordinate and radius encoding sequence, and map it to the interval [0,1]; Step 2.2.2: Set the number of input layer nodes of the deep Q network to be equal to the total dimension number of sensor parameters, that is, 3 times the number of sensors, and set the number of output layer nodes of the deep Q network to be equal to the number of optional source tasks; Step 2.2.3: Input the normalized individual encoding into the deep Q network in batches, and calculate the Q values of each individual corresponding to each source task; Step 2.2.4: For each individual, select the column index corresponding to the maximum value from the Q-value matrix output by the deep Q-network through the ε-greedy strategy as the source task matched by the corresponding individual. The source task of each individual corresponds to a deployment optimization strategy for a certain number of sensors.

6. The sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization according to claim 4, characterized in that The specific steps of Step 2.3 are as follows: Step 2.3.1: Use the CORAL method to align the covariance matrices of the source task and the target task to make the sensor coordinate distribution characteristics consistent among different tasks: Among them, * represents the source and the target, n is the number of individuals in the population, and X c is the centered matrix, and Σ is the covariance matrix; Perform singular value decomposition on the covariance matrices of the source and the target respectively to obtain the eigenvector matrices U s and U t as well as the eigenvalue diagonal matrices Λ s and Λ t , then use the source diagonal matrix Λ s to calculate the inverse square root of the covariance matrix, and use the target diagonal matrix Λ t to calculate the square root of the covariance matrix: Transform the matrix of the source task through the inverse square root and square root matrices to obtain a new intermediate matrix Y s : Step 2.3.2: Adjust the distribution shape between tasks through Procrustes analysis to optimize the global layout of sensor coordinates: First, through the formula the source data matrix X S and the target data matrix X T are mean-centered to obtain the centered matrices and Then, through singular value decomposition (SVD) we obtain UDV T , where U and V are orthogonal matrices: Calculate the rotation matrix R: R = VU T Rotate and scale with R to obtain the optimally aligned matrix X new :​ Step 2.3.3: Integrate the results of CORAL and Procrustes, and use R to perform rotation and scaling transformations on Y s and perform weighted averaging on Y s and X new to generate the final adjusted population.

7. The sensor coverage optimization method based on deep reinforcement learning and domain adaptive multi-task optimization according to claim 4, characterized in that The mutation strategy DE / current-to-best / 1 expression used in Step 2.4 is: Among them, randomly select individuals with the top p*Np fitness rankings in a target population, x i,g is the i-th individual in the g-th generation population, F i is the mutation factor of the i-th individual, Np is the population size, and p is a given ratio. is an individual randomly selected from the target population, is an individual randomly selected from the target population and its external archive set.

8. The sensor coverage optimization method based on deep reinforcement learning and domain adaptation multi-task optimization according to claim 4, wherein, In Step 2.5, calculating the fitness difference between the parent and offspring as the immediate reward r is the reward immediately obtained after taking the action a in the state s: r = sign(r′)·log(1 + abs(r′)) where r ′ is the difference in fitness between the parent individual and the offspring individual, sk parent and sk child are the skill factors of the parent and the offspring respectively, Fit(*) represents calculating the fitness value of this individual, and p is the p-th individual in the population with the same skill factor of the offspring.

Images