Remote sensing data discretization optimization method based on depth Q-learning

Abstract

The invention provides an optimization method based on depth Q-learning remote sensing data discretization, and the method comprises the steps: carrying out the feature extraction of a remote sensing image, obtaining a feature subset, carrying out the preprocessing of the feature subset, obtaining a candidate breakpoint set, carrying out the binary genetic coding of the candidate breakpoint set, and constructing a to-be-optimized discretization scheme; designing a state set of a crossing stage and a state set of a variation stage through the global search capability and the local search capability of a genetic operator; constructing a fitness function with a control variable, and constructing a self-adaptive reward function; a pair of depth Q-learning with the same structure is introduced to calculate the Q value in real time in the transition between the cross state and the variation state, and the cross fragment and the variation point of the discretization scheme to be optimized are determined. According to the method, a pair of depth Q-learning with the same structure is introduced into the discretization scheme to be optimized, so that the search efficiency of the feature discretization method based on the evolution model is improved, and the number of breakpoints can be further reduced while relatively high classification precision is obtained.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image feature extraction, and in particular to an optimization method for remote sensing data discretization based on deep Q-learning. Background Art

[0002] Remote sensing data comes from various types of sensors and has the characteristics of multi-dimensionality, incompleteness, randomness, fuzziness, strong interference, etc. At present, many works aim to extract valuable information from the massive and complex remote sensing data collected and process it effectively. Although neural networks have made important progress in the classification and segmentation of remote sensing images, there are still problems such as high computational complexity, low information utilization, noise interference, insufficient generalization ability and lack of interpretability.

[0003] As one of the most influential data preprocessing techniques, feature discretization can reduce the complexity of data by converting continuous features in massive data into discrete features, and obtain more concise, accurate and easy-to-understand rules, thereby improving the efficiency of data mining and machine learning algorithms in remote sensing data processing. However, the distribution of target attribute values ​​is usually difficult to know, and there are complex correlations between features, making it extremely difficult to obtain the optimal discretization scheme for remote sensing data. Although the feature discretization method based on evolutionary models can achieve impressive results, it is difficult to formulate a suitable strategy without prior knowledge as a guide, which will make the search in multidimensional space inefficient and easy to fall into local optimality. Q-learning, as an important reinforcement learning technique, is widely used to solve optimization problems in Markov decision processes (MDPs). In a study focusing on the optimization of discretization schemes for remote sensing data using Q-learning-based GA, each feature subset of the remote sensing image is mapped into a state, and in the crossover and mutation operations, the Q-learning mechanism is introduced to determine the crossover fragments and mutation points of the discretization scheme to be optimized. However, the value range of each feature in remote sensing images is huge, and the Q-tables established for each feature subset cannot carefully characterize the different cross-segments and variation points in each feature, making it difficult to converge the update of Q-values, ultimately leading to a suboptimal solution. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a genetic algorithm based on deep Q-learning to optimize the discretization scheme of remote sensing data.

[0005] To achieve the above-mentioned object of the invention, the present invention provides an optimization method for discretization of remote sensing data based on deep Q-learning, the method comprising:

[0006] S11, extracting features from the remote sensing image to obtain a feature subset, preprocessing the feature subset to obtain a candidate breakpoint set, and performing evolution processing on the candidate breakpoint sets respectively;

[0007] S12, inputting the candidate breakpoint set into GA for iterative evolution, saving the global optimal individual in each crossover and each mutation, and then performing binary genetic coding on the candidate breakpoint set to construct a discretization scheme to be optimized;

[0008] S13, mapping the feature subset of the remote sensing data to a state. In the crossover phase, the feature subset and the two-dimensional matrix consisting of the crossover segments of fixed length on the features contained therein are mapped into a state. In the mutation phase, only the feature subset is mapped into a state, and the mutation point of the current state is obtained in the form of Gaussian sampling according to the absolute dissimilarity between the local optimal individual and the global optimal individual;

[0009] S14, obtaining the fitness value of the individual state by constructing a fitness function with control variables to evaluate the quality of the evolution of the discretization scheme to be optimized, and realizing the conversion between the crossover state and the mutation state by constructing an adaptive reward function;

[0010] S15. Introduce a pair of deep Q-learning with the same structure to calculate the Q value in real time during the transition between the crossover state and the mutation state, and at the same time sample the Q value optimization samples of the discretization scheme to be optimized from the experience replay set to iteratively update the network parameters, and determine the crossover fragments and mutation points of the discretization scheme to be optimized;

[0011] S16, performing genetic operations on the crossover segments and mutation points determined in step S15 and the global optimal individuals in step S12 to optimize the discretization scheme to be optimized and obtain the optimal discretization scheme.

[0012] Furthermore, in step S11, the method of preprocessing the feature subset to obtain a candidate breakpoint set includes: the feature subset includes position information of all pixels and pixel values ​​of all pixels in each band, all pixels contained in each band are sorted and deduplicated according to brightness values ​​within the band to obtain a candidate breakpoint set.

[0013] Furthermore, in step S12, performing binary genetic coding on the candidate breakpoint set specifically includes:

[0014] S21, corresponding each candidate breakpoint in the candidate breakpoint set to each bit in the binary code;

[0015] S22, selecting the candidate breakpoints whose corresponding binary code values ​​are 1, and discarding the candidate breakpoints whose corresponding binary code values ​​are 0;

[0016] S23. Construct a discretization scheme, where the discretization scheme is a set of selected candidate breakpoints.

[0017] Furthermore, in step S13, the processing of the crossover phase specifically includes:

[0018] S31. Assuming that the remote sensing image contains m features, there are 2 m -1 feature subset, each subset of the candidate breakpoint set is a cross-segment, and each cross-segment corresponds to a state, then states, where n k is the number of candidate breakpoints on the kth band;

[0019] S32, set the candidate breakpoints to be crossed on each feature to a fixed length L c The fixed length expression is:

[0020] L c =min 1≤k≤m {n k} / 3

[0021] The number of states generated is:

[0022]

[0023] S33, the feature subset and the features it contains are of length L c The two-dimensional matrix composed of the cross-fragments is mapped into a state, and all states constitute the state set in the cross-stage.

[0024] Furthermore, in step S13, obtaining the variation point of the current state specifically includes:

[0025] S41. The mutation state is sampled in the form of Gaussian sampling, and the mutation point of the current state is obtained according to the absolute dissimilarity between the local optimal individual and the global optimal individual. Then the absolute dissimilarity of chromosome x in state s is:

[0026]

[0027] Among them, n i is the number of candidate breakpoints on the i-th band in state s, b(i,j) is the value of the j-th binary bit on the i-th band;

[0028] S42. Calculate the absolute difference between the historical best individual of each chromosome and the global best individual in the population in the current state:

[0029]

[0030] sum AD(l,g,s)=AD(l,s)+AD(g,s)

[0031] dif AD (l,g,s)=AD(l,s)-AD(g,s)

[0032] Among them, AD T () is the theoretical absolute dissimilarity function, AD() is the absolute dissimilarity function, l is the historical optimal individual of a given chromosome, g is the global optimal individual in the population, dif AD () is the absolute dissimilarity function difference function, sum AD () is the absolute dissimilarity function summation function, AD(l,s) is the absolute dissimilarity of the historical optimal individual of a given chromosome in state s, AD(g,s) is the absolute dissimilarity of the global optimal individual in the population in state s, and rand() is used to generate a uniformly distributed random number between 0 and 1;

[0033] S43. In state s, the absolute dissimilarity of chromosome x after mutation operation obeys Gaussian distribution with mean μ = (AD(l,s) + AD(g,s)) / 2 and standard deviation σ = |AD(l,s) - AD(g,s)|. Then the number of mutation points MN(x,s) of chromosome x is:

[0034]

[0035] dif AD (x,l,s)=AD(x,s)-AD(l,s)

[0036] Among them, MN() is the function of the number of mutation points, round() is the rounding function, and AD(x,s) is the absolute dissimilarity of chromosome x in state s.

[0037] Furthermore, in step S14, a fitness function with control variables is constructed, specifically including:

[0038] S51. The indistinguishable relationship constitutes the division of feature subsets. The fitness function is determined by the number of breakpoints of the discretization scheme and the change of the indistinguishable relationship:

[0039]

[0040] Among them, D is the discretization scheme, C is the conditional attribute set, and N C is the number of breakpoints in the candidate breakpoint set, N D is the number of breakpoints obtained after chromosome decoding, R D is the change of the upper and lower approximation sets obtained after discretizing the data using D, R DThe value of is "0" or "1", which means that the upper and lower approximation sets of D have changed or have not changed after discretization;

[0041] S52, add control variables to expand the fitness function:

[0042]

[0043] Among them, x is the chromosome, is the number of breakpoints obtained after x is decoded, is the change of the upper and lower approximation sets of x with respect to D after discretization, and p is R D The proportion of potential feasible solutions that are "0" in the population,

[0044]

[0045] Among them, z is the control item index, k is the number of categories, and d i For the i-th category, C l (d i ) is the discretization d i The lower approximation set, C u (d i ) is the discretization d i The upper approximation set of C l (d i )' is the discretized d i The lower approximation set, C u (d i )' is the discretized d i The upper approximate set of , setcmp() is the set comparison function.

[0046] Furthermore, in step S14, the changing pattern of the optimal discretized cases is simulated to construct an adaptive reward function in the reinforcement learning model, which specifically includes:

[0047] S61, the total number of states in the crossover phase and the mutation phase is:

[0048]

[0049] S62, current status t By performing an action a t Jump to next state s t+1 , whose expression is:

[0050] (s t ,a t ) → s t+1

[0051] S63. Construct an adaptive reward function by realizing the conversion between the crossover state and the mutation state, that is, compare the fitness of the next state with the fitness of the best individual in history to set the reward function after the current state executes the action:

[0052]

[0053] Among them, r() is the reward function, r(s t ,a t ) is state s t Execute action a t The reward value obtained, t is the number of iterations of genetic operation, s is t is the state under the tth genetic operation, s t+1 Execute action a for the state under the tth genetic operation t The next state to jump to, l is the historical optimal individual, and fit() is the fitness function.

[0054] Furthermore, in step S15, a pair of deep Q-learning with the same structure is introduced to update the Q value in the transition between the crossover state and the mutation state, specifically including:

[0055] S71. In the current state, the ε-greedy strategy is used to select the next action according to the Q value corresponding to each action, and the Q value is updated at each step in the learning process. The expression is:

[0056]

[0057] Among them, a t+1 is the next action, A is the action set, γ is the discount factor, 0≤γ≤1;

[0058] S72, divide the feature subset into 2 m -1 state, m is the number of bands, each state contains several band combinations, that is, the action set A={G,H,I} covering all states is:

[0059]

[0060] Among them, G is the subset acquisition function of the state, H is the subset acquisition function of the state complement, I is the set acquisition function that intersects with the state and is not a subset of the state, G(s t ) is a random jump to s t One of all subsets of t ) is a random jump to s t One of all subsets of the complement, I(s t ) is a random jump to s t The intersection of is not empty and is not s tOne of all sets of subsets, the w appearing in the three formulas are the required sets that satisfy the above three constraints;

[0061] S73. Evaluate the search of the Q network for the optimal solution by setting a reward function for individual fitness changes, wherein the Q network is deep Q-learning:

[0062]

[0063] Among them, fit(s t+1 ) is the individual fitness of the next state, and fit(l) is the historical best fitness of the discretization scheme to be optimized;

[0064] S74, introduce two Q networks with the same structure to obtain the Q value, that is, the current Q network Q c and target Q network Q T , then s t The target Q value and Q c The loss functions are:

[0065]

[0066] Among them, θ T Q T The network parameters, θ C Q c Network parameters, is_end t The value of "1" indicates that s t+1 Is the terminal state, is_end t The value of "0" means that s t+1 is a non-terminal state, h is the number of samples sampled from the experience replay set E;

[0067] S75, by setting a fixed time interval w, Q c θ C Make a copy to update Q T θ T , that is, θ C It is updated by gradient descent, executing θ every w time T =θ C ;

[0068] S76. For two Q networks with the same structure, the Q value is calculated in real time during the transition between the crossover state and the mutation state. At the same time, the Q value optimization samples of the discretization scheme to be optimized are sampled from the experience replay set to iteratively update the network parameters, and the crossover segments and mutation points of the discretization scheme to be optimized are determined.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) The feature subset and the two-dimensional matrix consisting of the fixed-length cross-segments of the features it contains are mapped into a cross-state, and only the feature subset is mapped into a mutation state. The mutation point of the current state is obtained according to the absolute difference between the local optimal individual and the global optimal individual in the form of Gaussian sampling, thereby improving the search efficiency of the evolutionary model.

[0071] (2) An action set that covers the entire band is used to realize the transition between the crossover state and the mutation state, and the changing pattern of the optimal discretization scheme is simulated to construct an adaptive reward function, thereby reflecting the evolutionary trend of the genetic population.

[0072] (3) By introducing a pair of deep Q networks with the same structure to calculate the Q values ​​of all states, the cross-segments and mutation points of the discretization scheme to be optimized are determined, thereby solving the storage difficulties of massive Q values ​​and the problem of dimensionality curse.

[0073] (4) An adaptive reward function is constructed by constructing a fitness function with control variables and simulating the change pattern of the optimal discretization scheme, thereby maintaining the diversity of the population and following the evolutionary laws of individuals.

[0074] (5) By designing two different genetic evolutionary routes and taking the best evolved individuals from each route for genetic operations, high-quality individuals are evolved to optimize the discretization of remote sensing data. This significantly improves the search efficiency of the discretization method based on the evolutionary model, and can further reduce the number of breakpoints while achieving higher classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0076] Figure 1 A schematic flow chart of an optimization method for discretization of remote sensing data based on deep Q-learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0078] Reference Figure 1The present invention provides an optimization method for discretization of remote sensing data based on deep Q-learning, the method comprising:

[0079] S11, extracting features from the remote sensing image to obtain a feature subset, preprocessing the feature subset to obtain a candidate breakpoint set, and performing evolutionary processing on the candidate breakpoint sets respectively.

[0080] As a preferred embodiment, the present invention performs discrete processing on the candidate breakpoint sets respectively. The first line uses conventional genetic evolution on the candidate breakpoint sets, and the second line constructs a discretization scheme to be optimized after processing the candidate breakpoint sets, and performs subsequent genetic evolution processing on it.

[0081] In step S11, the method of preprocessing the feature subset to obtain a candidate breakpoint set includes: the feature subset includes the position information of all pixels and the pixel values ​​of all pixels in each band, all pixels contained in each band are sorted and deduplicated according to the brightness value within the band to obtain a candidate breakpoint set.

[0082] In this embodiment, feature discretization is a process of dividing continuous attribute values ​​into a finite number of small intervals according to a certain specific rule, and associating these small intervals with a set of discrete values. After extracting features from the original remote sensing image, the pixel values ​​in each band are obtained. The normalized pixel value is a set of continuous values ​​ranging from 0 to 1, and each band in the image represents a continuous feature. The goal of remote sensing image feature discretization is to use as few discrete intervals as possible to represent the continuous pixel values ​​in each band while ensuring that the compatibility of the information system is not destroyed. After extracting features from the original remote sensing image, the location information of all pixels and their pixel values ​​in each band are obtained. In the process of preprocessing the extracted remote sensing image information, all pixels contained in each band are sorted and deduplicated according to the brightness value within the band to obtain a candidate breakpoint set.

[0083] S12, input the candidate breakpoint set into GA for iterative evolution, save the global optimal individual in each crossover and each mutation, and then perform binary genetic encoding on the candidate breakpoint set to construct a discretization scheme to be optimized. Searching for the optimal discretization scheme is a constrained optimization problem. GA has inherent implicit parallelism and powerful global search capabilities, and can achieve satisfactory results on the discretization problems of many data sets. In the first evolutionary route, the global optimal individual is saved in each mutation to provide high-quality evolutionary individuals for the evolutionary route of the second discretization scheme to be optimized, so as to obtain the optimal discretization scheme in crossover mutation.

[0084] In step S12, performing binary genetic coding on the candidate breakpoint set specifically includes:

[0085] S21. Correspond each candidate breakpoint in the candidate breakpoint set to each bit in the binary code.

[0086] S22: Select the candidate breakpoints whose corresponding binary code values ​​are 1, and discard the candidate breakpoints whose corresponding binary code values ​​are 0.

[0087] S23. Construct a discretization scheme, where the discretization scheme is a set of selected candidate breakpoints.

[0088] As a preferred embodiment, a binary encoding method is used to encode the candidate breakpoint set, each bit in the binary code corresponds to a candidate breakpoint, and the values ​​"1" and "0" represent that the breakpoint is selected and discarded respectively. The set of selected candidate breakpoints is a discretization scheme, according to which all attribute values ​​are respectively assigned to discrete intervals formed by their breakpoints.

[0089] S13. Map the feature subset of the remote sensing data to a state. In the crossover phase, map the feature subset and a two-dimensional matrix consisting of crossover segments of fixed length on the features it contains into a state. In the mutation phase, only map the feature subset into a state, and obtain the mutation point of the current state in the form of Gaussian sampling based on the absolute dissimilarity between the local optimal individual and the global optimal individual.

[0090] As a preferred embodiment, by designing the state set of the crossover stage and the mutation stage, the problem that the Q-tables established by the feature subset still cannot characterize the different crossover fragments in each feature in detail due to the huge value range of each feature in the remote sensing image, making it difficult to converge the update of Q-values, and the large number of initial breakpoints and the lack of effective heuristic information make it difficult to estimate the number and position of mutation points, so as to improve the search efficiency of the evolutionary model. In the mutation stage, only the feature subset is mapped into a mutation state, and the mutation point of the current state is obtained according to the absolute difference between the local optimal individual and the global optimal individual in the form of Gaussian sampling, so as to solve the problem that each subset of the candidate breakpoint set is regarded as a state, resulting in exponential computational complexity, and it is easy to convert the mutation operator into a crossover operator, thereby losing the local search ability.

[0091] In step S13, the processing of the crossover phase specifically includes:

[0092] S31. Assuming that the remote sensing image contains m features, there are 2 m -1 feature subset, each subset of the candidate breakpoint set is a cross-segment, and each cross-segment corresponds to a state, then states, where n kis the number of candidate breakpoints on the kth band. Each subset of the candidate breakpoint set is regarded as a cross-segment and corresponds to a state.

[0093] S32, set the candidate breakpoints to be crossed on each feature to a fixed length L c The fixed length expression is:

[0094] L c =min 1≤k≤m {n k} / 3

[0095] The number of states generated is:

[0096]

[0097] In order to reduce the computational complexity and speed up the convergence, we set the candidate breakpoints to be crossed on each feature to a fixed length L c , so the number of generated states is reduced to

[0098] S33, the feature subset and the features it contains are of length L c The two-dimensional matrix composed of the cross-fragments is mapped into a state, and all states constitute the state set in the cross-stage.

[0099] In step S13, the variation point of the current state is obtained, which specifically includes:

[0100] S41. The mutation state is sampled in the form of Gaussian sampling, and the mutation point of the current state is obtained according to the absolute dissimilarity between the local optimal individual and the global optimal individual. Then the absolute dissimilarity of chromosome x in state s is:

[0101]

[0102] Among them, n i is the number of candidate breakpoints on the i-th band in state s, and b(i,j) is the value of the j-th binary bit on the i-th band. In this way, we can calculate the absolute difference between the historical optimal individual of each chromosome and the global optimal individual in the population in the current state.

[0103] S42. Calculate the absolute difference between the historical best individual of each chromosome and the global best individual in the population in the current state:

[0104]

[0105] sum AD (l,g,s)=AD(l,s)+AD(g,s)

[0106] difAD (l,g,s)=AD(l,s)-AD(g,s)

[0107] Among them, AD T () is the theoretical absolute dissimilarity function, AD() is the absolute dissimilarity function, l is the historical optimal individual of a given chromosome, g is the global optimal individual in the population, dif AD () is the absolute dissimilarity function difference function, sum AD () is the absolute dissimilarity function summation function, AD(l,s) is the absolute dissimilarity of the historical optimal individual of a given chromosome in state s, AD(g,s) is the absolute dissimilarity of the global optimal individual in the population in state s, and rand() is used to generate random numbers uniformly distributed between 0 and 1.

[0108] S43. In state s, the absolute dissimilarity of chromosome x after mutation operation obeys Gaussian distribution with mean μ = (AD(l,s) + AD(g,s)) / 2 and standard deviation σ = |AD(l,s) - AD(g,s)|. Then the number of mutation points MN(x,s) of chromosome x is:

[0109]

[0110]

[0111] dif AD (x,l,s)=AD(x,s)-AD(l,s)

[0112] Among them, MN() is the number of mutation points function, round() is the rounding function, and AD(x,s) is the absolute dissimilarity of chromosome x in state s.

[0113] As a preferred embodiment, if AD T (x,s) is out of bounds, the number of mutation points is determined based on the historical optimal individual of chromosome x, and then the location of the mutation point is determined. When Temp<0, it means that the number of loci with a value of "1" in chromosome x is less than the number of loci with a value of "1" in chromosome x after the mutation operation in theory. In order to tend to evolve the theoretical value, MN(x,s) genes are randomly selected from the loci with a value of "0" in chromosome x to perform the mutation operation. Similarly, when Temp>0, MN(x,s) genes are randomly selected from the loci with a value of "1" in chromosome x to perform the mutation operation. If MN(x,s)==0, a mutation point is randomly selected on the feature contained in state s. The above method makes full use of the historical information of the chromosome and the shared information of the population. The mutation operator is directional, thereby improving the search efficiency of the population.

[0114] S14. The fitness value of the individual state is obtained by constructing a fitness function with control variables to evaluate the quality of the evolution of the discretized solution to be optimized, and the conversion between the crossover state and the mutation state is realized by constructing an adaptive reward function. The fitness function can more accurately evaluate the fitness of the individual through the control variables, thereby guiding the genetic algorithm to search the solution space more effectively and quickly approach the optimal solution. The adaptive reward function can dynamically adjust the reward value according to the current search state, which helps the algorithm maintain a balance between exploration and utilization during the evolution process, thereby improving the quality and diversity of evolution, and reducing the risk of the algorithm falling into the local optimum, because the change of the reward function can guide the search process to jump out of the current local optimal area and continue to evolve towards the global optimal solution.

[0115] In step S14, a fitness function with control variables is constructed, specifically including:

[0116] Rough sets understand classification as indistinguishable relations in feature space, and indistinguishable relations constitute the division of the space. The principle of selecting the optimal breakpoint set is to generate the least number of breakpoints without destroying the compatibility of the system, thereby constructing the fitness function.

[0117] S51. The indistinguishable relationship constitutes the division of feature subsets. The fitness function is determined by the number of breakpoints of the discretization scheme and the change of the indistinguishable relationship:

[0118]

[0119] Among them, D is the discretization scheme, C is the conditional attribute set, and N C is the number of breakpoints in the candidate breakpoint set, N D is the number of breakpoints obtained after chromosome decoding, R D is the change of the upper and lower approximation sets obtained after discretizing the data using D, R D The value of is "0" or "1", indicating that the upper and lower approximation sets of D have changed or have not changed after discretization.

[0120] Since the features of remote sensing images have complex correlations, in the initial population, R D The number of potential feasible solutions with a value of "0" far exceeds that of R D The feasible solution is "1", which means that there are a large number of individuals with a fitness value of "0" that cannot be selected in the evolution of the population. Therefore, the potential feasible solutions that account for a large proportion of the population will be eliminated, thus destroying the diversity of the population. In order to ensure the diversity of individuals in the early stage of the population and accelerate the speed at which individuals converge to the optimal solution in the later stage of evolution, the fitness function is expanded by adding control variables to dynamically adjust the diversity of the population.

[0121] S52, add control variables to expand the fitness function:

[0122]

[0123] Among them, x is the chromosome, is the number of breakpoints obtained after x is decoded, is the change of the upper and lower approximation sets of x with respect to D after discretization, and p is R D The proportion of potential feasible solutions that are "0" in the population,

[0124]

[0125]

[0126] Among them, z is the control item index, k is the number of categories, and d i For the i-th category, C l (d i ) is the discretization d i The lower approximation set, C u (d i ) is the discretization d i The upper approximation set of C l (d i )' is the discretized d i The lower approximation set, C u (d i )' is the discretized d i The upper approximate set of , setcmp() is the set comparison function.

[0127] In the early stages of evolution, there are no or only a few feasible solutions in the population, that is, p is relatively large; in this way, σ(p) is relatively large, and potential feasible solutions can be selected with a greater probability, guiding the search to gradually enter the feasible solution area. As the evolutionary process proceeds, more and more feasible solutions appear in the population, that is, p becomes smaller and smaller; accordingly, σ(p) also becomes smaller and smaller, which accelerates the convergence of feasible solutions to the optimal solution.

[0128] In step S14, the changing pattern of the optimal discretized cases is simulated to construct an adaptive reward function in the reinforcement learning model, which specifically includes:

[0129] The state set designed in the crossover stage carefully characterizes the different crossover fragments in each feature. In order to realize the conversion between the crossover state and the mutation state, the mutation state is regarded as a special representation of the crossover state with the same feature subset as it, in which all gene loci in the crossover fragment take the value of "0", thereby constructing an adaptive reward function.

[0130] S61, the total number of states in the crossover phase and the mutation phase is:

[0131]

[0132] S62, current status t By performing an action a t Jump to next state s t+1 , whose expression is:

[0133] (s t ,a t ) → s t+1

[0134] S63. Construct an adaptive reward function by realizing the conversion between the crossover state and the mutation state, that is, compare the fitness of the next state with the fitness of the best individual in history to set the reward function after the current state executes the action:

[0135]

[0136] Among them, r() is the reward function, r(s t ,a t ) is state s t Execute action a t The reward value obtained, t is the number of iterations of genetic operation, s is t is the state under the tth genetic operation, s t+1 Execute action a for the state under the tth genetic operation t The next state to jump to, l is the historical optimal individual, and fit() is the fitness function.

[0137] As a preferred embodiment, the adaptive reward function can reflect the changing trend of the reward value as the population evolves, that is, as the population continues to evolve, it will become more and more difficult to update the historical best individual, and accordingly, the reward value should increase. As the objective function value gradually approaches the optimal solution, the increase in the reward value becomes smaller and smaller to speed up the convergence of the global search. 2 (t) is an increasing function, whose slope decreases with the increase of iteration number, which can well describe the above population evolution trend.

[0138] S15. Introduce a pair of deep Q-learning with the same structure to calculate the Q value in real time during the transition between the crossover state and the mutation state. At the same time, sample the Q value optimization samples of the discretization scheme to be optimized from the experience playback set to iteratively update the network parameters and determine the crossover fragments and mutation points of the discretization scheme to be optimized. Q-learning has a wide range of application prospects in solving complex control and decision-making problems. It can be used to determine the crossover fragments and mutation points of the discretization scheme to be optimized. However, when the number of states of Q-learning is large, it is difficult to use Q-Table to store the Q value of each group of "state-action". In order to select the optimal crossover fragment and mutation point and the optimal global individual of the first evolutionary route for genetic iteration operation, introduce a pair of deep Q-learning with the same structure to calculate the Q value in real time during the transition between the crossover state and the mutation state.

[0139] In step S15, a pair of deep Q-learning with the same structure is introduced to update the Q value in the transition between the crossover state and the mutation state, specifically including:

[0140] S71. In the current state, the ε-greedy strategy is used to select the next action according to the Q value corresponding to each action, and the Q value is updated at each step in the learning process. The expression is:

[0141]

[0142] Among them, a t+1 is the next action, A is the action set, γ is the discount factor, 0≤γ≤1.

[0143] S72, divide the feature subset into 2 m -1 state, m is the number of bands, each state contains several band combinations, that is, the action set A={G,H,I} covering all states is:

[0144]

[0145] Among them, G is the subset acquisition function of the state, H is the subset acquisition function of the state complement, I is the set acquisition function that intersects with the state and is not a subset of the state, G(s t ) is a random jump to s t One of all subsets of t ) is a random jump to s t One of all subsets of the complement, I(s t ) is a random jump to s t The intersection of is not empty and is not s tOne of all sets of subsets, the w appearing in the three formulas are the required sets that satisfy the above three constraints. The setting of this action set can cover the action set of the entire band to realize the transition between the crossover state and the mutation state.

[0146] S73. Evaluate the search of the Q network for the optimal solution by setting a reward function for individual fitness changes, wherein the Q network is deep Q-learning:

[0147]

[0148] Among them, fit(s t+1 ) is the individual fitness of the next state, and fit(l) is the historical best fitness of the discretization scheme to be optimized; based on the designed reward value, the values ​​of the two Q-tables in the crossover operation and mutation operation of the discretization scheme to be optimized can be updated, thereby providing reliable decision support for the determination of crossover segments and mutation points.

[0149] S74, introduce two Q networks with the same structure to obtain the Q value, that is, the current Q network Q c and target Q network Q T , then s t The target Q value and Q c The loss functions are:

[0150]

[0151] Among them, θ T Q T The network parameters, θ C Q c Network parameters, is_end t The value of "1" indicates that s t+1 Is the terminal state, is_end t The value of "0" means that s t+1 is a non-terminal state, and h is the number of samples sampled from the experience playback set E. In two Q networks with the same structure, Q c Used to select actions, Q T Used to calculate the target Q value. The Q network contains 3 hidden layers with 20 nodes, m input nodes and 1 output node. Each input node receives a feature breakpoint vector, and the output node generates a Q value vector for the input state under all actions. t Enter Q c Then the Q values ​​corresponding to all actions are obtained. According to the current Q value, the ε-greedy strategy is adopted to select a t ,s t Execute a t Jump to s t+1 , and obtain rt . Quintuple (s t ,a t ,r t ,s t+1 ,is_end t ) is stored in E. h samples are randomly selected from E and passed through Q T Calculate the corresponding target Q value.

[0152] S75, by setting a fixed time interval w, Q c θ C Make a copy to update Q T θ T , that is, θ C It is updated by gradient descent, executing θ every w time T =θ C That is, θ T The network parameters are delayed updates and do not need to be updated iteratively, but are updated from Q at regular intervals. c Copy it, so that the correlation between the target Q value and the current Q value can be weakened.

[0153] S76. For two Q networks with the same structure, the Q value is calculated in real time during the transition between the crossover state and the mutation state. At the same time, the Q value optimization samples of the discretization scheme to be optimized are sampled from the experience replay set to iteratively update the network parameters, and the crossover segments and mutation points of the discretization scheme to be optimized are determined.

[0154] S16, performing genetic operations on the crossover segments and mutation points determined in step S15 and the global optimal individuals in step S12 to optimize the discretization scheme to be optimized and obtain the optimal discretization scheme.

[0155] As a preferred embodiment, the present invention proposes an optimization method algorithm flow based on deep Q-learning remote sensing data discretization, which includes:

[0156] The exploration rate ε is gradually reduced from 1 to 0.05 using a linear decay method, which helps the model to converge to a high-quality solution stably in the later stages of training and prevents the model from completely stopping exploration. In order to reduce the correlation between the target Q value and the current Q value and ensure a considerable convergence speed, the update frequency w is set to 5. The sampling size h of 8 is selected to balance memory requirements and model performance.

[0157] First, the candidate breakpoints of remote sensing data are binary genetically encoded, and the state set and action set of the discretization scheme to be optimized are generated. Then, a greedy algorithm is used to select actions to jump to the next state in the current state of the discretization scheme to be optimized. If the current state is a crossover state, the next state is a mutation state. According to the absolute dissimilarity between the local optimal individual and the global optimal individual, the mutation point of the discretization scheme to be optimized is determined on the feature subset corresponding to the mutation state; if the current state is a mutation state, the next state is a crossover state, and a crossover operation is performed with the global optimal individual on the crossover fragment of the feature subset corresponding to the crossover state.

[0158] By comparing the fitness of the next state with the fitness of the best individual in history, the reward value after the current state executes the action is obtained, and two Q networks with the same structure are introduced to update the Q value of each group of "state-action" to determine the crossover fragments and mutation points of the discretization scheme to be optimized. Perform conventional genetic operations on the candidate breakpoint set input to GA, and save the global best individual in each crossover and each mutation. After the current population has undergone a crossover and mutation, the top 90% of the individuals in fitness value and 10% of the randomly generated new individuals form a new population to enter the next genetic operation. Genetic operations are performed on the determined crossover fragments and mutation points with the global best individual to optimize the discretization scheme to be optimized and obtain the optimal discretization scheme. The program finally outputs the optimal discretization scheme among both global variables and local variables.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An optimization method for discretization of remote sensing data based on deep Q-learning, characterized in that: The method comprises: S11, extracting features from the remote sensing image to obtain a feature subset, preprocessing the feature subset to obtain a candidate breakpoint set, and performing evolutionary processing on the candidate breakpoint sets respectively; S12, inputting the candidate breakpoint set into GA for iterative evolution, saving the global optimal individual in each crossover and each mutation, and then performing binary genetic coding on the candidate breakpoint set to construct a discretization scheme to be optimized; S13, mapping the feature subset of the remote sensing data to a state. In the crossover phase, the feature subset and the two-dimensional matrix consisting of the crossover segments of fixed length on the features contained therein are mapped into a state. In the mutation phase, only the feature subset is mapped into a state, and the mutation point of the current state is obtained in the form of Gaussian sampling according to the absolute dissimilarity between the local optimal individual and the global optimal individual; S14, obtaining the fitness value of the individual state by constructing a fitness function with control variables to evaluate the quality of the evolution of the discretization scheme to be optimized, and realizing the conversion between the crossover state and the mutation state by constructing an adaptive reward function; S15. Introduce a pair of deep Q-learning with the same structure to calculate the Q value in real time during the transition between the crossover state and the mutation state, and at the same time sample the Q value optimization samples of the discretization scheme to be optimized from the experience replay set to iteratively update the network parameters, and determine the crossover fragments and mutation points of the discretization scheme to be optimized; S16, performing genetic operations on the crossover segments and mutation points determined in step S15 and the global optimal individuals in step S12 to optimize the discretization scheme to be optimized and obtain the optimal discretization scheme.

2. The optimization method based on deep Q-learning remote sensing data discretization according to claim 1, characterized in that: In step S11, the method of preprocessing the feature subset to obtain a candidate breakpoint set includes: the feature subset includes the position information of all pixels and the pixel values ​​of all pixels in each band, all pixels contained in each band are sorted and deduplicated according to the brightness value within the band to obtain a candidate breakpoint set.

3. The optimization method based on deep Q-learning remote sensing data discretization according to claim 1, characterized in that: In step S12, performing binary genetic coding on the candidate breakpoint set specifically includes: S21, corresponding each candidate breakpoint in the candidate breakpoint set to each bit in the binary code; S22, selecting the candidate breakpoints whose corresponding binary code values ​​are 1, and discarding the candidate breakpoints whose corresponding binary code values ​​are 0; S23. Construct a discretization scheme, where the discretization scheme is a set of selected candidate breakpoints.

4. The optimization method based on deep Q-learning remote sensing data discretization according to claim 3, characterized in that: In step S13, the processing of the crossover phase specifically includes: S31. Assuming that the remote sensing image contains m features, there are 2 m -1 feature subset, each subset of the candidate breakpoint set is a cross-segment, and each cross-segment corresponds to a state, then states, where n k is the number of candidate breakpoints on the kth band; S32, set the candidate breakpoints to be crossed on each feature to a fixed length L c The fixed length expression is: L c =min 1≤k≤m {n k } / 3 The number of states generated is: S33, the feature subset and the features it contains are of length L c The two-dimensional matrix composed of the cross-fragments is mapped into a state, and all states constitute the state set in the cross-stage.

5. The optimization method based on deep Q-learning remote sensing data discretization according to claim 4, characterized in that: In step S13, the variation point of the current state is obtained, which specifically includes: S41. The mutation state is sampled in the form of Gaussian sampling, and the mutation point of the current state is obtained according to the absolute dissimilarity between the local optimal individual and the global optimal individual. Then the absolute dissimilarity of chromosome x in state s is: Among them, n i is the number of candidate breakpoints on the i-th band in state s, b(i,j) is the value of the j-th binary bit on the i-th band; S42. Calculate the absolute difference between the historical best individual of each chromosome and the global best individual in the population in the current state: sum AD (l,g,s)=AD(l,s)+AD(g,s) dif AD (l,g,s)=AD(l,s)-AD(g,s) Among them, AD T () is the theoretical absolute dissimilarity function, AD() is the absolute dissimilarity function, l is the historical optimal individual of a given chromosome, g is the global optimal individual in the population, dif AD () is the absolute dissimilarity function difference function, sum AD () is the absolute dissimilarity function summation function, AD(l,s) is the absolute dissimilarity of the historical optimal individual of a given chromosome in state s, AD(g,s) is the absolute dissimilarity of the global optimal individual in the population in state s, and rand() is used to generate a uniformly distributed random number between 0 and 1; S43. In state s, the absolute dissimilarity of chromosome x after mutation operation obeys Gaussian distribution with mean μ = (AD(l,s) + AD(g,s)) / 2 and standard deviation σ = |AD(l,s) - AD(g,s)|. Then the number of mutation points MN(x,s) of chromosome x is: Among them, MN() is the function of the number of mutation points, round() is the rounding function, and AD(x,s) is the absolute dissimilarity of chromosome x in state s.

6. The optimization method based on deep Q-learning remote sensing data discretization according to claim 5, characterized in that: In step S14, a fitness function with control variables is constructed, specifically including: S51. The indistinguishable relationship constitutes the division of feature subsets. The fitness function is determined by the number of breakpoints of the discretization scheme and the change of the indistinguishable relationship: Among them, D is the discretization scheme, C is the conditional attribute set, and N C is the number of breakpoints in the candidate breakpoint set, N D is the number of breakpoints obtained after chromosome decoding, R D is the change of the upper and lower approximation sets obtained after discretizing the data using D, R D The value of is "0" or "1", which means that the upper and lower approximation sets of D have changed or have not changed after discretization. S52, add control variables to expand the fitness function: Among them, x is the chromosome, is the number of breakpoints obtained after x is decoded, is the change of the upper and lower approximation sets of x with respect to D after discretization, and p is R D The proportion of potential feasible solutions in the population that are "0", Among them, z is the control item index, k is the number of categories, and d i For the i-th category, C l (d i ) is the discretization d i The lower approximation set, C u (d i ) is the discretization d i The upper approximation set of C l (d i )' is the discretized d i The lower approximation set, C u (d i )' is the discretized d i The upper approximate set of , setcmp() is the set comparison function.

7. The optimization method for discretization of remote sensing data based on deep Q-learning according to claim 6, characterized in that: In step S14, the changing pattern of the optimal discretized cases is simulated to construct an adaptive reward function in the reinforcement learning model, which specifically includes: S61, the total number of states in the crossover phase and the mutation phase is: S62, current status t By performing an action a t Jump to next state s t+1 , whose expression is: (s t ,a t )→s t+1 S63. Construct an adaptive reward function by realizing the conversion between the crossover state and the mutation state, that is, compare the fitness of the next state with the fitness of the best individual in history to set the reward function after the current state executes the action: Among them, r() is the reward function, r(s t ,a t ) is state s t Execute action a t The reward value obtained, t is the number of iterations of genetic operation, s is t is the state under the tth genetic operation, s t+1 Execute action a for the state under the tth genetic operation t The next state to jump to, l is the historical optimal individual, and fit() is the fitness function.

8. The optimization method based on deep Q-learning remote sensing data discretization according to claim 7, characterized in that: In step S15, a pair of deep Q-learning with the same structure is introduced to update the Q value in the transition between the crossover state and the mutation state, specifically including: S71. In the current state, the ε-greedy strategy is used to select the next action according to the Q value corresponding to each action, and the Q value is updated at each step in the learning process. The expression is: Among them, a t+1 is the next action, A is the action set, γ is the discount factor, 0≤γ≤1; S72, divide the feature subset into 2 m -1 state, m is the number of bands, each state contains several band combinations, that is, the action set A={G,H,I} covering all states is: Among them, G is the subset acquisition function of the state, H is the subset acquisition function of the state complement, I is the set acquisition function that intersects with the state and is not a subset of the state, G(s t ) is a random jump to s t One of all subsets of t ) is a random jump to s t One of all subsets of the complement, I(s t ) is a random jump to s t The intersection of is not empty and is not s t One of all sets of subsets, the w appearing in the three formulas are the required sets that satisfy the above three constraints; S73. Evaluate the search of the Q network for the optimal solution by setting a reward function for individual fitness changes, wherein the Q network is deep Q-learning: Among them, fit(s t+1 ) is the individual fitness of the next state, and fit(l) is the historical best fitness of the discretization scheme to be optimized; S74, introduce two Q networks with the same structure to obtain the Q value, that is, the current Q network Q c and target Q network Q T , then s t The target Q value and Q c The loss functions are: Among them, θ T Q T The network parameters, θ C Q c Network parameters, is-end t The value of "1" means s t+1 Is-end t The value of "0" means s t+1 is a non-terminal state, h is the number of samples sampled from the experience replay set E; S75, by setting a fixed time interval w, Q c θ C Make a copy to update Q T θ T , that is, θ C It is updated by gradient descent, executing θ every w time T =θ C ; S76. For two Q networks with the same structure, the Q value is calculated in real time during the transition between the crossover state and the mutation state. At the same time, the Q value optimization samples of the discretization scheme to be optimized are sampled from the experience replay set to iteratively update the network parameters, and the crossover segments and mutation points of the discretization scheme to be optimized are determined.

Images