Ground transient electromagnetic parameter extraction method based on Q-learning algorithm

By defining the state space and using a perturbation-greedy strategy in the Q-learning algorithm, the problems of high computational cost and local optimization in transient electromagnetic parameter extraction are solved, and fast and high-precision parameter extraction is achieved.

CN119026371APending Publication Date: 2024-11-26JILIN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411222662.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing methods for extracting transient electromagnetic parameters are computationally intensive and inefficient. Traditional methods rely on initial models and are prone to getting trapped in local minima, making it difficult to achieve fast and high-precision parameter extraction.

Method used

The Q-learning algorithm is adopted to randomly generate a one-dimensional layered resistivity model, define the state space, action space and reward function, extract parameters by combining a perturbation greedy strategy, and use the agent's self-learning ability to find the optimal parameters.

Benefits of technology

It achieves rapid and high-precision extraction of ground transient electromagnetic parameters, avoids the local optimization trap of traditional methods, reduces dependence on the initial model, and improves computational efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026371A_ABST
    Figure CN119026371A_ABST
Patent Text Reader

Abstract

The invention relates to a ground transient electromagnetic parameter extraction method based on a Q-learning algorithm, and the method comprises the steps: 1), randomly generating a one-dimensional layered resistivity model, and randomly generating a resistivity value according to a resistivity range actually existing in the nature; 2) performing one-dimensional numerical simulation by using the generated one-dimensional layered resistivity model, and taking the obtained ground transient electromagnetic response data as an input quantity of a Q-learning parameter extraction model; (3) establishing a Q-learning parameter extraction model; and performing parameter extraction on the acquired transient electromagnetic response data d0 according to the parameter extraction model, and when iteration of the parameter extraction algorithm reaches a convergence condition, obtaining a layered resistivity parameter and completing parameter extraction. The problem that a traditional method depends too much on an initial model and is prone to falling into local optimum is avoided, and meanwhile the problem that a deep learning method needs to construct a large number of training sets in the early stage is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for extracting ground transient electromagnetic parameters in the field of geophysical exploration, and in particular to a method for extracting ground transient electromagnetic parameters based on a Q-learning algorithm. Background Art

[0002] The Transient Electromagnetic Method (TEM) is a geophysical detection method widely used today. A strong pulse current is passed from the transmitter to the transmitting coil, generating a strong pulse primary field in the surrounding space. Under the excitation of the primary field, the underground conductor generates a secondary magnetic field. The secondary magnetic field is observed through the receiving coil, and the parameters of the secondary magnetic field data are extracted to obtain relevant information of the underground geological body.

[0003] Since transient electromagnetic parameter extraction has a non-unique solution, the use of traditional methods to extract parameters from transient electromagnetic data will result in large amounts of calculation and low efficiency. Currently, commonly used parameter extraction methods generally include imaging methods, linear parameter extraction methods, and nonlinear parameter extraction methods. Imaging methods generally refer to conductivity depth imaging methods, which process data quickly but have low accuracy and are suitable for preliminary interpretation in the field. Linear parameter extraction methods mostly use repeated iterations to achieve approximate calculations of the model. This method is prone to falling into local minima, has a huge amount of calculations, and is highly dependent on the initial model.

[0004] With the development of geophysical parameter extraction methods, nonlinear methods have attracted widespread attention due to their global optimization characteristics. With the development of computer technology, nonlinear inversion methods are all implemented through machine learning. Reinforcement learning, as a branch of machine learning, has developed rapidly in recent years. The Q-learning algorithm is a model-free algorithm constructed using a Markov decision process in a finite discrete environment, and plays an important role in reinforcement learning. Its optimization idea is as follows: Based on the current strategy, the agent selects the action corresponding to the maximum Q value in the Q table and independently explores this unfamiliar environment without understanding all the characteristics of the environment. Summary of the invention

[0005] The purpose of the present invention is to provide a ground transient electromagnetic parameter extraction method based on Q-learning algorithm to address the deficiencies of existing parameter extraction methods, so as to achieve rapid and accurate extraction of parameters of geological models.

[0006] The present invention is implemented as follows: 1. A method for extracting ground transient electromagnetic parameters based on a Q-learning algorithm, characterized in that it comprises the following steps:

[0007] 1) Randomly generate a one-dimensional layered resistivity model, and the resistivity value is randomly generated according to the resistivity range actually existing in nature;

[0008] 2) Using the generated one-dimensional layered resistivity model to perform one-dimensional numerical simulation, the obtained ground transient electromagnetic response data is used as the input of the Q-learning parameter extraction model;

[0009] 3) Define the state space, reward function, action space, Q table update and action selection strategy, and establish a Q-learning parameter extraction model;

[0010] Further, in step 3):

[0011] State space definition, setting the parameter extraction target model to a layered earth model, setting the model resistivity variation range to (ρ min , ρ max ); Discretize the resistivity within the range of variation, and set the discrete step length to Δρ, that is, the state space is defined as S = [ρ min , ρ min +Δρ,ρ min +2Δρ,…,ρ max ];

[0012] Action space definition, the action space includes all possible resistivities, that is, the action space is defined as A = [ρ min , ρ min +Δρ,ρ min +2Δρ,…,ρ max ];

[0013] Reward function definition, the reward function is obtained by comparing the transient electromagnetic response data error before the action is executed with the transient electromagnetic response data error after the action is executed;

[0014] The transient electromagnetic response data error is defined as Where d0 represents the input transient electromagnetic response data, and d represents the transient electromagnetic response data obtained by the currently extracted resistivity parameters;

[0015] The reward function is defined as Among them, e0 represents the transient electromagnetic response data error before the action is executed, and e represents the transient electromagnetic response data error after the action is executed; when the transient electromagnetic response data error after the action is executed becomes larger, the reward value is negative, and the larger the error, the smaller the reward value; conversely, when the transient electromagnetic response data error after the action is executed becomes smaller, the reward value is positive, and the larger the error, the smaller the reward value;

[0016] The Q table is updated, and the feedback value Q value after each action is executed is recorded in the Q table, and the formula Q(s,a)=Q(s,a)+α[r+λmax a′ Q(s′,a′)-Q(s,a)] to update the Q value, where Q(s,a) represents the Q value under the current state s and the current action a, Q(s′,a′) represents the Q value under the estimated state s′ and the estimated action a′, α represents the learning rate, λ represents the reward attenuation coefficient, and α and λ are determined according to the characteristics of the input transient electromagnetic response data;

[0017] 4) According to the parameter extraction model of step 3), the transient electromagnetic response data d0 collected in step 2) is subjected to parameter extraction based on the Q-learning algorithm. When the iteration of the parameter extraction algorithm reaches the convergence condition, the layered resistivity parameters are obtained, and the parameter extraction is completed.

[0018] 2. A method for extracting ground transient electromagnetic parameters based on a Q-learning algorithm according to claim 1, characterized in that:

[0019] In step 3), the action selection strategy is the perturbation greedy strategy;

[0020] The perturbation greedy strategy specifically affects the selection of the current action by adding a perturbation term, making the current action selection more random. The perturbation decreases with the increase of iterations. The perturbation rule is:

[0021] in It is represented as a perturbation term, rand represents a random number in the range of (0, 1), k represents the maximum number of iterations of the required perturbation, and count represents the number of iterations.

[0022] The present invention achieves the following beneficial effects:

[0023] Compared with the prior art, the present invention avoids the problem that traditional methods are overly dependent on the initial model and easily fall into local optimality. It finds the optimal parameters through the self-learning ability of the intelligent body, avoids the problem that deep learning methods require the early construction of a large number of training sets, and is beneficial to the rapid and high-precision extraction of ground transient electromagnetic parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the ground transient electromagnetic parameter extraction method based on the Q-learning algorithm;

[0025] Figure 2 is the parameter extraction iteration number graph;

[0026] Figure 3 is a resistivity parameter trend diagram extracted by the present invention; DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. One of the core concepts of the present invention is that the invention provides a ground transient electromagnetic parameter extraction method based on the Q-learning algorithm, which can accurately extract parameters from transient electromagnetic detection data, thereby obtaining more accurate information about the geological target body. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] Example

[0029] Combination Figure 1 As shown, a method for extracting ground transient electromagnetic parameters based on a Q-learning algorithm includes:

[0030] 1) Randomly generate a one-dimensional layered resistivity model, and the resistivity value is randomly generated according to the resistivity range actually existing in nature;

[0031] A three-layer resistivity model is randomly generated in the range of 10Ω·m to 1000Ω·m. The resistivity parameters of the generated model are [100, 1000, 500] and the layer thickness is [400, 200, ∞];

[0032] 2) Using the generated one-dimensional layered resistivity model to perform one-dimensional numerical simulation, the obtained ground transient electromagnetic response data d0 is used as the model input for Q-learning parameter extraction;

[0033] 3) Establish a model for Q-learning parameter extraction;

[0034] Further, step 3) includes the following steps:

[0035] First, the state space definition:

[0036] The parameter extraction model is set as a layered earth model, and the model resistivity variation range is set to 10Ω·m to 1000Ω·m. In order to describe the accurate state, the resistivity is discretized within the variation range, and the discrete step length is set to 10Ω·m, that is, the state space is defined as S=[10, 20, 30, ..., 1000];

[0037] Second, the action space is defined:

[0038] In order to make the action selection span larger and the action space include all possible resistivities, the action space is defined as A = [10, 20, 30, ..., 1000];

[0039] Secondly, the reward function is defined as:

[0040] The reward function is obtained by comparing the transient electromagnetic response data error before the action is executed with the transient electromagnetic response data error after the action is executed;

[0041] The transient electromagnetic response data error is defined as Where d0 represents the input transient electromagnetic response data, and d represents the transient electromagnetic response data obtained by the currently extracted resistivity;

[0042] The reward function is defined as Wherein, e0 represents the transient electromagnetic response data error before the action is executed, and e represents the transient electromagnetic response data error after the action is executed. When the transient electromagnetic response data error after the action is executed becomes larger, the reward value is negative, and the larger the error, the smaller the reward value. Conversely, when the transient electromagnetic response data error after the action is executed becomes smaller, the reward value is positive, and the larger the error, the smaller the reward value;

[0043] Secondly, the Q table is updated:

[0044] The feedback value Q after each action is executed is recorded in the Q table, and the formula Q(s,a)=Q(s,a)+α[r+λmax a′ Q(s′,a′)-Q(s,a)] to update the Q value, where Q(s,a) represents the Q value under the current state s and the current action a, Q(s′,a′) represents the Q value under the estimated state s′ and the estimated action a′, α represents the learning rate, λ represents the reward decay coefficient, and α is set to 0.5 and λ to 0.8, where the Q table is expressed as:

[0045]

[0046] The rows of the matrix represent states s, and the columns represent actions a. S = [10, 20, ..., 990, 1000], A = [10, 20, ..., 990, 1000], Q(s i ,a i )i=1,2,3,…,100 means in state s i Next, perform action a i Q value;

[0047] Finally, the action selection strategy:

[0048] Combination Figure 2 As shown, in order to prevent the agent from relying too much on the Q value and weakening the exploration of the unknown space, the action selection strategy of the present invention is a perturbation greedy strategy, and 5 cycles are performed to explore more space;

[0049] The perturbation greedy strategy specifically affects the selection of the current action by adding a perturbation term, making the current action selection more random. The perturbation decreases with the increase of iterations. The perturbation rule is:

[0050] in It is represented as a disturbance term, rand represents a random number in the range of (0, 1), k represents the maximum number of iterations required for the disturbance, count represents the number of iterations, and k=200 is set.

[0051] 4) According to the parameter extraction model of step 3), the transient electromagnetic response data d0 collected in step 2) is subjected to parameter extraction based on the Q-learning algorithm. When the iteration of the parameter extraction reaches the convergence condition, the layered resistivity parameters are obtained, and the parameter extraction is completed.

[0052] The ground transient electromagnetic parameter extraction results are as follows Figure 3 As shown, the target parameters are: the resistivity at 0-400 meters underground is 100Ω·m, the resistivity at 400-600 meters underground is 1000Ω·m, and the resistivity below 600 meters underground is 500Ω·m; the extracted parameters are: the resistivity at 0-400 meters underground is 100Ω·m, the resistivity at 400-600 meters underground is 890Ω·m, and the resistivity below 600 meters underground is 510Ω·m.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting ground transient electromagnetic parameters based on Q-learning algorithm, characterized in that: The steps include: 1) Randomly generate a one-dimensional layered resistivity model, and the resistivity value is randomly generated according to the resistivity range actually existing in nature; 2) Using the generated one-dimensional layered resistivity model to perform one-dimensional numerical simulation, the obtained ground transient electromagnetic response data is used as the input of the Q-learning parameter extraction model; 3) Define the state space, reward function, action space, Q table update and action selection strategy, and establish a Q-learning parameter extraction model; Further, in step 3): State space definition, setting the parameter extraction target model to a layered earth model, setting the model resistivity variation range to (ρ min , ρ max ); Discretize the resistivity within the range of variation, and set the discrete step length to Δρ, that is, the state space is defined as S = [ρ min , ρ min +Δρ,ρ min +2Δρ,…,ρ max ]; Action space definition, the action space includes all possible resistivities, that is, the action space is defined as A = [ρ min , ρ min +Δρ,ρ min +2Δρ,…,ρ max ]; Reward function definition, the reward function is obtained by comparing the transient electromagnetic response data error before the action is executed with the transient electromagnetic response data error after the action is executed; The transient electromagnetic response data error is defined as Where d0 represents the input transient electromagnetic response data, and d represents the transient electromagnetic response data obtained by the currently extracted resistivity parameters; The reward function is defined as Among them, e0 represents the transient electromagnetic response data error before the action is executed, and e represents the transient electromagnetic response data error after the action is executed; when the transient electromagnetic response data error after the action is executed becomes larger, the reward value is negative, and the larger the error, the smaller the reward value; conversely, when the transient electromagnetic response data error after the action is executed becomes smaller, the reward value is positive, and the larger the error, the smaller the reward value; The Q table is updated, and the feedback value Q value after each action is executed is recorded in the Q table, and the formula Q(s,a)=Q(s,a)+α[r+λmax a′ Q(s′,a′)-Q(s,a)] to update the Q value, where Q(s,a) represents the Q value under the current state s and the current action a, Q(s′,a′) represents the Q value under the estimated state s′ and the estimated action a′, α represents the learning rate, λ represents the reward attenuation coefficient, and α and λ are determined according to the characteristics of the input transient electromagnetic response data; 4) According to the parameter extraction model of step 3), the transient electromagnetic response data d0 collected in step 2) is subjected to parameter extraction based on the Q-learning algorithm. When the iteration of the parameter extraction algorithm reaches the convergence condition, the layered resistivity parameters are obtained, and the parameter extraction is completed.

2. A method for extracting ground transient electromagnetic parameters based on a Q-learning algorithm according to claim 1, characterized in that: In step 3), the action selection strategy is the perturbation greedy strategy; The perturbation greedy strategy specifically affects the selection of the current action by adding a perturbation term, making the current action selection more random. The perturbation decreases with the increase of iterations. The perturbation rule is: in It is represented as a perturbation term, rand represents a random number in the range of (0, 1), k represents the maximum number of iterations of the required perturbation, and count represents the number of iterations.