Magnetotelluric Data Inversion Method Based on MADDPG
By combining the MADDPG algorithm and Bayesian statistics, the problems of initial model dependence and overfitting in magnetotelluric data inversion were solved, and fast and accurate inversion of multiple geoelectric parameters and acquisition of uncertain information were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-18
- Publication Date
- 2026-04-03
AI Technical Summary
Existing magnetotelluric data inversion methods struggle to obtain reliable results when the initial geological model is inaccurate or the objective function is complex. Furthermore, deep learning methods rely heavily on labeled data and suffer from overfitting issues.
The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is used to define magnetotelluric data inversion as a reinforcement learning problem. Inversion is carried out in the exploration space by agents with cooperative relationships, and a geological model is obtained by combining Bayesian statistics.
It achieves rapid and accurate inversion of multiple geoelectric parameters, reduces dependence on the initial geological model, avoids local extreme value trapping, and provides information on uncertainties in the inversion results.
Smart Images
Figure CN115630573B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of geological exploration, specifically involving a magnetotelluric data inversion method based on MADDPG (Multi-Agent Deep Deterministic Policy Gradient). Background Technology
[0002] With the global demand for mineral resources increasing rapidly, shallow mineral resource exploration is gradually failing to meet people's daily needs, making the exploration and development of deep mineral resources a key to human sustainable development.
[0003] Magnetotelluric (MT) methods infer subsurface resistivity by measuring changes in the Earth's natural geomagnetic and geoelectric fields. They are characterized by convenient data acquisition, long-term detection, and deep penetration. MMT has become an important exploration method for studying deep geological structures and is currently widely used in mineral and oil and gas resource exploration, groundwater discovery, geothermal exploration, and crustal and upper mantle tectonic exploration.
[0004] There are numerous methods for inverting magnetotelluric data. Based on differences in the inversion process, they can be categorized into linear or nonlinear iterative inversion methods based on objective functions and global search optimal inversion methods. Iterative inversion methods based on objective functions, such as the Occam method, fast relaxation inversion method, and conjugate gradient inversion method, remain the mainstream magnetotelluric inversion methods. These methods, given an initial geological model, calculate the changes in model parameters and iterate repeatedly until the fitting difference meets the convergence condition. This iterative inversion method based on objective functions has proven effective and reliable over time. However, for complex objective functions with multiple local minima or inappropriate initial geological models, providing reliable geological model inversion results remains challenging.
[0005] Nonlinear global search optimal inversion methods, such as simulated annealing, artificial neural network inversion, and particle swarm optimization, have been successfully applied to one-dimensional and even three-dimensional magnetotelluric (MTI) inversions. These methods have low dependence on the initial geological model and are less prone to getting trapped in local maxima. However, MSI methods typically only provide a single inversion result, and MSI inversion suffers from multiple solutions. To obtain the uncertain information from geophysical inversions, traditional Bayesian methods have been introduced into solving the MSI problem. However, due to their high computational cost, traditional Bayesian methods can only invert relatively simple geological models.
[0006] With the development of artificial intelligence such as deep learning, deep learning has been applied to magnetotelluric inversion. However, the various neural networks currently used in deep learning are essentially supervised learning. Supervised learning requires a large amount of high-quality labeled geodetic observation data as training samples to support the inversion. The difficulty in obtaining actual geodetic observation data and the low quality of the data limit the application of supervised learning in magnetotelluric inversion. At the same time, using supervised learning for magnetotelluric inversion also has the problem of overfitting. Summary of the Invention
[0007] In view of the above, the purpose of this invention is to provide a magnetotelluric data inversion method based on MADDPG, which introduces deep reinforcement learning-based MADDPG into magnetotelluric data inversion to achieve fast and accurate inversion of magnetotelluric data.
[0008] To achieve the above-mentioned objectives, an embodiment provides a magnetotelluric data inversion method based on MADDPG, comprising the following steps:
[0009] Acquire observational electromagnetic data and extract observational geoelectric parameter data for inversion from the observational electromagnetic data;
[0010] The electromagnetic data inversion problem is defined as a reinforcement learning problem, which includes: constructing a single agent for each type of geoelectric parameter; establishing a cooperative relationship among multiple agents corresponding to multiple types of geoelectric parameters in the exploration space; treating the environmental state of the agent's exploration space as the predicted geoelectric parameter data obtained by forward modeling of the geological model; considering the decision action output by each agent as the update amount of each type of geoelectric parameter data; the update amount affects the predicted geoelectric parameter data of the forward modeling of the geological model at the next time step; and constructing a reward value from the fitting difference between the predicted geoelectric parameter data and the observed geoelectric parameter data.
[0011] Based on the defined reinforcement learning problem, the MADDPG algorithm is used to perform reinforcement learning on all agents. In each iteration, the agent parameters are updated, and the geoelectric parameter data of each class are also updated, thereby updating the geological model and realizing the inversion of magnetotelluric data.
[0012] Preferably, when using the MADDPG algorithm to perform reinforcement learning on all agents, each agent explores the entire space starting from the initial geological model, and the agents follow a multi-agent Markov decision process during the search process.
[0013] Preferably, the agent corresponding to each type of geoelectric parameter adopts the DDPG algorithm, including a decision network, an evaluation network, a target decision network, and a target evaluation network;
[0014] The decision network generates an update amount for each type of geoelectric parameter data based on a decision strategy, and this update amount updates each type of geoelectric parameter data, thereby updating the geological model parameters accordingly.
[0015] The evaluation network calculates the true Q value based on the update amount of all geoelectric parameter data output by the decision network of all agents, the predicted geoelectric parameter data of the geological model at the next time step under the influence of the update amount of all geoelectric parameter data, and updates the decision network based on the true Q value.
[0016] The target decision network and target evaluation network periodically copy parameters from the decision network and evaluation network and update their own parameters. The target decision network generates the update amount of each type of geoelectric parameter data based on the sampled predicted geoelectric parameter data. The target evaluation network calculates the target Q value based on the update amount of all types of geoelectric parameter data and the predicted geoelectric parameter data of the geological model forward model under the influence of the update amount of all types of geoelectric parameter data, and constructs a loss function based on the difference between the target Q value and the true Q value to update the evaluation network.
[0017] Preferably, when using the MADDPG algorithm for reinforcement learning on all agents, the agent's decision network receives a reward value when outputting the update amount of each type of geoelectric parameter data. This reward value is constructed based on the fitting difference between the predicted geoelectric parameter data and the observed geoelectric parameter data, and is designed using a potential energy shaping reward method. That is, the reward value Reward is:
[0018] Reward=(ε t -ε t+1 )*k (1)
[0019]
[0020]
[0021] Where, ε t and ε t+1 ε and t+1 are the fitting differences between the data at time t and time t+1, respectively, and k is the reward coefficient. The reward is positive only when the agent outputs a reasonable action that reduces the fitting difference ε. t and ε t+1 Both calculations are performed using formulas (2) and (3). To calculate the fitting difference, N represents the sampling frequency, and i is the index of the sampling frequency, with each sampling frequency corresponding to a set of predicted geoelectric parameters. With observed geoelectric parameter data α represents the noise percentage.
[0022] Preferably, when using the MADDPG algorithm to perform reinforcement learning on all agents, the predicted geoelectric parameters data of the agents before and after executing the decision action, the update amount of each type of geoelectric parameter data, the reward value, and the predicted geoelectric parameters data of the next moment are stored in the experience replay pool. When the training experience reaches a certain amount, the expected reward is calculated by random sampling, and the decision network and evaluation network are updated to optimize the agent's distance-to-vehicle actions so that it searches in the direction of decreasing fitting difference.
[0023] Preferably, when using the MADDPG algorithm to perform reinforcement learning on all agents, a reinforcement learning termination condition is set. When the termination condition is met, the reinforcement learning ends, and the geoelectric parameter data of each type of geoelectricity updated in a single inversion and the geological model obtained in a single inversion are obtained.
[0024] The termination conditions include a preset number of explorations or a fitting difference between the predicted geoelectric parameter data and the observed geoelectric parameter data that is less than a preset threshold.
[0025] Preferably, the method further includes: based on the agent parameters updated in the previous inversion and the initial geological model, using the MADDPG algorithm to perform reinforcement learning on all agents again to perform magnetotelluric data inversion, and repeating this process multiple times to perform magnetotelluric data inversion through reinforcement learning;
[0026] For each type of geoelectric parameter data obtained from multiple reinforcement learning processes, Bayesian probability statistics are performed. Based on the statistical probability distribution, the geoelectric parameter data value with the highest probability value is selected, and the final geological model is obtained based on the geoelectric parameter data value with the highest probability value.
[0027] Preferably, for isotropic strata, the inverted geoelectric parameter data includes resistivity data and layer thickness data. When anisotropy is considered, the geoelectric parameter data also includes resistivity data and rotation angle of each layer in each axial direction.
[0028] Compared with the prior art, the beneficial effects of the present invention include at least the following:
[0029] The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) approach, based on deep reinforcement learning, is introduced into the one-dimensional inversion of magnetotelluric (MT) data. This approach constructs a cooperative multi-agent system for various geoelectric parameters. The predicted geoelectric parameters obtained from forward modeling of the geological model represent the environmental state. The inversion of MT data is achieved through agents exploring the exploration space, similar to a global search for optimal solutions. This inversion method can obtain hundreds or thousands of inversion results in a short time. Furthermore, by incorporating Bayesian statistics to statistically analyze a large number of inversion results, it can quickly obtain the probability distribution of numerous inversion results and the geological model with the highest probability, effectively capturing the uncertainty information of the MT inversion results. This inversion method possesses the advantages of global search-optimal inversion methods, has less dependence on the initial geological model, is less prone to getting trapped in local optima, and can invert more complex geological models. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 This is a flowchart illustrating the magnetotelluric data inversion method based on MADDPG provided in an embodiment of the present invention.
[0032] Figure 2 This is a basic flowchart of the intelligent agent Markov decision process (MAMDP) provided in the embodiments of the present invention;
[0033] Figure 3 This is a schematic diagram of two-dimensional planar navigation for multiple unmanned aerial vehicles provided in an embodiment of the present invention;
[0034] Figure 4 The inversion results and probability distribution statistics of the initial geological model provided in this embodiment of the invention are 10, 100, 200 ohm-m uniform half space.
[0035] Figure 5 This is a data fitting diagram of the forward modeling response and observation data when the initial geological model is a uniform half-space of 10, 100, and 200 ohm-m provided in the embodiments of the present invention.
[0036] Figure 6 This is a schematic diagram of the cumulative reward function for an initial geological model of 10, 100, 200 ohm-m uniform half-space provided in this embodiment of the invention. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0038] To address the problems existing in current geodetic data inversion techniques, this embodiment aims to employ reinforcement learning for geodetic data inversion. Research has led to the proposal of a MADDPG-based geodetic data inversion method. This method introduces the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) approach from deep reinforcement learning into the one-dimensional geodetic data inversion process. Global search inversion improves the efficiency of geodetic data inversion, resulting in a more accurate geological model.
[0039] Figure 1 This is a schematic flowchart of the magnetotelluric data inversion method based on MADDPG provided in an embodiment of the present invention.
[0040] Step 1: Acquire observational electromagnetic data and extract observational geoelectric parameter data for inversion from the observational electromagnetic data.
[0041] In the embodiments, the acquired observation electromagnetic data includes many types of data. The observation geoelectric parameter data used for inversion are extracted from them. When isotropic, the parameters specifically include resistivity data and layer thickness data. When anisotropic, the parameters also include resistivity data and rotation angle of each layer in each axial direction.
[0042] Step 2: Define the electromagnetic data inversion problem as a reinforcement learning problem.
[0043] In the embodiments, when the electromagnetic data inversion problem is defined as a reinforcement learning problem, it includes: constructing a single agent for each type of geoelectric parameter, initializing the network structure and network parameters of the agent, defining the environmental state of the exploration space, the agent's decision-making actions, and the corresponding variables of decision rewards in the electromagnetic data inversion problem.
[0044] In this embodiment, a single agent is constructed for each type of geoelectric parameter. Multiple agents cooperate in the exploration space. The dimension of the agent's output decision action and the Gaussian noise of the action are set according to the assumed number of inversion layers. When the geological model has N layers, each agent can be represented as Agent(R) = R1 × R2 × ... R N , where R represents a type of geoelectric parameter.
[0045] In this embodiment, the agents corresponding to each type of geoelectric parameter employ the DDPG algorithm, including an Actor network, a Critic network, a Target Actor network, and a Target Critic network. The Actor network generates an update amount for each type of predicted geoelectric parameter data based on a decision strategy. This update amount updates the geoelectric parameter data for each type, and consequently updates the geological model parameters. The Critic network calculates the true Q-value based on the update amounts of all types of geoelectric parameter data output by the decision networks of all agents, and the predicted geoelectric parameter data for the next time step under the influence of these update amounts. The Actor network then updates the Actor network based on the true Q-value. The Target Actor network and the Target Critic network periodically copy parameters from the Actor network and the Critic network and update their own parameters. The Target Actor network generates the update amount of each type of geoelectric parameter data based on the sampled predicted geoelectric parameter data for each type. The Target Critic network calculates the target Q value based on the update amounts of all types of geoelectric parameter data generated and the predicted geoelectric parameter data of the geological model forward model under the influence of the update amounts of all types of geoelectric parameter data. It then constructs a loss function based on the difference between the target Q value and the true Q value to update the Critic network.
[0046] When initializing the network structure and parameters, a neural network of appropriate depth and reasonable hyperparameters are designed based on the complexity of the inversion target. In this embodiment, the designed Actor network structure is a three-layer fully connected neural network with a network structure of [128; 64; n]. The first two layers use ReLU as the activation function, and the last layer uses tanh activation, making the range of the output action [-1, 1], and then scaling the action to an appropriate scale. The Critic network structure is a four-layer fully connected neural network, all using ReLU activation. The network structure is designed to be relatively simple because for multi-agent deep reinforcement learning, deeper neural networks are more difficult to backpropagate gradients and require more data for longer training time, which is not conducive to convergence. The structure and initial parameters of the TargetActor network and TargetCritic network are the same as those of the corresponding Actor network and Critic network, respectively.
[0047] In this embodiment, the environmental state of the space explored by the agent is regarded as the predicted geoelectric parameter data obtained by forward modeling of the geological model. The decision action output by each agent is regarded as the update amount of each type of geoelectric parameter data. This update amount will be applied to each type of geoelectric parameter data to update each type of geoelectric parameter data. The updated geoelectric parameter data of each type will affect the predicted geoelectric parameter data of the forward modeling of the geological model at the next time step. The fitting difference between the predicted geoelectric parameter data and the observed geoelectric parameter data is used to construct the reward value.
[0048] Step 3: Based on the defined reinforcement learning problem, the MADDPG algorithm is used to perform reinforcement learning on all agents. In each iteration, the agent parameters are updated, and the geoelectric parameter data of each type are also updated, thereby updating the geological model and realizing the inversion of magnetotelluric data.
[0049] MADDPG is a multi-agent reinforcement learning algorithm based on the Actor-Critic (AC) network architecture. It can be viewed as an extension of Deep Deterministic Policy Gradient (DDPG) in a multi-agent environment. In the Actor-Critic network architecture, the Actor network is responsible for optimizing the agent's exploration policy, while the Critic network acts as a policy critic to evaluate the exploration policy. MADDPG innovatively proposes the idea of "centralized training and decentralized execution." That is, each agent i has a centralized Critic network that receives the actions of all other agents and the environment state as input during training. However, during execution, the decision action output by each agent's policy network is only related to the agent's own state. Experiments have shown that this idea can effectively solve the instability problem of traditional multi-agent training environments.
[0050] In this embodiment, when the MADDPG algorithm is used to perform reinforcement learning on all agents, each agent explores the entire space starting from the initial geological model, and the agents follow a multi-agent Markov decision process during the search process.
[0051] The initial geological model is predefined and contains various geoelectric parameters that need to be inverted. Figure 2 This describes the basic process of a multi-agent Markov decision process (MAMDP). At a certain time t, each agent first outputs an N-dimensional decision action (i.e., the update amount of the geoelectric parameter data) based on its own environmental state (i.e., the predicted geoelectric parameter data). All agents input the set of decision actions into the environment in the form of joint actions. The environment provides feedback on this set of decision actions, which includes the overall environmental state (i.e., the predicted geoelectric parameter data for the forward modeling of the geological model at the next time step) and the overall reward value. Each agent obtains its own environmental state and its own reward value from this, and the agents as a whole enter the next state. This process is continuously iterated and updated in MAMDP.
[0052] In this embodiment, the MADDPG algorithm is applied to a multi-UAV two-dimensional planar navigation task. Figure 3 Taking P as an example, the inversion process is explained. Each UAV starts from the assumed initial position P(P x ,P y Starting from a given point, each action A is generated through a deterministic strategy. x A y Each drone performs action A and then moves to the next position. The agent obtains a reward value Reward = D - D' by observing the distance D between its current position and the navigation endpoint before and after performing the action. D and D' represent the distance between the agent and the endpoint before and after the action, respectively. It can be seen that when the agent moves closer to the endpoint, D decreases and a positive reward value is obtained, while when it moves further away from the endpoint, D increases and a negative reward (penalty) is obtained. In order to obtain a larger positive reward, the agent will gradually learn to search and move closer to the endpoint, thereby achieving the navigation purpose.
[0053] In this embodiment, when using the MADDPG algorithm for reinforcement learning of all agents, each agent starts from a set initial geological model. The agent's Actor network generates an N-dimensional decision action based on the current decision strategy. By applying the set of decision actions to various predicted geoelectric parameter data, the overall geoelectric parameters in the geological model are perturbed to update the geological model. Simultaneously, a reward value is obtained. This reward value is constructed based on the fitting difference between the predicted and observed geoelectric parameter data, and is designed using a potential energy shaping reward method. That is, the reward value is:
[0054] Reward=(ε t -ε t+1 )*k (1)
[0055]
[0056]
[0057] Where, ε t and ε t+1 The fit difference ε represents the difference between the data at time t and time t+1. t and ε t+1 Both calculations are performed using formulas (2) and (3). To calculate the fitting difference, N represents the sampling frequency, and i is the index of the sampling frequency, with each sampling frequency corresponding to a set of predicted geoelectric parameters. With observed geoelectric parameter data α represents the noise percentage, i.e. the standard deviation of Gaussian random numbers, and k is the reward coefficient. The reward is positive only when the agent outputs a reasonable action that reduces the fitting error ε. In order to obtain more positive rewards, the agent will continuously explore regions with smaller fitting errors in the space. After hundreds of explorations, the fitting error gradually decreases to less than the set threshold, thereby achieving the inversion purpose.
[0058] In practice, when using the MADDPG algorithm to perform reinforcement learning on all agents, the predicted geoelectric parameters (current state), the update amount of each type of geoelectric parameter data (decision action), the reward value (reward), and the predicted geoelectric parameters (state) of the next time step are stored in the experience replay pool before and after the agent executes the decision action. When the training experience reaches a certain amount, the expected reward is calculated by random sampling, and the decision network and evaluation network are updated to optimize the agent's distance-to-vehicle actions so that it searches in the direction of decreasing fitting difference.
[0059] Specifically, MADDPG borrows the experience replay mechanism from the DQN algorithm, storing the experience data of each agent in the form of tuples in the experience replay pool, which can be represented as:
[0060] D i =(o1,...,o N ,a1,...,a N ,r1,...,r N ,o1',...,o' N (4)
[0061] Among them, o1,...,o N This represents the predicted geoelectric parameters at the current moment in N dimensions, a1,...,a N Represents the update amount of each type of geoelectric parameter data in N dimensions, r1,...,r N Let o1',...,o' represent the reward value corresponding to the N-dimensional update. N This represents the predicted geoelectric parameters for the next time step in N dimensions.
[0062] In the early stages of training, the experience replay pool is only responsible for collecting experience data. The network is only updated after a certain amount of experience has been collected. Each time, a minimum batch of samples is randomly sampled from the experience pool and input into the network for policy training. The random sampling method reduces the correlation between different experiences, thereby improving the network training efficiency.
[0063] During training, the Actor network for each agent can be updated using the following stochastic gradient descent method:
[0064]
[0065] Among them, oi and a i μ represents the state and decision action of the i-th agent, respectively. i (a i |o i ) represents the action strategy of the intelligent agent. This indicates selecting the optimal true Q-value and performing gradient descent on the action direction, E o,a～D (·) represents the average expectation of a state-decision action pair, while Indicates according to action strategy μ i Choose the decision action a of agent i. i θ represents the parameters of the Actor network. This represents the actual Q-value obtained by the agent when performing the current action considering the global state.
[0066] The Critic network of each agent is updated iteratively by minimizing the loss function, i.e.
[0067]
[0068]
[0069] Where y represents the cumulative target Q-value of agent i in the Target Actor network, and γ represents the discount factor. E represents the expected Q value calculated when performing an action in the next state. o，a，r，o′ Expressing expectations, The parameter μ' represents the Critic network. i and Represent the Target Actor network and the Target Critic network, respectively, using y and the true Q-value. The expected mean squared error is used to construct the loss function. By minimizing the loss function Update the Critic network.
[0070] In this embodiment, the Target Actor network and Target Critic network are periodically copied to their corresponding Target Actor network and Target Critic network using a soft update method, as shown below:
[0071] θ i '=τθ i +(1-τ)θ i (8)
[0072]
[0073] Where, θi 'and τ represents the number of network adoptions for the Target Actor network and the Target Critic network, respectively. τ is a control parameter for adjusting the update frequency, which is usually chosen as a small number such as 0.01 to ensure that each parameter is updated to the current network in a small increment, thereby easing the span of parameter updates and ensuring the stability of training.
[0074] In this embodiment, when the MADDPG algorithm is used to perform reinforcement learning on all agents, a reinforcement learning termination condition is set. When the termination condition is reached, the reinforcement learning ends and the geoelectric parameter data of each type of single inversion update and the geological model obtained by single inversion are obtained. The termination condition includes a preset number of explorations or the fitting difference between the predicted geoelectric parameter data and the observed geoelectric parameter data is less than a preset threshold. The fitting difference is calculated using the above formulas (2) and (3).
[0075] Step 4: Based on the agent parameters updated in the previous inversion and the initial geological model, Step 3 is repeated multiple times to perform multiple reinforcement learning magnetotelluric data inversions, and the final geological model is determined based on Bayesian probability statistics.
[0076] In this embodiment, based on the agent parameters updated in the previous inversion and the initial geological model, the MADDPG algorithm is used to perform reinforcement learning on all agents again to invert magnetotelluric data. This process is repeated hundreds or thousands of times to perform multiple reinforcement learning magnetotelluric data inversions. Then, Bayesian probability statistics are applied to each type of geoelectric parameter data obtained from multiple reinforcement learnings. Based on the statistical probability distribution, the geoelectric parameter data value with the highest probability value is selected. The final geological model is obtained based on the geoelectric parameter data value with the highest probability. This approach, to a certain extent, obtains uncertain information from the inversion results and is essentially more in line with the laws of geophysical inversion.
[0077] The MADDPG-based magnetotelluric data inversion method provided in this embodiment starts with an initial geological model. During the early training phase, each agent accumulates experience using a method similar to a global search, gradually narrowing the search scope through trial and error. In the later training phase, most of the lower-value solutions are discarded, significantly reducing the time required for a single inversion. The MADDPG inversion algorithm possesses the advantages of a global search-optimal inversion method, with less dependence on the initial geological model and less susceptibility to local optima. Furthermore, by introducing Bayesian probability statistics to quickly obtain the probability distribution and maximum probability model of hundreds of inversion results, some degree of uncertainty information in the inversion results can be obtained.
[0078] The following section uses a five-layer geological model to perform magnetotelluric data inversion:
[0079] To verify the effectiveness of the MADDPG-based magnetotelluric data inversion method proposed in the embodiments, the following geological model was designed, with resistivity and layer thickness used as geoelectric parameters. The resistivity and layer thickness parameters of the initial geological model are shown in Table 1. The magnetotelluric frequency range was designed to be 0.001–1000 Hz, and 15 frequency points were taken at equal intervals after conversion to the logarithmic domain. Forward modeling was performed on the theoretical model in Table 1, and 2% Gaussian noise was added as observation data. The dependence of MADDPG inversion on the initial geological model was tested when the initial geological model was set to a five-layer homogeneous half-space of 10 ohm-m, 100 ohm-m, and 200 ohm-m.
[0080] Table 1
[0081] resistivity / ohm-m 400 100 10 40 200 Thickness / m 2400 1800 2300 1500 ∞
[0082] Figure 4 This invention provides inversion results and probability distribution statistics for an initial geological model of 10, 100, and 200 ohm-m uniform half-space. Figure 4 It can be seen that, except for the central high-resistivity layer, which is less sensitive to magnetotelluric methods, the MADDPG inversion results show good fit for all other layers under different initial geological models, indicating that the initial geological model has little impact on the inversion results. For the first 100, 200, and 300 inversion iterations, there are small differences in the statistical results and the maximum probability model, but the overall trend is similar. In the later stages of the inversion, the agent's exploration weakens, and the inversion becomes more stable. Figure 5 The results compare the fitting between the forward response of the inversion results and the observation data under the initial geological model of 10, 100, and 200 ohm-m uniform half space. It can be seen that the apparent resistivity and phase of the inversion results under different initial geological models are well fitted, and the errors between each frequency point are all within 5%.
[0083] Figure 6 This is a cumulative reward curve of the agent inversion process under different initial geological models provided in embodiments of the present invention. Figure 6 It can be seen that during the training warm-up phase, the agent conducts a comprehensive and random exploration of the environment, storing the exploration experience in the experience replay pool, resulting in random fluctuations in the reward curve. After training begins, the agent's reward increases rapidly. In the early stages of training, to ensure sufficient exploration, the agent's actions are subject to relatively high noise, thus causing oscillations in the reward curve. After hundreds or thousands of training rounds, the agent inversion with different initial geological models all achieves convergence (i.e., reaches the maximum reward). The relative reward values of the reward curves for different initial geological models differ, which is related to the magnitude of the difference between the initial geological model and the real model, but has a relatively small impact on the inversion convergence effect.
[0084] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A magnetotelluric data inversion method based on MADDPG, characterized in that, Includes the following steps: Acquire observational electromagnetic data and extract observational geoelectric parameter data for inversion from the observational electromagnetic data; The electromagnetic data inversion problem is defined as a reinforcement learning problem, which includes: constructing a single agent for each type of geoelectric parameter; establishing a cooperative relationship among multiple agents corresponding to multiple types of geoelectric parameters in the exploration space; treating the environmental state of the agent's exploration space as the predicted geoelectric parameter data obtained by forward modeling of the geological model; considering the decision action output by each agent as the update amount of each type of geoelectric parameter data; the update amount affects the predicted geoelectric parameter data of the forward modeling of the geological model at the next time step; and constructing a reward value from the fitting difference between the predicted geoelectric parameter data and the observed geoelectric parameter data. Based on the defined reinforcement learning problem, the MADDPG algorithm is used to perform reinforcement learning on all agents. In each iteration, the agent parameters are updated, and the geoelectric parameter data of each class are also updated, thereby updating the geological model and realizing the inversion of magnetotelluric data.
2. The magnetotelluric data inversion method based on MADDPG according to claim 1, characterized in that, When using the MADDPG algorithm to perform reinforcement learning on all agents, each agent explores the entire space starting from the initial geological model, and the agents follow a multi-agent Markov decision process during the search process.
3. The magnetotelluric data inversion method based on MADDPG according to claim 2, characterized in that, The agent corresponding to each type of geoelectric parameter adopts the DDPG algorithm, including decision network, evaluation network, target decision network, and target evaluation network; The decision network generates an update amount for each type of geoelectric parameter data based on a decision strategy, and this update amount updates each type of geoelectric parameter data, thereby updating the geological model parameters accordingly. The evaluation network calculates the true Q value based on the update amount of all geoelectric parameter data output by the decision network of all agents, the predicted geoelectric parameter data of the geological model at the next time step under the influence of the update amount of all geoelectric parameter data, and updates the decision network based on the true Q value. The target decision network and target evaluation network periodically copy parameters from the decision network and evaluation network and update their own parameters. The target decision network generates the update amount of each type of geoelectric parameter data based on the sampled predicted geoelectric parameter data. The target evaluation network calculates the target Q value based on the update amount of all types of geoelectric parameter data and the predicted geoelectric parameter data of the geological model forward model under the influence of the update amount of all types of geoelectric parameter data, and constructs a loss function based on the difference between the target Q value and the true Q value to update the evaluation network.
4. The magnetotelluric data inversion method based on MADDPG according to claim 1, characterized in that, When using the MADDPG algorithm for reinforcement learning on all agents, the agent's decision network receives a reward value when outputting the update amount of each type of geoelectric parameter data. This reward value is constructed based on the fitting difference between the predicted and observed geoelectric parameter data, and is designed using a potential energy shaping reward method. The reward value is: Reward=(e t -e t+1 )*k (1) Where, ε t and ε t+1 ε and t+1 are the fitting differences between the data at time t and time t+1, respectively, and k is the reward coefficient. The reward is positive only when the agent outputs a reasonable action that reduces the fitting difference ε. t and ε t+1 Both calculations are performed using formulas (2) and (3). To calculate the fitting difference, N represents the sampling frequency, and i is the index of the sampling frequency, with each sampling frequency corresponding to a set of predicted geoelectric parameters. With observed geoelectric parameter data α represents the noise percentage.
5. The magnetotelluric data inversion method based on MADDPG according to claim 1, characterized in that, When using the MADDPG algorithm to perform reinforcement learning on all agents, the predicted geoelectric parameters data of the agents before and after executing decision actions, the update amount of geoelectric parameters data for each class, the reward value, and the predicted geoelectric parameters data for the next time step are stored in the experience replay pool. The expected reward is also calculated by random sampling, and the decision network and evaluation network are updated to optimize the agents' distance-to-vehicle actions so that they search in the direction of decreasing fitting difference.
6. The magnetotelluric data inversion method based on MADDPG according to claim 1, characterized in that, When using the MADDPG algorithm to perform reinforcement learning on all agents, a termination condition for reinforcement learning is set. When the termination condition is met, the reinforcement learning ends, and the geoelectric parameter data of each type of geoelectricity updated in a single inversion and the geological model obtained in a single inversion are obtained. The termination conditions include a preset number of explorations or a fitting difference between the predicted geoelectric parameter data and the observed geoelectric parameter data that is less than a preset threshold.
7. The magnetotelluric data inversion method based on MADDPG according to claim 1, characterized in that, The method further includes: based on the agent parameters updated in the previous inversion and the initial geological model, using the MADDPG algorithm to perform reinforcement learning on all agents again to perform magnetotelluric data inversion, and repeating this process to perform magnetotelluric data inversion multiple times through reinforcement learning. For each type of geoelectric parameter data obtained from multiple reinforcement learning processes, Bayesian probability statistics are performed. Based on the statistical probability distribution, the geoelectric parameter data value with the highest probability value is selected, and the final geological model is obtained based on the geoelectric parameter data value with the highest probability value.
8. The magnetotelluric data inversion method based on MADDPG according to claim 1, characterized in that, For isotropic strata, the geoelectric parameter data retrieved includes resistivity data and layer thickness data. When anisotropy is considered, the geoelectric parameter data also includes resistivity data and rotation angle of each layer in each axis direction.