A Reinforcement Learning-Based Method and System for Predicting Formation Porosity in Oil Logging Tools

By reconstructing formation porosity prediction into a Markov decision process and combining well logging curves and physically constrained reward functions, the overfitting and non-physical oscillation problems in traditional methods are solved, achieving high-precision and stable porosity prediction, which is applicable to formation porosity prediction in oil well logging tools.

CN122862893BActive Publication Date: 2026-11-17ZHONGSONGNENG (XIAN) TECHNOLOGY CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202611350821.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-09-02
Publication Date
2026-11-17
Estimated Expiration
2046-09-02

AI Technical Summary

Technical Problem

Traditional formation porosity prediction methods cannot effectively embed physical equations, which leads to overfitting or non-physical oscillations in sparse core calibration data blocks, making it difficult to meet the accuracy and stability requirements of fine reservoir evaluation.

Method used

Formation porosity prediction is reconstructed as a Markov decision process. A three-component state space is constructed, including a sliding window of logging curves, upper porosity sequence, and working state. A physical constraint reward function is introduced, which includes core error, Archie formula residual, and formation continuity penalty. An offline actor-commentator algorithm is used for pre-training. The hidden layer parameters are fixed and only the output layer is fine-tuned to achieve rolling inference at depth points.

Benefits of technology

It generates physically consistent full-well porosity curves, which have the advantages of high accuracy, high stability and low data dependence, significantly improving the prediction stability and engineering practicality under sparse calibration conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122862893B_ABST
    Figure CN122862893B_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for predicting formation porosity using a petroleum logging tool based on reinforcement learning. The method comprises the following steps: acquiring an offline historical dataset of old blocks with sufficient core samples; constructing a state space and reward function for a Markov decision process based on the offline historical dataset; constructing a Markov decision process model based on the state space and reward function; pre-training the Markov decision process model using the offline historical dataset to obtain a basic porosity prediction model; collecting full logging data and a small amount of porosity calibration data for the current prediction block, and fine-tuning the basic porosity prediction model to obtain a formation porosity prediction model; acquiring real-time logging data and real-time operating status data for the current prediction block, inputting them into the formation porosity prediction model, outputting the predicted porosity value at the current depth point, and finally generating the formation porosity prediction result for the logging tool.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, specifically to a method and system for predicting formation porosity in oil well logging tools based on reinforcement learning. Background Technology

[0002] Formation porosity is a core parameter in oil well logging reservoir evaluation, and its accuracy directly affects oil and gas reserve calculations and development plans. With the development of logging technology, multiple logging curves, such as natural gamma ray, neutron, density, sonic transit time, and resistivity, can reflect formation physical characteristics from different perspectives, providing abundant indirect data for porosity prediction. However, due to the complexity and heterogeneity of subsurface geological conditions, accurately and stably extracting porosity information from multi-source logging data has always been a challenge in the field of well logging interpretation.

[0003] Traditional formation porosity prediction methods often employ supervised regression based on deep learning. This method uses well logging curve sequences as input features and core-calibrated porosity as labels. A deep neural network is trained to learn the nonlinear mapping relationship from well logging response to porosity. During training, the goal is to minimize the mean square error between the predicted and calibrated values. Error backpropagation is performed independently for each depth point, resulting in an end-to-end regression model.

[0004] However, this method treats porosity prediction at each depth point as an independent regression task, and cannot embed physical equations into the training objective to constrain the physical consistency of the prediction results. As a result, in blocks with sparse core calibration data, the model is prone to overfitting or non-physical oscillations due to the lack of sufficient supervision signals. Ultimately, the fluctuation of the prediction curve in the vertical direction of the formation does not conform to the actual geological laws, and it is difficult to meet the requirements of fine reservoir evaluation for porosity prediction accuracy and stability. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a reinforcement learning-based method and system for predicting formation porosity in oil well logging tools, thereby resolving the problems existing in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method and system for predicting formation porosity using a reinforcement learning-based oil well logging tool, comprising the following steps: Step S1: Obtain the offline historical dataset of old blocks with sufficient core samples. The offline historical dataset includes historical logging data, historical working status data, and historical porosity calibration data. Construct the state space of a Markov decision process based on the historical logging data and historical working status data, and construct a reward function based on the historical porosity calibration data. Build a Markov decision process model based on the state space and reward function. Step S2: Based on the offline historical dataset, the Markov decision process model is pre-trained using an offline actor-critic algorithm with a conservative regularization term, wherein the actor network is a policy network and the critic network is a double Q-value network, to obtain the basic model for porosity prediction. Step S3: Collect the full logging data and a small amount of porosity calibration data of the current prediction block, fix the hidden layer parameters of the porosity prediction basic model, and fine-tune the output layer of the strategy network and the double Q value network in the porosity prediction basic model based on the full logging data and a small amount of porosity calibration data to obtain the formation porosity prediction model. Step S4: Obtain real-time logging data and real-time working status data of the current prediction block, and input them into the formation porosity prediction model point by point according to depth, and output the porosity prediction value of the current depth point; update the state space based on the porosity prediction value of the current depth point and recursively push to the next depth point until all depth points are predicted, and finally generate the formation porosity prediction result of the logging tool. This invention reconstructs formation porosity prediction as a sequential decision problem within a Markov decision process. It constructs a three-component state space comprising a sliding window of logging curves, an upper porosity sequence, and the working state, along with a physical constraint reward function that integrates core error, Archie formula residuals, and formation continuity penalties. This allows the model to learn both data mapping and physical laws simultaneously during training. An offline actor-commentator algorithm is used for pre-training, and a conservative regularization term is employed to suppress action estimate inflation, solving the problem of non-interactive behavior on static datasets. Furthermore, a migration strategy that fixes the hidden layer and only fine-tunes the output layer allows adaptation to new blocks with only a small amount of calibration data, significantly reducing core sampling costs. Ultimately, it achieves depth-by-depth rolling inference, generating physically consistent full-well porosity curves, combining high accuracy, high stability, and low data dependency.

[0007] Preferably, step S1 includes: Obtain offline historical datasets of old blocks with sufficient core samples, including historical logging data, historical working status data, and historical porosity calibration data; The historical logging data includes numerical sequences of natural gamma curves, neutron curves, density curves, sonic transit time curves, and resistivity curves, as well as depth indices for each curve. The historical working status data includes temperature data, pressure data, and well inclination angle data of the logging tool at various depth points; The historical porosity calibration data includes the core calibration porosity values ​​at each core depth point and their corresponding depth index. The historical logging data is subjected to curve filtering and standardization. A sliding window construction logic similar to image sequence samples is adopted. A logging curve window of a preset length is slidably extracted along the depth direction with a preset step size as the first component of the state space. The logging curve window records the feature values ​​of various logging curves along the depth in matrix form. The porosity value sequence of multiple consecutive depths above the current depth is extracted as the second component of the state space. The historical working state data corresponding to the current depth is extracted as the third component of the state space. In the offline pre-training stage, the second component is obtained by interpolation or regional averaging of historical porosity calibration data. In the online inference stage, it is obtained by rolling updates of the output historical prediction values. Combining the first component, the second component, and the third component yields the state space of the Markov decision process. A reward function is constructed based on the historical porosity calibration data. A Markov decision process model is constructed based on the state space and reward function. The specific composition of logging curves, working status, and calibration data in the historical dataset was clarified, providing a complete data foundation for the construction of the state space and reward function, ensuring that the model can make full use of multi-source information and improve prediction reliability.

[0008] Preferably, the step of constructing the reward function based on the historical porosity calibration data includes: A core error term is constructed. When historical porosity calibration data exists, the core error term is the absolute error between the predicted porosity and the core calibration porosity; when no historical porosity calibration data exists, the core error term is zero. An Archie formula residual term is constructed based on the logarithmic domain absolute residual between the measured resistivity calculated by substituting the predicted porosity into the Archie formula and the formula-fitted resistivity. A formation continuity penalty term is constructed based on the absolute value of the difference between the predicted porosity at the current depth and the predicted porosity at the previous depth. The negative core error term, the negative Archie formula residual term, and the negative formation continuity penalty term are weighted and summed to obtain the reward function of the Markov decision process. The water saturation in the Archie formula residual term is provided by the existing independent logging interpretation results of the current prediction block, or is predetermined by logging methods independent of this prediction, or uses the typical value of the block as the input constant. By using three physical constraint reward functions, the model can simultaneously perceive the longitudinal trend, physical equation constraints, and geological continuity, effectively suppressing non-physical oscillations and maintaining the physical consistency of predictions even under sparse core conditions.

[0009] Preferably, the construction of the Markov decision process model based on the state space and reward function includes: Based on the state space and reward function, the formation porosity prediction problem is reconstructed into a Markov decision process; The action space of the Markov decision process is the porosity prediction value at the current depth point, and the porosity prediction value is a continuous real number with a value range of 0 to 0.4. The state transition of the Markov decision process is a deterministic transition: after the current depth point is predicted, the state is rolled down by one sampling interval in the depth direction, the logging curve sliding window is updated accordingly, and the state is automatically transferred to the next depth point. The discount factor in the Markov decision process is a preset value used to balance the importance of current rewards and future rewards; Based on the state space, action space, state transition probability, reward function, and discount factor, a Markov decision process model is constructed. Defining the action space as a continuous porosity value avoids the quantization error caused by discretization; the deterministic state transition conforms to the actual process of logging point by point; the discount factor balances the impact of current and future decisions, making the Markov decision process modeling more in line with engineering needs.

[0010] Preferably, step S2 includes: Based on the offline historical dataset, an experience replay buffer is constructed, which is used to store training samples. Construct the policy network and the double Q-value network, and initialize the network parameters of the policy network and the double Q-value network; Randomly sample a small batch of samples from the experience replay buffer, calculate the loss functions of the policy network and the double Q-value network based on the small batch of samples, and iteratively update the network parameters by gradient descent until the policy network converges to obtain the basic model for porosity prediction. The offline actor-critic algorithm can be used to learn policies on static historical datasets without interacting with the real environment. This avoids the security risks of online learning and makes full use of the rich data resources of old blocks.

[0011] Preferably, the iterative update of network parameters using gradient descent includes: Based on the small batch of samples, the Q-values ​​of the current state-action pairs are estimated using a dual Q-value network, and a smaller Q-value is selected as the target value to reduce overestimation bias. A conservative regularization term is introduced when calculating the loss function to suppress the inflation of the Q-value estimate for unobserved actions; The parameters of the main Q-value network are updated by minimizing the loss function, and the parameters of the main Q-value network are soft-copied to the target Q-value network after each training step. Meanwhile, based on the updated master Q-value network, the network parameters of the strategy are updated by maximizing the expected cumulative reward; Repeat the above steps until the policy network converges; By minimizing the overestimation bias through a dual Q-value network, introducing a conservative regularization term to suppress Q-value inflation of unobserved actions, and combining this with a soft update mechanism to stabilize the training process, the pre-trained model maintains robust policy evaluation capabilities even under data distribution shifts.

[0012] Preferably, step S3 includes: Obtain full logging data and a small amount of porosity calibration data for the current predicted block; The porosity prediction model is transferred to the current prediction block. The hidden layer parameters of the strategy network and the double Q-value network are fixed. Based only on the full logging data and a small amount of porosity calibration data of the current prediction block, the output layer of the strategy network and the output layer of the double Q-value network are fine-tuned and trained to obtain the formation porosity prediction model. The fixed hidden layer only fine-tunes the output layer, retaining the ability to extract general logging features learned from old blocks. Only 5 to 10 core calibration points are needed to complete the adaptation to new blocks, greatly reducing calibration costs and preventing overfitting and negative migration.

[0013] Preferably, step S4 includes: Obtain real-time logging data and real-time working status data for the current predicted block; The real-time logging data is extracted according to a preset sliding window and combined with the predicted upper porosity sequence and the real-time working status data to construct the state space of the current depth point. The state space is input into the formation porosity prediction model, and the porosity prediction value at the current depth point is output. Based on the porosity prediction value at the current depth point, generate the formation porosity prediction result of the logging tool; It realizes a rolling inference process of constructing state, inputting model and outputting prediction for each depth point. The state is automatically updated and the inference speed is fast, which meets the timeliness requirements of real-time well logging interpretation and is convenient for field engineering deployment.

[0014] Preferably, the step of generating the formation porosity prediction result based on the porosity prediction value at the current depth point includes: The predicted porosity value at the current depth point is stored in the predicted upper porosity sequence, and the sliding window is updated by rolling it down one sampling interval along the depth direction to construct the state space of the next depth point. Repeat the above steps until all depth points are predicted. Then, stitch together the porosity prediction values ​​of all depth points according to the depth index to generate the formation porosity prediction results of the logging tool. By stitching together the predicted values ​​of all depth points according to the depth index to form a complete porosity curve, it can be directly used for geological interpretation and reserve assessment, improving the practicality and readability of the prediction results.

[0015] A reinforcement learning-based formation porosity prediction system for oil well logging tools includes: Memory is used to store computer programs and running data; A processor that, when executing the computer program, implements the steps of the above method; The processor drives the following functional modules to work together: The historical data acquisition and modeling module is used to acquire offline historical datasets of old blocks with sufficient core samples, and to construct a Markov decision process model based on the offline historical datasets. The offline pre-training module is used to pre-train the Markov decision process model based on the offline historical dataset and an offline actor-critic algorithm with a conservative regularization term to obtain the porosity prediction base model. The migration fine-tuning module is used to acquire the full logging data and a small amount of porosity calibration data of the current prediction block, fix the hidden layer parameters of the porosity prediction basic model, and fine-tune the training only on the output layers of the strategy network and the double Q value network to obtain the formation porosity prediction model. The real-time inference module is used to acquire real-time logging data and real-time working status data of the current prediction block, input the formation porosity prediction model, output the porosity prediction value at the current depth point, and generate the formation porosity prediction result of the logging tool. The historical data acquisition and modeling module, offline pre-training module, transfer fine-tuning module, and real-time inference module interact and collaborate through a shared data storage unit, which is configured in the memory. In this embodiment, the shared data storage unit is a three-level hierarchical industrial storage array with an integrated bus arbitration controller, which is bidirectionally connected to the four major functional modules via a PCIe high-speed bus. The storage unit is divided into three layers: memory cache, SSD persistent storage, and mechanical hard disk archive. It is internally divided into independent logical partitions and the module read and write priorities are set. It uniformly stores the original logging curves, MDP training samples, network model weights and porosity prediction time series curves. It relies on the bus arbitration controller to coordinate the concurrent read and write of multiple modules to achieve copy-free data sharing and interaction among the modules. By integrating four functional modules—historical data modeling, offline pre-training, transfer fine-tuning, and real-time inference—through modular design and sharing data storage, the entire process from training to deployment is automated, facilitating engineering implementation and system maintenance.

[0016] This invention provides a method for predicting formation porosity using oil well logging tools based on reinforcement learning, which relates to reinforcement learning technology and has the following beneficial effects: (1) By reconstructing the formation porosity prediction problem into a Markov decision process and introducing a sequential decision framework, the agent, in each prediction step, not only relies on the current logging response but also fully considers the historical trajectory of the predicted porosity in the upper layer and the cumulative reward at future depths. This effectively utilizes the gradual change law of formation porosity and avoids the non-physical oscillations caused by ignoring the correlation between depths in traditional methods. At the same time, the introduction of the discount factor allows the model to balance short-term prediction accuracy with long-term formation trends, making the overall prediction curve more consistent with the actual geological deposition law and significantly improving the prediction stability under sparse calibration conditions.

[0017] (2) By drawing on the sliding window concept in image processing, the well logging curves are truncated along the depth direction at preset step sizes to form a two-dimensional numerical matrix containing multiple depth points and multiple curves as the first component of the state space. On the one hand, this preserves the multi-parameter correlation information of the well logging curves in the horizontal direction, and on the other hand, it captures the sedimentary sequence characteristics of the strata through the vertical sliding window, so that the state space has both the spatial structure of the image and the temporal dependence of the sequence. In addition, the introduction of the second and third components of the state space further enriches the agent's perception dimension of the current geological environment, providing a more complete and physically meaningful input representation for subsequent reinforcement learning decisions, thereby improving the model's adaptability to complex strata conditions.

[0018] (3) By innovatively introducing the Archie formula residual term into the reward function, the rock physics equation is directly embedded into the reinforcement learning training objective. This allows the model to obtain a dense physical consistency supervision signal even at depth points lacking core calibration data. This is achieved by using the logarithmic domain residual between the measured resistivity and the predicted porosity after substituting them into the Archie formula. This effectively suppresses overfitting and non-physical solutions caused by data sparsity. At the same time, by combining the core error term and the formation continuity penalty term, the reward function achieves multi-level synergy between sparse calibration signals, dense physical constraints, and geological priors. This enables the model to produce physically reasonable and vertically smooth porosity curves even in new blocks with very few calibration points, significantly improving the generalization ability and engineering applicability of the method. Attached Figure Description

[0019] Figure 1 This is a flowchart of the formation porosity prediction method for oil well logging tools based on reinforcement learning proposed in this invention.

[0020] Figure 2 The hierarchical diagram of the basic model for porosity prediction is obtained for the formation porosity prediction method based on reinforcement learning proposed in this invention.

[0021] Figure 3 The hierarchical map of formation porosity prediction results obtained by the reinforcement learning-based oil well logging tool formation porosity prediction method proposed in this invention is shown.

[0022] Figure 4 This is a module architecture diagram of the formation porosity prediction system for oil well logging tools based on reinforcement learning proposed in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see Figures 1-3 This invention provides a technical solution: a reinforcement learning-based method for predicting formation porosity using oil well logging tools. Specifically, the following reinforcement learning-based method for predicting formation porosity using oil well logging tools is provided; please refer to [link to relevant documentation]. Figure 1 The method includes the following steps: Step S1: Obtain the offline historical dataset of old blocks with sufficient core samples. The offline historical dataset includes historical logging data, historical working status data, and historical porosity calibration data. Construct the state space of a Markov decision process based on the historical logging data and historical working status data, and construct a reward function based on the historical porosity calibration data. Construct a Markov decision process model based on the state space and reward function.

[0025] Please see Figure 2 This step first obtains an offline historical dataset of old blocks with sufficient core samples, and then constructs the state space and reward function of a Markov decision process based on this dataset, thereby building a complete Markov decision process model. This step is the foundation of the entire method, as it reconstructs the formation porosity prediction problem into a sequential decision problem, laying the framework for subsequent offline reinforcement learning pre-training and transfer fine-tuning.

[0026] First, offline historical datasets of well-developed blocks with sufficient core samples are obtained. In practical engineering, well-developed blocks typically refer to mature development blocks with multiple core wells and a wealth of core analysis data. The offline historical dataset includes historical logging data, historical operational status data, and historical porosity calibration data.

[0027] The historical logging data includes numerical sequences of natural gamma curves, neutron curves, density curves, sonic transit time curves, and resistivity curves, along with depth indices for each curve. These curves are physical response signals continuously acquired by the logging tool along the depth in the well, reflecting the formation's radioactivity, hydrogen index, bulk density, sonic propagation velocity, and electrical conductivity. Each curve is recorded at sampling intervals of 0.1 meters or 0.125 meters. The historical operating status data includes temperature, pressure, and well inclination angle data at various depth points. These data record the environmental conditions during downhole operation and play an important role in correcting logging responses and evaluating data quality. The historical porosity calibration data includes core calibration porosity at each core sampling depth and its corresponding depth index. The core analysis porosity was measured in the laboratory using the helium porosimeter method or the liquid saturation method and serves as the baseline true value for porosity prediction. It should be noted that core sampling is costly in actual engineering, and each well typically has only a few dozen to a few hundred core sampling points. Furthermore, there is a depth matching error between the core sampling depth and the logging depth. Therefore, core repositioning and depth alignment are required during the data preprocessing stage to ensure that the calibration data corresponds one-to-one with the depth of the logging curve.

[0028] After obtaining the offline historical dataset, constructing a Markov decision process model first requires building the state space of the Markov decision process. The design of this state space draws on the sliding window construction idea of ​​image sequence samples. The core is to slide and group isolated point samples according to the depth direction to form a sequence sample that simultaneously contains lateral parameter correlation and vertical deposition relationship.

[0029] Specifically, the historical logging data and historical operational status data were first normalized. The selection of logging data curves was based on the Pearson correlation coefficient between each logging parameter and porosity; curves with high correlation to porosity were retained, while those with low correlation were removed. Both logging data and operational status data underwent maximum-minimum normalization, mapping their values ​​uniformly to the [0,1] interval to eliminate the influence of different dimensions and orders of magnitude on model training.

[0030] Then, a sliding window construction logic based on image-like sequence samples is adopted. A well logging curve window of a preset length is extracted along the depth direction with a preset step size of one sampling point. The numerical matrix of multiple depth points and multiple curves within this window is used as the first component of the state space. The sliding step size is one sampling point, meaning it moves downwards by one depth interval each time. Preliminary tests showed that a window length of four sampling points achieves a good balance between information richness and focus. Furthermore, a porosity value sequence of four consecutive depths above the current depth is extracted as the second component of the state space. The data source for this sequence varies depending on the operational stage. In the offline pre-training stage, it is filled by interpolation of historical porosity calibration data, or the average porosity of the region is taken as the initial value at uncalibrated points. In the real-time inference stage, it is updated continuously by the historical prediction values ​​output by the model. This second component allows the agent to perceive the longitudinal trend of porosity changes, providing historical memory for decision-making. Finally, historical working state data corresponding to the current depth, namely temperature, pressure, and well inclination angle, are extracted as the third component of the state space. It should be noted that temperature, pressure, and well inclination angle have different dimensions and significantly different numerical ranges. Therefore, before inputting them into the network, the maximum and minimum value normalization method is also used to uniformly map them to the [0,1] interval, which is consistent with the normalization method of well logging curves.

[0031] Combining the first, second, and third components yields the state space of the Markov decision process. .

[0032] After constructing the state space, it is necessary to construct the reward function of the Markov decision process. The design of the reward function directly determines the learning objective and optimization direction of the agent. The reward function proposed in this invention... It is obtained by weighted summation of three terms, and its expression is: ; in, In the state Take action below The immediate reward obtained is the predicted porosity. The instant reward received; This refers to the core error term; This is the residual term of Archie's formula; This is a penalty term for stratigraphic continuity. , , The weight coefficients for each item are dimensionless and were obtained through a grid search method using five-fold cross-validation on a training set containing 5000 samples. The final determined values ​​are... , , In areas with dense core calibration, the level can be appropriately increased. For sparsely labeled areas, more reliance should be placed on... and It should be noted that the core error term... This only applies to depth points where calibration data exists; at other depth points, this term is zero. In this case, the reward function mainly consists of the Archie formula residual term and the formation continuity penalty term.

[0033] First item For the core error term, the construction logic is as follows: when porosity calibration data exists at the current depth, calculate the absolute error between the predicted porosity and the core calibration porosity, and take a negative value as a penalty. The specific formula is as follows: ; in, This is a dimensionless predicted value for porosity at the current depth. The porosity of the core sample at this depth is calibrated. This dimensionless calibration is only valid if calibration data exists. Although porosity calibration data is sparse, it is highly accurate. This ensures that the agent approximates the true value at depths with supervised signals. If no porosity calibration data exists at the current depth, then... =0.

[0034] Second item The residual term of the Archie formula is constructed by substituting the predicted porosity into the Archie formula, calculating the absolute residual in the logarithmic domain between the measured resistivity from the well logging and the resistivity fitted by the formula, and taking a negative value as a penalty. The calculation formula is as follows: ; in, The resistivity is measured in well logging, and the unit is ohm-meter, which is directly measured by resistivity logging instruments. The fitted resistivity, calculated using Archie's formula, is expressed in ohmmeters. The formula is as follows: ; in, The Archie proportionality coefficient, dimensionless, was obtained by fitting a double logarithmic curve of the Archie formula using the least squares method based on rock electrical experimental data from 30 rock samples within the study block. The specific value is... ; The formation water resistivity, measured in ohmmeters, was obtained by averaging the analysis data from water samples from five wells within the study area. The specific value is... ; The cementation index, dimensionless, reflects the tortuosity of the rock's pore structure. It was obtained by fitting a curve showing the relationship between porosity and formation factors using the least squares method based on the rock electrical experimental data from the aforementioned 30 rock samples. The specific values ​​are as follows: ; The water saturation level is dimensionless. The water saturation index, dimensionless, reflects the distribution characteristics of oil and water in pores. It was obtained by fitting the relationship curve between water saturation and resistivity index using the least squares method based on the rock electrical experimental data of the aforementioned 30 rock samples. The specific values ​​are as follows: Predicting porosity Substituting into the above formula, the fitted resistivity is calculated. Then calculate the measured resistivity. With fitted resistivity The absolute residual of the logarithmic field is used as the residual term in Archie's formula.

[0035] The traditional Archie formula uses measured porosity to calculate water saturation, while this invention uses water saturation... The porosity prediction is based on known constraints. Firstly, water saturation can be obtained as a stable prior value from block water samples and regional geological data, making it applicable across the entire region. Secondly, the objective of this scheme is porosity, and continuously measured porosity across the entire region is scarce; therefore, a fixed water saturation value is necessary. It can generate effective physical constraint loss throughout the process, making up for the deficiencies in a small amount of core calibration data; thirdly, it can fix the water saturation. Eliminate the problem of multiple solutions due to the coupling of two parameters in the formula, and uniquely constrain the porosity output to ensure that the prediction results fit the rock electrical characteristics of the block.

[0036] It should be noted that water saturation Provided by existing well logging interpretation results within the study block, or predetermined using methods independent of the current predicted porosity such as nuclear magnetic resonance logging, it serves as a known constant input, independent of the predicted porosity at the current depth point. Change, for those without independence The well interval for which the data is taken can be approximated using typical values ​​of that layer within the block, but the applicable range must be specified. In this embodiment, for a single oil-bearing interval, a value of [missing value] is taken. As an example; for the water layer section, a different value should be assigned, such as... In practical applications, whether a current depth point belongs to an oil layer or a water layer is automatically determined using the following rules: If an independent water saturation interpretation result already exists for that depth point, then the result provided is used directly. Value; if there is no independent interpretation, then the judgment is based on the resistivity curve threshold: when the measured resistivity When the resistivity of the oil layer in the block is higher than the lower limit, it is considered an oil layer. Otherwise, it is judged as a water layer, and [the following is taken]: This threshold can be adjusted based on the actual rock electrical statistical patterns of the block. The lower limit of the oil layer resistivity of the block is determined by statistical analysis of core samples and well logging data from multiple wells in the block: the measured resistivity values ​​corresponding to the oil and water layers confirmed by well testing are extracted from each well, and the minimum resistivity value of the oil layer sample is taken as the lower limit of the oil layer resistivity of the block. This can be flexibly adjusted according to the differences in reservoir lithology and formation water salinity.

[0037] Third item This is a formation continuity penalty term. The construction logic involves calculating the absolute value of the difference between the predicted porosity at the current depth and the predicted porosity at the previous depth, and taking a negative value as the penalty. The calculation formula is as follows: ; in, and These are the predicted porosity values ​​for the current depth and the previous depth, respectively, and are dimensionless. This is based on prior geological knowledge that porosity typically changes gradually along the vertical direction of the formation, without drastic jumps. Therefore, penalizing excessive differences in porosity between adjacent depths can suppress non-physical oscillations in the prediction results. For the starting depth point of each well section, since there is no predicted porosity from the previous depth... Stratigraphic continuity penalty The value is zero.

[0038] In completing the state space and reward function After constructing the Markov decision process, the action space needs to be defined. The action space is defined as the predicted porosity value at the current depth point. This value is a continuous real number, ranging from 0 to 0.4, corresponding to the physical range of actual formation porosity. The reason for choosing a continuous action space rather than a discrete action space is that porosity is a continuously changing physical quantity, and discretization would introduce quantization errors and reduce prediction accuracy.

[0039] State transition probability The definition is closely related to the method of acquiring well logging data. After the current depth point is predicted, the state transition consists of two parts: the part unrelated to the action is determined by the well logging curve sliding window and the working state data, which are rolled deterministically along the depth direction for one sampling interval; the part related to the action is the current action, i.e., the predicted porosity. This is added to the upper porosity sequence, while the oldest depth point in the sequence is removed, forming the second component for the next time step. The length of this upper porosity sequence is four sampling points, consistent with the sliding window length of the well logging curve. Therefore, the state transition probability... Partially dependent on agent actions, the overall transition is denoted as a deterministic transition, i.e., given s and a, the next state. It is uniquely determined and does not require random modeling.

[0040] Discount factor A preset value between 0.9 and 0.99 is used to balance the importance of current and future rewards. The closer the discount factor is to 1, the more the agent focuses on long-term cumulative rewards, which is particularly important for porosity prediction tasks, as the prediction error at the current depth point affects the state space of subsequent depth points, thus impacting the accuracy of subsequent predictions. Optimization on the validation set using a grid search method resulted in a final discount factor value of 0.95.

[0041] Based on the above state space Action space State transition probability Reward function and discount factor A complete Markov decision process model is constructed, denoted as . .

[0042] Step S2: Based on the offline historical dataset, the Markov decision process model is pre-trained using an offline actor-critic algorithm with a conservative regularization term, where the actor network is the policy network and the critic network is a double Q-value network, to obtain the basic model for porosity prediction.

[0043] This step is based on the Markov decision process model constructed in step S1. A basic porosity prediction model was obtained by pre-training using offline historical datasets from old blocks with abundant core samples. The core of this step is an offline actor-critic algorithm with a conservative regularization term, which learns the optimal strategy on a static dataset that cannot interact with the environment, while simultaneously addressing the problems of data distribution offset and action estimation inflation in well logging scenarios.

[0044] First, construct the experience replay buffer D. Store all experience samples from the offline historical dataset as quadruples, each quadruple containing the current state. Current action Instant rewards and the next state The current state s is extracted from the logging data by the state space construction rules defined in step S1; the current action a is to take the core calibration porosity at the current depth point. Alternatively, the porosity estimate obtained from traditional well logging interpretation methods can be used as a proxy action; the immediate reward r is the reward function defined in step S1. Calculated; Next state This is the state constructed according to the same rules after scrolling down one sampling point in depth.

[0045] The experience playback buffer contains experience samples for all depth points. For depth points where porosity calibration data exists, the action... Take the calibrated porosity For depth points where no calibration data exists, the action... Porosity estimates obtained from traditional well logging interpretation methods This estimate can be obtained by inverse calculation using the Archie formula or regional empirical formulas. Although its accuracy is limited, it provides a reasonable distribution of motion. All samples are used in training to ensure that the model fully utilizes the physical constraint signals in the full logging data, i.e., the Archie formula residuals. and stratigraphic continuity penalty The design intent of the reward function in S1 Figure 1 The capacity of the experience playback buffer D is set to... A certain number of samples are used to ensure sufficient sample diversity.

[0046] Then, a policy network and a double-Q value network are constructed. The policy network is denoted as... The parameters are Its input is the state. The output is the predicted porosity value at the current depth point. That is, action The last layer of the policy network is a linear layer, and its output is denoted as... This is followed by a Sigmoid activation function and multiplied by a scaling factor of 0.4 to obtain the final output. This mechanism ensures that the output action strictly falls within the specified time frame. Within the range, the differentiability of the Sigmoid function allows the gradient to be backpropagated normally.

[0047] A dual-Q network consists of two main Q-value networks with identical structures but independent parameters. and The parameters are respectively and and two corresponding target Q-value networks. and The parameters are respectively and All networks employ a fully connected neural network architecture, with the input layer dimension equal to the state space. The target Q-value network has two hidden layers with 256 neurons each, using ReLU activation. The output layer is a single neuron. The policy network outputs action values, and the main Q-value network outputs Q-values. During initialization, all network parameters are initialized using a random normal distribution, and the target Q-value network parameters are initialized to the same values ​​as the main network. A target policy network is also constructed simultaneously. ,parameter Initialize as a policy network Same value.

[0048] It should be noted that in this embodiment, the state space S is composed of three types of feature components, with a total dimension of 25, corresponding to a network input layer dimension of 25. The first component is the preferred logging curve feature extracted by the sliding window. After being filtered by the Pearson coefficient, 5 logging curves are retained. The sliding window contains 3 sampling points: the current depth, the previous adjacent depth, and the next adjacent depth. This part has a dimension of 5×3=15. The second component is the historical predicted porosity value of the 4 consecutive depths above the current depth. This part has a dimension of 4. The third component is the real-time working condition feature corresponding to the current depth, which includes 5 parameters: formation depth, drilling speed, mud resistivity, well diameter, and clay content. This part has a dimension of 6. The dimensions of the three components are added together to form a complete state space of 15+4+6=25. Therefore, all Q-value networks and strategy network input layers are uniformly set to 25 dimensions, which strictly matches the total dimension of the state space features.

[0049] A small batch of samples is randomly sampled from the experience replay buffer, denoted as . ,in The batch size is typically 256. Based on this batch of samples, the loss function of the double Q-value network is calculated. The training objective of the double Q-value network is to make the estimated Q-value approximate the target value of the Bellman optimal equation, while introducing a conservative regularization term to suppress the inflation of the Q-value estimate for unobserved actions. The specific form of the loss function is: ; in, For the j-th principal Q-value network The loss function; It is the j-th principal Q-value network parameter; This indicates the buffer from the experience replay. Calculate the expectation of the sampled data. The main Q-value network estimates the Q-value of the current state-action pair; It is the number of sampling actions; This is the kth sampling action; The target Q-value is calculated from the smaller value in the dual-Q value network, specifically as follows: ,in The immediate reward is calculated by the reward function in step S1. This is a discount factor with a value of 0.95. It is the Q-value network for the j-th objective. For the policy network in the next state The action to be output; The coefficient of the conservative regularization term is dimensionless and obtained through grid search on the validation set. Its specific value is... ; Actions actually observed in the dataset The corresponding Q value, These are the actions actually observed in the dataset, i.e., the actions in the current sample. , with quadruple In The same. Because the action space is a continuous interval. It is impossible to exhaustively list all actions. (The conservative regularization term...) Importance sampling approximation: random sampling from the action space One action ,use Comparative experiments show that the approximate logarithmic partition function achieves a better balance between the conservative approximation accuracy of Q value and the single-step computation cost when K=10; when K<5, the approximation deviation of the logarithmic partition function increases, and when K>20, the single-step computation time increases by more than 60%, while the regularization effect is not significantly improved.

[0050] The sampling distribution employs a hybrid strategy: based on probability From the action space Sampling is performed in a uniform distribution with probability. From the output of the policy network Mean and standard deviation The sampling is performed within a truncated Gaussian distribution. This hybrid strategy balances comprehensive coverage of the action space with fine-grained sampling of policy preference regions, effectively suppressing Q-value inflation for out-of-distribution actions while avoiding excessive deviation from the true actions in the dataset. In practical engineering, The value of Q needs careful adjustment: too large a value will lead to an overly conservative Q-value, making it difficult for the strategy to learn effective actions; too small a value will fail to effectively suppress valuation inflation. Optimization was performed on a validation set containing 5000 samples using five-fold cross-validation, and the final value was determined. .

[0051] By minimizing the loss function Update main Q value network parameters and The optimizer used is the Adam optimizer, and the learning rate is set to... Immediately after each training step, the parameters of the main Q-network are soft-copied to the target Q-network. The soft update method is as follows: ; in, These are the network parameters for the j-th objective Q-value; The soft update coefficient is dimensionless and is obtained through a grid search method using five-fold cross-validation on a validation set containing 5000 samples. The specific value is... The purpose of soft updates is to allow the target Q-value network parameters to slowly track the main network parameters, thereby improving training stability. In practical engineering, if the soft update coefficients... If the value is too large, the target Q-value may be updated too quickly, potentially leading to training instability; if If the target Q value is too small, the network update will be too slow, which may lead to a decrease in convergence speed. The target policy network parameters are synchronized using soft updates. ,in, These are the target policy network parameters. These are the main policy network parameters, specifically the policy network soft update coefficients. The value is set to 0.002, the same as the main Q-value network. Soft updates are performed immediately after each training step.

[0052] While updating the master Q-value network, the policy network parameters are updated based on the updated master Q-value network. The training objective of the policy network is to maximize the expected cumulative reward, i.e., to maximize the output of the master Q-value network through gradient ascent. The loss function of the policy network is: ; in, Let be the loss function of the policy network; This indicates the buffer from the experience replay. Calculate the expectation of the sampled states; For the current state in a double Q-value network and policy network output actions The smaller of the two Q values ​​is used as the target value. This technique of using the smaller of the two Q values ​​effectively reduces the overestimation bias of the Q value and improves the stability of the policy. The policy network also uses the Adam optimizer, with a learning rate set to... .

[0053] Repeat the sampling, loss calculation, and network parameter update steps described above until the policy network converges. The convergence criterion uses early stopping: calculate the average reward for each training epoch on the validation set; if the average reward on the validation set does not exceed the historical high value over 100 consecutive training epochs, the policy network is considered converged. The validation set consists of logging data from a single complete well reserved in the old block; this well is not used in training but is used to evaluate the model's generalization ability. After training, save the policy network parameters to obtain the basic porosity prediction model.

[0054] In practical engineering, it is important to note that the range of the Q-value needs to be monitored during training. If the Q-value diverges, such as when its absolute value exceeds the anomaly warning threshold, it will be detected. Training should be stopped immediately and the coefficients of the conservative regularization term should be checked. Is it too small? The logging data of old blocks may contain noise anomalies. The data should be cleaned before training to remove obviously abnormal depth points, such as well diameter enlargement sections and severe mud invasion sections, to avoid the model learning incorrect mapping relationships. The division of the training set and the validation set should be done by well, that is, using data from several complete wells for training and data from another complete well for validation, in order to test the model's cross-well generalization ability.

[0055] It should be noted that the calibration basis for the aforementioned abnormal warning threshold is as follows: In this scheme, the immediate reward R is obtained by weighted summation of three terms: the core error term, the Archie formula residual term, and the formation continuity penalty term. The first three terms are all absolute value errors or residuals, typically on the order of magnitude of... Within this range, the theoretical upper limit of the single-step instant reward R is approximately [value missing]. With a discount factor γ=0.95, under the infinite horizontal discount cumulative return framework, the theoretical steady-state upper limit of Q is approximately R / (1-γ)=0.1 / 0.05=2. Therefore, the anomaly warning threshold is set to... This threshold is approximately 50 times the theoretical steady-state upper limit, which is far greater than the reasonable fluctuation range of Q-values ​​during normal training, and can effectively identify Q-value estimation divergence caused by insufficient conservative regularization. This threshold is dimensionless and consistent with the dimensions of the porosity prediction value.

[0056] Step S3: Collect the full logging data and a small amount of porosity calibration data of the current prediction block, fix the hidden layer parameters of the porosity prediction basic model, and fine-tune the output layer of the strategy network and the double Q value network in the porosity prediction basic model based on the full logging data and the small amount of porosity calibration data to obtain the formation porosity prediction model.

[0057] Please see Figure 3 This step, based on the porosity prediction model obtained in step S2, transfers it to the current prediction block. It then uses the full volume of well logging data and a small amount of porosity calibration data from the new block for fine-tuning and training to obtain the formation porosity prediction model. The core of this step is to address the scarcity of porosity calibration data in the new block through transfer learning, while retaining the well logging-porosity mapping knowledge and physical constraint capabilities learned from the old block in the porosity prediction model.

[0058] First, acquire the full well logging data and a small amount of porosity calibration data for the current predicted block. The full well logging data includes numerical sequences of natural gamma curves, neutron curves, density curves, sonic transit time curves, and resistivity curves (the same type as in step S1), along with depth indices for each curve, and corresponding temperature, pressure, and well inclination angle data. The small amount of porosity calibration data typically consists of porosity calibration values ​​and their depth indices at 5 to 10 depth points. This data originates from cored sections of the new block. In practical engineering, new blocks often have only a few cored wells or scattered cored points, making core data extremely valuable. Therefore, it is necessary to fully utilize these limited calibration signals to guide the model in adapting to the rock physical properties of the new block.

[0059] Load the porosity prediction base model trained in step S2 into the current prediction block. This model includes a policy network. and double Q value network , The target Q-value network and its corresponding target Q-value network are used. The key strategy for transfer training is to fix the hidden layer parameters of the strategy network and the dual Q-value network, and only fine-tune the output layers of the strategy network and the dual Q-value network. The target Q-value network parameters automatically track the fine-tuning progress of the corresponding main network through a soft update mechanism, without the need to set a separate fine-tuning strategy. Since the physical response mechanism of the logging curves in the old and new blocks is the same, the features extracted by the hidden layer have cross-block transferability. This is because the hidden layer mainly learns the general mapping relationship between the logging curve morphology and the formation physical properties. These features have good stability across different blocks, while the output layer is responsible for mapping the general features to specific porosity value ranges. It needs to be calibrated according to the lithological characteristics and porosity distribution of the new block. Therefore, only the output layers of the strategy network and the dual Q-value network are fine-tuned.

[0060] The fine-tuning training process is similar to the pre-training in step S2, but with the following important differences. The experience replay buffer is constructed from the full logging data of the new block. However, since the amount of logging data in the new block may be small, such as only one or a few wells, the capacity of the experience replay buffer can be appropriately reduced, set to... Samples. The learning rate needs to be significantly reduced during fine-tuning to prevent damage to the backbone network features of the porosity prediction base model. Fine-tuning the learning rate... The validation set, consisting of approximately 2000 samples representing 20% ​​of the logging depths reserved in the new block, was determined through a grid search on a pre-reserved validation set within the new block, excluding core calibration points. The average reward on the validation set comprises only the Archie formula residual term and the formation continuity penalty term, used to monitor the model's adaptation to the physical laws of the new block; calibration accuracy is monitored separately using the core error term from the training set. This design avoids splitting the validation set on scarce calibration data while ensuring independent verification of physical constraint signals. The final result is determined by comparing the average reward function on the validation set under different learning rates. The number of training rounds for fine-tuning is much less than that for pre-training. An early stopping method is used to monitor the average reward on the validation set. Training is stopped when the average reward on the validation set does not exceed the historical high value for 20 consecutive training rounds.

[0061] This embodiment presents unique technical challenges in the offline reinforcement learning cross-block fine-tuning process: Differences in the distribution of formation logging data across different blocks can easily lead to Q-value distribution shifts, causing the general formation estimation features learned during the pre-training phase to become invalid; simultaneously, the porosity values ​​output by the policy network's output layer directly participate in the calculation of the Archie formula residual within the reward function, resulting in deep coupling between the policy output layer and the reward function. Large-scale updates to network parameters can disrupt the rock physics matching relationship established during pre-training. To address these issues, this embodiment sets specific constraints: the coefficient of the conservative regularization term remains constant throughout the fine-tuning process, continuously suppressing the expansion of Q-value estimations for unobserved actions; model convergence determination does not solely rely on the accuracy of sparse core calibration but also simultaneously monitors the physical residuals of the Archie formula, prioritizing that the predicted curve conforms to the formation gradient law, significantly reducing the overfitting problem caused by a small number of calibration points.

[0062] Target policy network parameters Through soft update mechanism The system automatically tracks the fine-tuning progress of the main policy network, ensuring that the target Q-value calculation is consistent with the current policy. During fine-tuning, the form of the loss function is exactly the same as during pre-training, including the Bellman residual term and conservative regularization term of the dual Q-value network, as well as the expected cumulative reward maximization term of the policy network. However, it is worth noting that the reward function for new blocks... In this model, the core error term is only valid at depths where porosity calibration data exists, while the Archie formula residual term and formation continuity penalty term can be calculated at all depths. This allows the model to obtain sufficient learning signals through dense physical constraint signals and geological priors, even with only 5 to 10 core calibration points. It should be noted that the Archie parameters determined in step S1 are based on rock electrical experimental data from the old block. If there are significant differences in lithology and formation water properties between the new and old blocks, the Archie parameters should be re-determined before fine-tuning; if the differences are small, the parameters from the old block can be used as initial values, and the Archie formula residual term can be used during fine-tuning. Indirect correction via residual feedback. Conservative regularization coefficient during fine-tuning. It remains unchanged at 0.5 because the data distribution of the new block may be offset relative to the old block. The conservative regularization term helps to suppress the valuation inflation of off-distribution actions and prevent negative migration.

[0063] After fine-tuning, a formation porosity prediction model adapted to the current prediction block is obtained. It retains the general logging feature extraction capability of the pre-training stage, and at the same time, the output layer is calibrated by a small number of calibration points, so that it can accurately predict the porosity value of the new block.

[0064] In practical engineering, it is important to note that the logging curves of the new block may have different calibration deviations or instrument model differences compared to the old block. Therefore, before migration, the logging data needs to be normalized and aligned. This means using the normalization parameters of the old block to perform the same normalization processing on the logging curves and working status data of the new block, ensuring that the input distribution of temperature, pressure, and well inclination angle is consistent with the pre-training stage. If the porosity calibration data of the new block has less than 5 depth points, fine-tuning cannot be performed. In this case, the porosity prediction basic model can be used directly for zero-sample prediction, but the prediction accuracy may decrease. In this case, the physical rationality of the prediction results can be ensured by relying on the residual term of Archie's formula and the formation continuity penalty term. During the fine-tuning process, the range of Q value changes should be monitored. If abnormal fluctuations in the Q value occur, the data of the new block should be checked for serious noise or systematic deviations. If necessary, the data should be filtered.

[0065] Step S4: Obtain real-time logging data and real-time working status data of the current prediction block, and input them into the formation porosity prediction model point by point according to depth, and output the porosity prediction value of the current depth point; update the state space based on the porosity prediction value of the current depth point and recursively push to the next depth point until all depth points are predicted, and finally generate the formation porosity prediction result of the logging tool.

[0066] This step, based on the formation porosity prediction model obtained in step S3, uses real-time logging data and real-time operational status data of the current prediction block to perform depth-by-depth inference, generating formation porosity prediction results for the entire well section. This step is the final application stage of the entire method, deploying the trained model to actual logging interpretation operations, achieving end-to-end prediction from logging curves to porosity curves.

[0067] First, real-time logging data and real-time operational status data for the current predicted block are acquired. Real-time logging data includes numerical sequences of natural gamma curves, neutron curves, density curves, sonic transit time curves, and resistivity curves (the same type as in step S1), along with depth indices for each curve. This data is continuously acquired downhole along the depth and transmitted to the surface in real-time by the logging tool. Real-time operational status data includes temperature, pressure, and well inclination angle data at the current depth point. This data is used to correct the logging response and evaluate data quality. In practical engineering, real-time logging data may experience transmission delays or packet loss; therefore, a buffer queue needs to be set up at the receiving end to ensure data integrity before subsequent processing.

[0068] The acquired real-time logging data is captured by a preset sliding window and combined with the predicted upper porosity sequence and real-time working status data to construct the state space at the current depth. The construction method of the state space is completely consistent with step S1. The sliding window captures the values ​​of each preferred logging curve at the current depth and the adjacent depths above and below as the first component of the state space. Each preferred logging curve is the logging curve retained after being filtered by Pearson correlation coefficient in step S1. The threshold for filtering Pearson correlation coefficient is set to 0.7, and only logging curves with a correlation coefficient higher than 0.7 with the measured core porosity value are retained as preferred logging curves. The predicted porosity values ​​of the four consecutive depths traversed above the current depth are extracted as the second component of the state space. The real-time working status data corresponding to the current depth is extracted as the third component of the state space. For the first depth point, since there are no predicted porosity values ​​yet, the predicted upper porosity sequence is empty. Therefore, the average porosity value of this block is used to fill the gap. The average porosity is initially interpreted by analyzing the logging responses of all depth points in the full logging data of the new block in step S3. The approximate porosity values ​​for each depth point are calculated using Archie's formula, and then the arithmetic mean is taken, resulting in an average porosity of 0.12. This initialization strategy ensures the integrity of the state space at the starting position and avoids inference failures caused by empty values.

[0069] It should be noted that the threshold of 0.7 for the Pearson correlation coefficient screening was determined through interval comparison experiments: four screening thresholds of 0.5, 0.6, 0.7, and 0.8 were tested respectively, and the number of retained curves and porosity prediction accuracy of each group were statistically analyzed. When the threshold is below 0.7, a large number of weakly correlated interference curves will be included, and the model prediction error will increase significantly. When the threshold is above 0.7, some moderately effective features will be removed, resulting in insufficient input information. Considering the average absolute error of the model and the completeness of features, 0.7 was finally selected as the fixed screening threshold, and only logging curves with a correlation coefficient higher than 0.7 with the measured core porosity values ​​were retained as preferred logging curves.

[0070] The constructed state space is input into the formation porosity prediction model. This model, obtained through fine-tuning and training in step S3, includes a policy network. and double Q value network , During the inference phase, only the policy network is used for prediction; the dual-Q network does not participate in the forward computation. The policy network is based on the current state. As input, output the predicted porosity value at the current depth point. The calculation process is as follows: ; in, For policy networks, parameters This has been fixed in step S3 of the fine-tuning training and will not be updated again. , The final predicted value is obtained by the linear output of the last layer of the policy network after Sigmoid activation and scaling, which is consistent with the definition in step S2. This is the predicted porosity value at the current depth point, dimensionless, and ranging from 0 to 0.4. The state space for the current depth point is composed of the three components mentioned above. The forward propagation process of the policy network does not involve random sampling or gradient calculation, so the inference speed is extremely fast. The prediction time for a single depth point is usually in the millisecond range, which meets the timeliness requirements of real-time well logging interpretation.

[0071] Porosity prediction at the current depth point After output, it is stored in the predicted upper-layer porosity sequence, and the sliding window is updated by rolling down one sampling interval along the depth direction to construct the state space for the next depth point. The specific process of state update is as follows: predict the porosity at the current depth. Append the data to the end of the predicted upper porosity sequence, while removing the oldest element in the sequence, maintaining a sequence length of 4 sampling points, consistent with the sliding window length in step S1; move the logging curve sliding window downwards along the depth direction by one sampling interval, discarding the uppermost depth point and incorporating the newly added depth point below; combine the updated logging curve window, the updated predicted porosity sequence, and the real-time working status data of the next depth point to obtain the state space of the next depth point. This state update process can be represented as: ; in, This is the state space for the next depth point; This represents the state space of the current depth point; This is the predicted porosity value at the current depth point; Depth d The well logging curve data vector at the location contains the values ​​of each preferred curve; For depth d The real-time operating status data vector includes temperature, pressure, and well inclination angle; It is a state update function that performs sliding window scrolling and sequence update operations, i.e., the deterministic state transition described in step S1. Both the action-independent and action-dependent parts are consistent with the definition of S1, ensuring that the Markov properties of the reasoning process and the training process remain consistent.

[0072] Repeat the steps of constructing the state space, inputting model predictions, and updating the state until all depth points are predicted. Concatenate the predicted porosity values ​​of all depth points in ascending order of depth index to generate the well logging tool's formation porosity prediction result, i.e., a complete whole-well porosity curve. This curve has the same depth range and sampling interval as the input well logging curve and can be directly used for subsequent geological interpretation and reserve assessment.

[0073] This technical solution reconstructs the formation porosity prediction problem in oil well logging as a Markov decision process, constructing a three-component state space including a sliding window of the logging curve, the upper porosity sequence, and working state data. A reward function is designed that integrates core error terms, Archie's formula physical constraints, and formation continuity penalty terms. An offline actor-commentator algorithm with conservative regularization is pre-trained in old blocks with abundant core samples to obtain a basic porosity prediction model. Addressing the scarcity of calibration data in new blocks, hidden layer parameters are fixed, and only the output layers of the policy network and the dual-Q value network are fine-tuned for transfer learning. Finally, through recursive inference at depth points, a porosity prediction curve for the entire well section is generated. This invention effectively solves the problems of traditional methods failing to embed physical equations, neglecting vertical geological patterns, and overfitting under sparse calibration conditions, significantly improving prediction accuracy and physical consistency.

[0074] Please see Figure 4 The present invention also protects a reinforcement learning-based formation porosity prediction system for oil well logging tools, comprising a memory and a processor. The memory stores computer programs and runtime data, and the processor executes the computer program to implement the steps of the above-described method. The processor drives the following functional modules to work collaboratively: The historical data acquisition and modeling module reads the offline historical dataset of old blocks with sufficient core samples, constructs a Markov decision process model based on this dataset, and writes the constructed model data into the shared data storage unit. The offline pre-training module reads the offline historical dataset and the Markov decision process model from the shared data storage unit, completes the model pre-training with an offline actor-commentator algorithm with conservative regularization terms, generates a porosity prediction base model, and saves it back to the shared data storage unit. The transfer fine-tuning module reads the full logging data and a small amount of porosity calibration data of the current block to be predicted, retrieves the porosity prediction base model from the shared data storage unit, fixes the hidden layer parameters of the base model, and performs fine-tuning training only on the output layers of the strategy network and the double Q-value network to obtain a formation porosity prediction model for the target block. The real-time inference module collects the real-time logging data and instrument real-time working status data of the current prediction block, inputs the trained formation porosity prediction model, outputs the porosity prediction value at the corresponding depth point, and finally generates the formation porosity prediction result of the logging tool. In this embodiment, the historical data acquisition and modeling module, the offline pre-training module, the migration fine-tuning module, and the real-time inference module complete data interaction and business collaboration through a shared data storage unit set inside the memory.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, the phrase "comprising an element defined as..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their likenesses.

Claims

1. A method for predicting formation porosity of a petroleum logging instrument based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Obtain the offline historical dataset of old blocks with sufficient core samples. The offline historical dataset includes historical logging data, historical working status data, and historical porosity calibration data. A state space for a Markov decision process is constructed based on the historical logging data and historical working status data, and a reward function is constructed based on the historical porosity calibration data; a Markov decision process model is then constructed based on the state space and the reward function. Obtain offline historical datasets of old blocks with sufficient core samples, including historical logging data, historical working status data, and historical porosity calibration data; The historical logging data includes numerical sequences of natural gamma curves, neutron curves, density curves, sonic transit time curves, and resistivity curves, as well as depth indices for each curve. The historical working status data includes temperature data, pressure data, and well inclination angle data of the logging tool at various depth points; The historical porosity calibration data includes the core calibration porosity values ​​at each core depth point and their corresponding depth index. The historical logging data is subjected to curve filtering and standardization. A sliding window construction logic similar to image sequence samples is adopted. A logging curve window of a preset length is slidably extracted along the depth direction with a preset step size as the first component of the state space. The logging curve window records the feature values ​​of various logging curves along the depth in matrix form. The porosity value sequence of multiple consecutive depths above the current depth is extracted as the second component of the state space. The historical working state data corresponding to the current depth is extracted as the third component of the state space. In the offline pre-training stage, the second component is obtained by interpolation or regional averaging of historical porosity calibration data. In the online inference stage, it is obtained by rolling updates of the output historical prediction values. Combining the first component, the second component, and the third component yields the state space of the Markov decision process. A reward function is constructed based on the historical porosity calibration data. A core error term is constructed. When historical porosity calibration data exists, the core error term is the absolute error between the predicted porosity and the core calibration porosity; when no historical porosity calibration data exists, the core error term is zero. An Archie formula residual term is constructed based on the logarithmic domain absolute residual between the measured resistivity calculated by substituting the predicted porosity into the Archie formula and the formula-fitted resistivity. A formation continuity penalty term is constructed based on the absolute value of the difference between the predicted porosity at the current depth and the predicted porosity at the previous depth. The negative core error term, the negative Archie formula residual term, and the negative formation continuity penalty term are weighted and summed to obtain the reward function of the Markov decision process. The water saturation in the Archie formula residual term is provided by the existing independent logging interpretation results of the current prediction block, or is predetermined by logging methods independent of this prediction, or uses the typical value of the block as the input constant. A Markov decision process model is constructed based on the state space and reward function. Step S2: Based on the offline historical dataset, the Markov decision process model is pre-trained using an offline actor-critic algorithm with a conservative regularization term, wherein the actor network is a policy network and the critic network is a double Q-value network, to obtain the basic model for porosity prediction. Step S3: Collect the full logging data and a small amount of porosity calibration data of the current prediction block, fix the hidden layer parameters of the porosity prediction basic model, and fine-tune the output layer of the strategy network and the double Q value network in the porosity prediction basic model based on the full logging data and a small amount of porosity calibration data to obtain the formation porosity prediction model. Step S4: Obtain real-time logging data and real-time working status data of the current prediction block, and input them into the formation porosity prediction model point by point according to depth, and output the porosity prediction value of the current depth point; update the state space based on the porosity prediction value of the current depth point and recursively push to the next depth point until all depth points are predicted, and finally generate the formation porosity prediction result of the logging tool.

2. The method for predicting formation porosity using a reinforcement learning-based oil well logging tool according to claim 1, characterized in that, The Markov decision process model constructed based on the state space and reward function includes: Based on the state space and reward function, the formation porosity prediction problem is reconstructed into a Markov decision process; The action space of the Markov decision process is the porosity prediction value at the current depth point, and the porosity prediction value is a continuous real number with a value range of 0 to 0.

4. The state transition of the Markov decision process is a deterministic transition: after the current depth point is predicted, the state is rolled down by one sampling interval in the depth direction, the logging curve sliding window is updated accordingly, and the state is automatically transferred to the next depth point. The discount factor in the Markov decision process is a preset value used to balance the importance of current rewards and future rewards; Based on the state space, action space, state transition probability, reward function, and discount factor, a Markov decision process model is constructed.

3. The method for predicting formation porosity using a reinforcement learning-based oil well logging tool according to claim 1, characterized in that, Step S2 includes: Based on the offline historical dataset, an experience replay buffer is constructed, which is used to store training samples. Construct the policy network and the double Q-value network, and initialize the network parameters of the policy network and the double Q-value network; A small batch of samples is randomly sampled from the experience replay buffer. The loss functions of the policy network and the double Q-value network are calculated based on the small batch of samples. The network parameters are iteratively updated using the gradient descent method until the policy network converges, thus obtaining the basic model for porosity prediction.

4. The method for predicting formation porosity using a reinforcement learning-based oil well logging tool according to claim 3, characterized in that, The iterative update of network parameters using gradient descent includes: Based on the small batch of samples, the Q-values ​​of the current state-action pairs are estimated using a dual Q-value network, and a smaller Q-value is selected as the target value to reduce overestimation bias. A conservative regularization term is introduced when calculating the loss function to suppress the inflation of the Q-value estimate for unobserved actions; The parameters of the main Q-value network are updated by minimizing the loss function, and the parameters of the main Q-value network are soft-copied to the target Q-value network after each training step. At the same time, based on the updated master Q-value network, the network parameters of the strategy are updated by maximizing the expected cumulative reward; Repeat the above steps until the policy network converges.

5. The method for predicting formation porosity using a reinforcement learning-based oil well logging tool according to claim 1, characterized in that, Step S3 includes: Obtain full logging data and a small amount of porosity calibration data for the current predicted block; The porosity prediction model is transferred to the current prediction block. The hidden layer parameters of the strategy network and the double Q-value network are fixed. Based only on the full logging data and a small amount of porosity calibration data of the current prediction block, the output layer of the strategy network and the output layer of the double Q-value network are fine-tuned and trained to obtain the formation porosity prediction model.

6. The method for predicting formation porosity using a reinforcement learning-based oil well logging tool according to claim 1, characterized in that, Step S4 includes: Obtain real-time logging data and real-time working status data for the current predicted block; The real-time logging data is extracted according to a preset sliding window and combined with the predicted upper porosity sequence and the real-time working status data to construct the state space of the current depth point. The state space is input into the formation porosity prediction model, and the porosity prediction value at the current depth point is output. The formation porosity prediction result of the logging tool is generated based on the porosity prediction value at the current depth point.

7. The reinforcement learning-based formation porosity prediction method for oil well logging tools according to claim 6, characterized in that, The generation of formation porosity prediction results based on the porosity prediction value at the current depth point includes: The predicted porosity value at the current depth point is stored in the predicted upper porosity sequence, and the sliding window is updated by rolling it down one sampling interval along the depth direction to construct the state space of the next depth point. Repeat the above steps until all depth points are predicted. Then, stitch together the porosity prediction values ​​of all depth points according to the depth index to generate the formation porosity prediction results of the logging tool.

8. A formation porosity prediction system for oil well logging tools based on reinforcement learning, characterized in that, include: Memory is used to store computer programs and running data; A processor that, when executing the computer program, implements the steps of the method according to any one of claims 1-7; The processor drives the following functional modules to work together: The historical data acquisition and modeling module is used to acquire offline historical datasets of old blocks with sufficient core samples, and to construct a Markov decision process model based on the offline historical datasets. The offline pre-training module is used to pre-train the Markov decision process model based on the offline historical dataset and an offline actor-critic algorithm with a conservative regularization term to obtain the porosity prediction base model. The migration fine-tuning module is used to acquire the full logging data and a small amount of porosity calibration data of the current prediction block, fix the hidden layer parameters of the porosity prediction basic model, and fine-tune the training only on the output layers of the strategy network and the double Q value network to obtain the formation porosity prediction model. The real-time inference module is used to acquire real-time logging data and real-time working status data of the current prediction block, input the formation porosity prediction model, output the porosity prediction value at the current depth point, and generate the formation porosity prediction result of the logging tool. The historical data acquisition and modeling module, offline pre-training module, transfer fine-tuning module, and real-time inference module interact and collaborate through a shared data storage unit configured in the memory.

Citation Information

Patent Citations

  • Process for real time geological localization with reinforcement learning

    US20210310307A1