Automatic horizon tracking method based on wavelet geometrical morphology attributes

By adopting an automatic stratigraphic tracking method based on wavelet geometry properties, the problems of deviation from the trajectory and noise interference in complex seismic data in existing automatic stratigraphic tracking technologies have been solved, achieving efficient and accurate stratigraphic picking and improving the efficiency of oil and gas exploration.

CN119024427APending Publication Date: 2024-11-26CHINA PETROLEUM & CHEMICAL CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310584690.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing automatic horizon tracking methods are prone to deviating from the picking trajectory when processing seismic data with weak signals, discontinuous reflection signals, or complex structures. Furthermore, noise interference can cause misalignment of horizon peaks, resulting in low horizon interpretation efficiency.

Method used

An automatic layer-tracking method based on wavelet geometric morphology attributes is adopted. By preprocessing seismic data, extracting multi-attribute features, establishing a high-dimensional morphological attribute set, selecting seed points, and using a similarity coefficient and reward function model combined with the Markov optimal decision process, layer picking is achieved.

Benefits of technology

It improves the efficiency and accuracy of stratigraphic interpretation, enabling accurate stratigraphic extraction from complex seismic data and shortening the timeframe for oil and gas exploration projects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024427A_ABST
    Figure CN119024427A_ABST
Patent Text Reader

Abstract

The invention provides an automatic horizon tracking method based on wavelet geometric morphology attributes, and the method comprises the steps: inputting seismic data, and carrying out the preprocessing of the seismic data; extracting multi-attribute features, and establishing a high-dimensional morphological attribute set; multiple pieces of attribute information related to seismic event information are selected, a more direct relation between data and effective information is established, and seed points are selected manually; extracting common information of spatial horizon attributes through the similarity coefficient, and establishing a reward function model; and obtaining a horizon pickup result based on the seed points and multi-attribute constraints. And the horizon interpretation efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of geophysical exploration, and in particular to an automatic layer tracking method based on wavelet geometric morphological attributes. Background Art

[0002] Stratum interpretation is a routine and crucial task in analyzing underground structures in oil and gas exploration. Accurate stratigraphic interpretation plays a vital role in constraining velocity modeling, improving velocity model and imaging accuracy, establishing accurate reservoir models, and reliably estimating the oil and gas volume in the target reservoir. However, stratigraphic interpretation has always been a labor-intensive and time-consuming task.

[0003] Therefore, the industry is constantly committed to researching efficient and accurate automatic picking algorithms, hoping to improve the efficiency of horizon interpretation through automatic picking algorithms. Good automatic horizon tracking technology can significantly improve the efficiency of reservoir structure analysis, thereby shortening the time frame of oil and gas exploration projects. However, some characteristics of seismic data may bring certain difficulties to these picking methods. For example, the size of the horizon and the spacing between horizons are not constant in the entire seismic work area; when a single horizon is discontinuous, even experienced interpreters often find it difficult to track a single horizon completely based on seismic data. The presence of noise in seismic images will cause the peak of the detected seismic horizon to be misplaced. When using linear feature detection filters, these noise points with small-scale features may also be regarded as the response of the real horizon. Although simple and rough smoothing can improve the continuity of the horizon and suppress noise, it will also eliminate important structural features and produce horizons that do not really exist in the original data. At present, the horizon picking methods are mainly divided into three categories: dip-driven method, global optimization method, and horizon slicing method. Among all currently available automatic layer tracking methods, waveform-based automatic tracking combined with cross-correlation of adjacent traces is the most commonly used in practice.

[0004] However, the waveform-based automatic tracking method has a prominent disadvantage, that is, when there are strong reflections nearby, the tracking tool is easy to deviate from the established picking trajectory. Generally speaking, when the target reflection has strong amplitude contrast, good lateral continuity, and high signal-to-noise ratio, the layer picking tool can produce reliable results. However, when the processing signal is weak, the reflection signal is discontinuous, and the structure is complex, automatic tracking is still challenging. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide an automatic layer tracking method based on wavelet geometric attributes, which overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of the present invention, a layer automatic tracking method based on wavelet geometric attributes is provided, and the automatic tracking method comprises:

[0007] Step 1: input seismic data and preprocess the seismic data;

[0008] Step 2: Extract multi-attribute features and establish a high-dimensional morphological attribute set;

[0009] Step 3: Select multiple attribute information related to the seismic event information, establish a more direct connection between the data and the effective information, and manually select seed points;

[0010] Through the similarity coefficient, the common information of spatial layer attributes is extracted and the reward function model is established;

[0011] Step 4: Based on the seed points and multi-attribute constraints, the layer picking results are obtained.

[0012] Optionally, the step 2: extracting multi-attribute features and establishing a high-dimensional morphological attribute set specifically includes:

[0013] According to the positive and negative relationship of the seismic data on the seismic trace, the geometrical characteristic attributes of the wavelet are described by using the relationship between a main lobe and two side lobes;

[0014] Calculate the geometric feature attributes.

[0015] Optionally, the geometric feature attributes are amplitude attributes, inclination attributes, image edge attributes and texture attributes.

[0016] Optionally, the step 4: obtaining the layer picking result based on the seed point and the multi-attribute constraint specifically includes:

[0017] Based on seed points and multi-attribute constraints, Markov optimal decision layer picking is performed;

[0018] Connect the picked points into complete horizons and perform post-processing;

[0019] Based on the attribute calculation, the wavelet feature constraint evaluation is performed to obtain the layer picking result.

[0020] Optionally, the seismic data includes: time-offset profile and imaging gathers.

[0021] Optionally, the preprocessing of the seismic data specifically includes: extracting main event axis information of the seismic data by using a preprocessing technique for denoising and enhancing seismic structure information.

[0022] Optionally, the step 2: extracting multi-attribute features and establishing a high-dimensional morphological attribute set specifically includes:

[0023] Converting the seismic data into high-dimensional space attributes, wherein the high-dimensional space attributes include: wavelet morphology attribute parameters, dip angle attributes, amplitude attributes, image edge attributes, and structure tensors;

[0024] The sub-wave morphological attribute parameters are calculated through positive and negative relationships, including the total width of the sub-wave, peak width, trough width, peak position, trough position, extreme point position, peak amplitude and trough amplitude.

[0025] Optionally, extracting the commonality information of spatial layer attributes through the similarity coefficient specifically includes: measuring the similarity of the wavelet morphological attributes of adjacent channels by calculating the similarity coefficient. The calculation formula of the similarity coefficient r is as follows:

[0026]

[0027] Among them, x and y correspond to the vectors composed of the attributes of the current track and the adjacent track respectively;

[0028] When constructing the vector, the length n of the vector is determined according to actual conditions.

[0029] Optionally, the establishing of the reward function model specifically includes:

[0030] Construct the reward function R based on the similarity coefficients of each attribute:

[0031]

[0032] A model is built for each attribute, and different attributes determine the proportion;

[0033] According to the ratio, multiply the reward value R by a coefficient;

[0034] a1, a2, and a3 represent the thresholds of the partition similarity coefficient r;

[0035] The reward function R is the reward or penalty signal fed back by the environment after the agent takes an action and interacts with the environment; it represents the quality of the action taken under a certain state, and is also an important basis for accurate layer identification.

[0036] Express the amplitude graph I(x,y) / I(x,y,z) or even I(x,y; t) / I(x,y,z; t) as I Vector (x,y) / I Vector (x, y, z) or even I Vector (x,y;t) / I Vector (x, y, z; t), which is to transform the image represented by amplitude into an image represented by vector;

[0037] The components in the vector are the attribute values ​​introduced into the feature expression of the image;

[0038] For each point in the profile, it is expressed as

[0039] I Vector (x i ,y j )=[a;b;c;d;e;f;g;h;i;j;k]

[0040] Characterize each point in the section;

[0041] The normalized absolute error is used to measure the similarity between points in adjacent paths, and the similarity coefficients corresponding to each attribute are weighted averaged:

[0042]

[0043] Wherein, ω1, ω2, ω3, and ω4 are the corresponding weight coefficients, k is the window range of horizontal dynamic matching, l is the window range of vertical dynamic matching, and (i, j) is taken as the reference wavelet to compare and match the waveform features within a certain window; when k=1, the wavelet waveform feature sequence is matched on two adjacent channels.

[0044] Optionally, the step 4: obtaining the layer picking result based on the seed point and the multi-attribute constraint specifically includes:

[0045] According to the reward value defined by the similarity in step 3, the layer identification problem is converted into tracing a path starting from the event axis seed point to maximize the total reward value of the path;

[0046] In layer identification, the seismic profile is regarded as an environmental model. Different coordinate points containing multi-attribute information in the environmental model represent different states, and each state corresponds to an element in the state space set S;

[0047] In the environment model corresponding to the seismic profile, there are states of points on the event axis and states of points on the non-event axis;

[0048] The Markov decision process is based on the state space, reward function, combined with the action space and the evaluation of the reward value, forming a complete Markov decision process.

[0049] The present invention provides an automatic layer tracking method based on wavelet geometric morphological attributes, the automatic tracking method comprising: inputting seismic data and preprocessing the seismic data; extracting multi-attribute features and establishing a high-dimensional morphological attribute set; selecting multiple attribute information related to seismic event information, establishing a more direct connection between data and effective information, and manually selecting seed points; extracting commonality information of spatial layer attributes through similarity coefficients, and establishing a reward function model; and obtaining layer picking results based on seed points and multi-attribute constraints. The efficiency of layer interpretation is improved.

[0050] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0052] Figure 1 A schematic flow chart of a method for automatic layer tracking based on wavelet geometric attributes provided by an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of the geometric characteristics of Ricker wavelet.

[0054] Figure 3 A schematic diagram of local seismic data of an application example of the present invention.

[0055] Figure 4 This is the seismic attribute profile extracted by the present invention.

[0056] Figure 5 The characteristic attribute vector constructed for the present invention.

[0057] Figure 6 This is the horizon picking result of the present invention. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0060] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0061] The invention discloses an automatic layer tracking method based on wavelet geometric morphological attributes. The method selects high-dimensional and multi-attribute basis, relies on Markov optimal state transfer and optimal decision framework, and performs automatic layer tracking. Among them, the wavelet morphological characteristics on the event axis are one of the typical characteristics of seismic data. Starting from the most basic element of seismic data, the seismic wavelet is taken as a meta-mode to study the expression of the seismic wavelet morphological characteristics and highlight the layer characteristics. The extraction of multiple attributes aims at high correlation between event axis structure information and attributes and low correlation between multiple attributes, removes invalid information such as noise interference in the source data body, and characterizes the structural information in the data from more angles, thereby establishing a more direct connection between data and effective information. Based on the morphological characteristics of the wavelet and the Markov transfer process, a set of algorithms and processes for automatic event axis tracking and picking are designed to realize the interaction process between the intelligent agent and the environment, and realize the process of maximizing the expected target of the intelligent agent in selecting actions.

[0062] The present invention is an automatic layer tracking method based on wavelet geometric morphology attributes, relying on the Markov optimal state transfer framework, mainly used for fast and accurate picking of seismic reflection event axes, providing regularization constraint information for seismic velocity modeling, so as to improve the accuracy of velocity modeling.

[0063] The method of the present invention first pre-processes the seismic data to obtain the corresponding state space. Then, according to the positive and negative changes of the seismic data (or the derivative value of the seismic data) along the time direction, the geometric morphological characteristic attributes of the wavelet are described by the relationship between a main lobe and two side lobes, and a high-dimensional characteristic attribute space is formed to establish an attribute characteristic vector. According to the vector, the similarity between two sampling points in adjacent channels is calculated respectively, and then the reward value is calculated. The tracking path with the largest reward value is the optimal path picked in the seismic profile.

[0064] like Figure 1 As shown, in step 1, input post-stack seismic data. The sampling interval of the post-stack seismic profile in the time direction is 4 milliseconds; the sampling interval in the spatial direction is 25 meters. The profile is filtered along the event axis direction to suppress noise and effectively highlight the event axis.

[0065] In step 2, the seismic profile is differentiated to obtain a new data profile consisting of positive and negative values. Based on the profile, the wavelet geometric characteristics are described, including the main lobe width, side lobe width, trough position, peak position and corresponding amplitude parameters, and the wavelet geometric characteristic attributes such as wavelet width, peak position, trough position, and peak amplitude are extracted respectively. Then, the attributes such as amplitude envelope and dip information extracted from the seismic profile are combined to form a high-dimensional feature attribute space, and the feature attribute vector w is constructed based on the above attributes:

[0066]

[0067] In step 3, according to step 2 and Figure 4 The high-dimensional morphological attribute space constructed is used to transform the original seismic profile I(x,y) into I vector (x,y), Figure 3 The information of each sampling point in is expanded into a vector, which is as follows Figure 5 shown. Figure 3 Each sampling point in Figure 5 The feature attribute vector is shown in . Then the attribute vector corresponding to each sampling point is measured for similarity, its similarity coefficient is calculated, and then the reward function R is constructed. Figure 3 Each sampling point (x, y) on the seismic profile in is represented by a characteristic attribute vector w = (a; b; c; d; e; f; g; h; i; j). Then the extracted high-dimensional attribute profile, such as Figure 4 As shown in the figure, the similarity coefficient of each attribute at the current sampling point is calculated respectively. Taking the attribute profile A corresponding to attribute a as an example, at the sampling point (x, y), a neighborhood window of size 7×1 is taken, and the similarity coefficient r a(x,y) :

[0068]

[0069] like Figure 4 As shown in the figure, the similarity coefficients are calculated at the corresponding sampling points (x, y) of the high-dimensional attribute profile. The calculation results are recorded as:

[0070] r(x,y)=(r a(x,y) ; r b(x,y) ; r c(x,y) ; r d(x,y) ; r e(x,y) ; r f(x,y) ; r g(x,y) ; r h(x,y) ; r i(x,y) ; r j(x,y) )

[0071] In step 4, Figure 3 Each point in the Zhongyuan seismic profile is regarded as a state space, and the action space is given. According to the reward space constructed in step 3, the evaluation under the wavelet feature constraint is performed to achieve the optimal picking under the Markov optimal state transfer framework. On the one hand, the optimal attribute is selected; on the other hand, the optimal layer is picked. Both are carried out under the Markov optimal state transfer framework.

[0072] Figure 6 The green five-pointed star in the figure starts from the path where the seed point is located and traverses the neighborhood window of size 7×1. Figure 4Each single attribute section in the high-dimensional attribute space in the , performs preliminary layer picking, and according to the result, according to the standard of the accuracy of the picking result, Figure 4 The attributes in the multi-attribute space in are preliminarily screened. The screened results are fused using attribute fusion methods such as PCA.

[0073] Finally, according to the Markov decision process, several paths are tracked and the one with the largest reward value is selected as the final pickup result.

[0074] like Figure 2 As shown in the figure, the Ricker wavelet is divided into six segments: AB, BC, CD, DE, EF, and FG. Among them, AC and EG represent the side lobe part of the seismic wavelet, CE is the main lobe part of the seismic wavelet, D, B, and F are extreme points, D is the peak position, and B and F are the trough positions.

[0075] like Figure 3 As shown in the figure above, the post-stack seismic section after preprocessing, the main phase axis in the figure is relatively clear. This seismic section is subsequently used for attribute extraction and layer identification.

[0076] like Figure 4 As shown in the figure, the attribute profiles from left to right are wavelet width, main lobe width, side lobe width, peak position, trough position, number of extreme values, peak amplitude, trough amplitude, amplitude envelope, and dip information. As can be seen from the figure, the above geometric characteristic attributes clearly describe the characteristics of the event axis in the seismic profile. In view of the fact that some attributes have missing information in local areas, further attribute optimization is carried out, and multiple dominant attributes are combined for picking.

[0077] like Figure 5 As shown in the figure, Figure 3 The seismic data and extracted wavelet characteristic attributes at the 200th position in the profile shown. From left to right in the figure, the seismic data and wavelet characteristic attributes such as wavelet width, main lobe width, side lobe width, peak position, trough position, number of extreme values, peak amplitude, trough amplitude, amplitude envelope, and dip information are shown. It is found that due to the influence of noise, the required event axis cannot be directly determined and extracted through seismic data. The comprehensive use of multi-attribute information constraints can improve the picking effect.

[0078] like Figure 6 As shown, Figure 5 The waveform corresponding to the multiple thinning. In the figure, the five-pointed star symbol represents the manually given picking seed point. Based on the method of the present invention, automatic tracking is performed starting from the seed point, and the tracking picking result is consistent with the phase axis.

[0079] Beneficial effects: The method of the present invention can complete automatic picking, and is more efficient and accurate. Conventional picking methods are time-consuming and labor-intensive, and have low precision. The method of the present invention improves the efficiency of layer interpretation through an automatic picking algorithm. Good layer automatic tracking technology can significantly improve the efficiency of reservoir structure analysis, thereby shortening the time frame of oil and gas exploration projects; layer picking on imaging sections is of great significance for both processing and interpretation.

[0080] The method of the present invention is based on actual seismic data from an oil field in eastern China, extracts attribute information to construct a high-dimensional attribute feature vector, and obtains the picking results of the main event axis under the Markov optimal state transfer framework. The results show that the layer can be picked by extracting the geometric morphological characteristic attributes of the wavelet (such as the crest and trough positions, the wavelet width, etc.). The theoretical framework of this method is clear and complete, and has strict theoretical support. The focus of the method of the present invention is on the multi-attribute extraction method, and then combines the extraction of common information of spatial layer attributes and the optimal evolution method to achieve automatic layer picking.

[0081] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A layer automatic tracking method based on wavelet geometric attributes, characterized in that: The automatic tracking method comprises: Step 1: input seismic data and preprocess the seismic data; Step 2: Extract multi-attribute features and establish a high-dimensional morphological attribute set; Step 3: Select multiple attribute information related to the seismic event information, establish a more direct connection between the data and the effective information, and manually select seed points; Through the similarity coefficient, the common information of spatial layer attributes is extracted and the reward function model is established; Step 4: Based on the seed points and multi-attribute constraints, the layer picking results are obtained.

2. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The step 2: extracting multi-attribute features and establishing a high-dimensional morphological attribute set specifically includes: According to the positive and negative relationship of the seismic data on the seismic trace, the geometrical characteristic attributes of the wavelet are described by using the relationship between a main lobe and two side lobes; Calculate the geometric feature attributes.

3. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The geometrical morphological characteristic attributes are amplitude attributes, inclination attributes, image edge attributes and texture attributes.

4. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The step 4: based on the seed point and the multi-attribute constraint, obtaining the layer picking result specifically includes: Based on seed points and multi-attribute constraints, Markov optimal decision layer picking is performed; Connect the picked points into complete horizons and perform post-processing; Based on the attribute calculation, the wavelet feature constraint evaluation is performed to obtain the layer picking result.

5. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The seismic data include: time-offset profile and imaging gathers.

6. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The preprocessing of the seismic data specifically includes: extracting main event axis information of the seismic data by using a preprocessing technique of denoising and enhancing seismic structure information.

7. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The step 2: extracting multi-attribute features and establishing a high-dimensional morphological attribute set specifically includes: Converting the seismic data into high-dimensional space attributes, wherein the high-dimensional space attributes include: wavelet morphology attribute parameters, dip angle attributes, amplitude attributes, image edge attributes, and structure tensors; The sub-wave morphological attribute parameters are calculated through positive and negative relationships, including the total width of the sub-wave, peak width, trough width, peak position, trough position, extreme point position, peak amplitude and trough amplitude.

8. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The method of extracting the commonality information of spatial layer attributes through the similarity coefficient specifically includes: measuring the similarity of the wavelet morphological attributes of adjacent channels by calculating the similarity coefficient. The calculation formula of the similarity coefficient r is as follows: Among them, x and y correspond to the vectors composed of the attributes of the current track and the adjacent track respectively; When constructing the vector, the length n of the vector is determined according to actual conditions.

9. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The establishment of the reward function model specifically includes: Construct the reward function R based on the similarity coefficients of each attribute: A model is built for each attribute, and different attributes determine the proportion; According to the ratio, multiply the reward value R by a coefficient; a1, a2, and a3 represent the thresholds of the partition similarity coefficient r; The reward function R is the reward or penalty signal fed back by the environment after the agent takes an action and interacts with the environment. It represents the quality of the action taken under a certain state and is an important basis for accurate layer identification. Express the amplitude graph I(x,y) / I(x,y,z) or even I(x,y; t) / I(x,y,z; t) as I Vector (x,y) / I Vector (x, y, z) or even I Vector (x,y;t) / I Vector (x, y, z; t), which is to transform the image represented by amplitude into an image represented by vector; The components in the vector are the attribute values ​​introduced into the feature expression of the image; For each point in the profile, it is expressed as I Vector (x i ,y j )=[a;b;c;d;e;f;g;h;i;j;k] Characterize each point in the section; The normalized absolute error is used to measure the similarity between points in adjacent paths, and the similarity coefficients corresponding to each attribute are weighted averaged: Wherein, ω1, ω2, ω3, and ω4 are the corresponding weight coefficients, k is the window range of horizontal dynamic matching, l is the window range of vertical dynamic matching, and (i, j) is taken as the reference wavelet to compare and match the waveform features within a certain window; when k=1, the wavelet waveform feature sequence is matched on two adjacent channels.

10. The automatic layer tracking method based on wavelet geometric attributes according to claim 1 is characterized in that: The step 4: based on the seed point and the multi-attribute constraint, obtaining the layer picking result specifically includes: According to the reward value defined by the similarity in step 3, the layer identification problem is converted into tracing a path starting from the event axis seed point to maximize the total reward value of the path; In layer identification, the seismic profile is regarded as an environmental model. Different coordinate points containing multi-attribute information in the environmental model represent different states, and each state corresponds to an element in the state space set S; In the environment model corresponding to the seismic profile, there are states of points on the event axis and states of points on the non-event axis; The Markov decision process is based on the state space, reward function, combined with the action space and the evaluation of the reward value, forming a complete Markov decision process.