Underground control method for wave code communication underground water distributor based on Q-learning

Abstract

The invention provides a Q-learning-based underground control method for a wave code communication underground water distributor, and the method comprises the steps: receiving a starting instruction transmitted by a ground controller, and analyzing the starting instruction; after analysis succeeds, stratum environment data of the underground water distributor are obtained; whether a sudden change value exists in the stratum environment data or not is judged, and if the sudden change value does not exist, an action is obtained from the current Q table according to an epsilon-greedy decision algorithm to control a water nozzle switch of the underground water distributor; the Q table is a state-action value strategy table determined by a Q-learning model; a Q value is given to each of different actions in each state in the Q table, spontaneous adjustment of the water nozzle switch is achieved through the Q-learning model, and fine adjustment and control of the water nozzle switch can be achieved without ground control.

Description

Technical Field

[0001] The invention belongs to the technical field of oilfield water injection, and in particular relates to a downhole control method of a downhole water distributor based on wave code communication of Q-learning. Background Art

[0002] Water injection into oil fields is a key step in the oil extraction process, which can replenish formation energy and improve recovery efficiency. Due to geological influences, different physical properties and energy differences between oil layers, resource utilization of different oil layers is uneven. In order to improve the efficiency of oil layer extraction, oil fields have begun to develop the fourth generation of water injection technology, namely digital intelligent injection technology, which includes cable intelligent injection technology and wave code intelligent injection technology. Cable intelligent injection technology has complex process, high cost, and weak pressure operation capability. It is only suitable for some well conditions in oil fields and has a small application range; wave code intelligent injection technology has a wide range of applications due to its low cost, simple process and ability to meet pressure operations. Downhole water distributors can use wave code intelligent injection technology to carry out downhole operations, and communicate with downhole water distributors through ground controllers to control information-carrying carriers such as water injection pressure or flow. However, the traditional communication through water injection is one-way, and the ground controller can only send commands to the downhole water distributor, and then wait for the return code information from the downhole water distributor. The downhole water distributor cannot actively send messages to the ground controller, so that the formation environment data collected by the downhole water distributor cannot be obtained by the ground controller in the first time. The improper water injection of the ground controller is very likely to cause energy imbalance between oil layers, resulting in reduced oil production efficiency. In addition, the current downhole water distributor with wave code communication can only perform actions according to the instructions of the ground controller, and cannot actively transmit information to the ground, and cannot fully utilize the formation environment data. Therefore, how to make the downhole water distributor fully utilize the formation environment data for spontaneous operation is an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a downhole control method for a downhole water distributor with wave code communication based on Q-learning, in which the Q-learning model obtains the corresponding water nozzle switch execution action through the formation environment data, and can make full use of the formation environment data collected by the downhole water distributor to solve the problem that the downhole water distributor cannot be spontaneously operated and regulated underground.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: A downhole control method for a downhole water distributor using wave code communication based on Q-learning, the method comprising: Receive a start command sent by a ground controller and parse the start command; After successful analysis, the formation environment data of the downhole water distributor is obtained; Determine whether there is a mutation value in the formation environment data. If there is no mutation value, ε-greedy The decision algorithm starts from the current Q The meter obtains an action to control the water nozzle switch of the underground water distributor; Q The table is a state-action value strategy table determined by the Q-learning model; Q Each action in each state in the table is assigned a Q value.

[0005] Further, after the water nozzle switch of the downhole water distributor is executed, new formation environment data is obtained; Update the new formation environment data and error curve Q Table; the error curve is the return value converted from the difference between the preset value and the new formation environment data.

[0006] Furthermore, the update Q Table Q The iteration formula of the value is as follows: ; in, Indicates the status, Indicates action, Indicates updated Q value, Indicates the last updated value, is the learning rate, Represents the return value, is the discount factor, Indicates execution of an action The maximum value of the next state Q value.

[0007] Furthermore, the return value R The calculation formula is as follows: ; in, Indicates the collected flow value. Indicates the preset flow value. Indicates the collected pressure value, Indicates the preset pressure value, represents the normalized weight coefficient of the traffic, represents the normalized weight coefficient of pressure, and , represents the flow difference threshold, Indicates the pressure difference threshold.

[0008] Furthermore, if there is a mutation value, the mutation value is transmitted back to the ground controller according to the reverse communication protocol, so that the ground controller resends the start instruction according to the mutation value.

[0009] Further, the judging whether there is a mutation value in the formation environment data is specifically: comparing the formation environment data with a preset value; if the formation environment data does not exceed the preset value, then there is no mutation value in the formation environment data; if the formation environment data exceeds the preset value, then there is a mutation value in the formation environment data.

[0010] Further, the formation environment data includes flow value, pressure value and temperature value, and the preset value includes a preset flow value, a preset pressure value and a preset temperature value. If the preset flow value exceeds the flow value or the preset temperature value exceeds the temperature value or the preset pressure value exceeds the pressure value, then there is a mutation value in the formation environment data; if the preset flow value does not exceed the flow value and the preset temperature value does not exceed the temperature value and the preset pressure value does not exceed the pressure value, then there is no mutation value in the formation environment data.

[0011] Furthermore, the method further comprises: transmitting back to the ground controller at intervals of a preset time length Q value and error curve so that the ground controller Q The value and the error curve adjust the working mode of the downhole water distributor.

[0012] The present invention adopts the above technical solution, which has the following advantages and effects: The present invention provides a downhole control method for a wave code communication downhole water distributor based on Q-learning. The Q-learning model is used to obtain the corresponding execution action of the water nozzle switch from the formation environment data, so that the water nozzle switch of the downhole water distributor can be adjusted spontaneously, and the fine regulation of the water nozzle switch can be achieved without ground control, and the Q-learning model is continuously updated. Q The table can ensure that the regulation of the downhole water distributor continuously tends to the expected result; the present invention fully utilizes the formation environment data collected by the downhole water distributor itself to make it operate spontaneously, effectively making up for the deficiency that the traditional downhole water distributor cannot operate intelligently. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The present invention is a flow chart of a downhole control method of a downhole water distributor using wave code communication based on Q-learning. DETAILED DESCRIPTION

[0014] The embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings so that the purpose, features and advantages of the present invention can be more clearly understood. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0015] The downhole water distributor and the ground controller communicate through the wave code communication protocol, which consists of a forward communication protocol and a reverse communication protocol. The ground controller sends commands to the downhole water distributor according to the forward communication protocol and receives return codes from the downhole water distributor according to the reverse communication protocol.

[0016] The downhole water distributor includes a water distributor control unit, a downhole wave code decoder, a downhole wave code encoder, a data acquisition unit and a data storage unit, wherein the downhole wave code decoder, the downhole wave code encoder, the data acquisition unit and the data storage unit are all connected to the water distributor control unit. Q The downhole wave code decoder is used to identify the code type from the ground controller and pass the result to the water distributor control unit. The downhole wave code encoder is used to generate wave code signal to transmit information. The data acquisition unit is used to collect the formation environment data in the well, and the data storage unit is used to store the collected formation environment data.

[0017] The ground controller includes a ground control unit, a ground wave code decoder, a ground wave code encoder and a host computer, wherein the ground wave code decoder, the ground wave code encoder and the host computer are all connected to the ground control unit in communication. The ground control unit is used to receive decoded data from the ground wave code decoder, and at the same time send downhole instructions to the ground wave code encoder for encoding and receive commands from the host computer. The ground wave code encoder is used to generate wave code signals to the downhole water distributor, the ground wave code decoder is used to identify the code type from the downhole water distributor, and pass the result to the ground control unit, and the host computer is used to configure the ground control unit.

[0018] like Figure 1 The present invention provides a downhole control method for a downhole water distributor using wave code communication based on Q-learning, comprising the following steps: Step S1, the ground controller sends an operation start instruction to the downhole water distributor through the ground wave code encoder according to the forward communication protocol. This start instruction is used as an opening instruction and does not need to specify the layer address.

[0019] Step S2, the downhole wave code decoder of the downhole water distributor receives and parses the start instruction, and after the parse is successful, the downhole water distributor starts to operate and enters the dispensing mode.

[0020] Before entering the deployment mode, the water distributor control unit of the downhole water distributor first creates Q The table is then initialized Q After the table is displayed, it starts to receive the formation environment data collected by the data acquisition unit of the downhole water distributor.

[0021] Q The table is the state-action value strategy table determined by the Q-learning model. Q In the table, rows represent states and columns represent actions. Q Each action in each state in the table is assigned a Q Value, all Q The value is set to 0 by default in the initial stage. If the downhole water distributor is not the first time to go down the well, the historical Q The table will do.

[0022] Specifically, Q Tables are used to store and update Q value, Q The value update is calculated through the Q-learning model. Q The value is the cumulative value brought by the faucet switch performing a certain action in a certain state. The state is the flow control state of the downhole water distributor, that is, the opening size of the flow, and the action is the flow control execution action of the downhole water distributor, that is, the increase or decrease of the opening.

[0023] For example, Q The table is shown in Table 1. Status s Indicates the opening status of the flow control. s The opening of the flow rate is in an arithmetic progression, starting at 1%, and increasing with the flow rate opening accuracy as the tolerance. That is, when the flow rate opening accuracy is set at 1%, the state s 1 is 1% opening, state s 2 is 2% opening, state s 3 is 3% opening, state s 4 is 4% opening, state s 5 is 5% opening; when the flow opening accuracy is set to 2%, the state s 1 is 1% opening, state s 2 is 3% opening, state s 3 is 5% opening, state s 4 is 7% opening, state s 5 means 9% opening, and so on. The opening accuracy can be set freely.

[0024] action Increase or decrease with 1% accuracy, among which, the action Indicates that the opening is reduced by 3%, action Indicates that the opening is reduced by 2%, action Indicates that the opening is reduced by 1%, action Indicates that the opening is 0, action Indicates that the opening increases by 1%, the action Indicates that the opening increases by 2%, the action Indicates that the opening increases by 3%.

[0025] like, Q ( s 3, ) indicates that in the state s 3 times, execute the action The cumulative value brought.

[0026] Table 1

[0027] Further update Q Table Q The iteration formula of the value is the Bellman equation, as shown in formula (1): ; (1) (1) In the formula, s Indicates the status, Indicates action; Indicates updated Q Value, that is, the water distributor is in the current state Next action Then the Q value that is closest to the preset value; Indicates the last updated value; , is the learning rate, indicating Q The update rate of the value; R represents the return value; Indicates execution of an action The maximum value of the next state Q value; , is the discount factor, indicating participation.

[0028] Step S3: The data acquisition unit of the downhole water distributor starts working. The data acquisition unit collects formation environment data at regular intervals and records its timestamp. At the same time, the collected formation environment data is stored in the data storage unit of the downhole water distributor. When the collected formation environment data reaches 5 groups, the 5 groups of formation environment data are transmitted to the water distributor control unit.

[0029] Step S4: The water distributor control unit compares the collected formation environment data with the preset value and determines whether there is a mutation value in the formation environment data. If the formation environment data exceeds the preset value, there is a mutation value. At this time, the mutation value is transmitted back to the ground controller through the downhole wave code encoder according to the reverse communication protocol. The ground controller re-issues the operation start instruction according to the mutation value. Because the mutation value requires human participation in analysis, if the ground controller operates spontaneously, it will lead to adverse results.

[0030] If the formation environment data does not exceed the preset value, there is no mutation value. At this time, the water distributor control unit will send the collected formation environment data to the Q-learning model. The Q-learning model is integrated in the water distributor control unit. The Q-learning model is responsible for the storage and calculation of the formation environment data. Q Formation environment data includes flow rate, temperature and pressure.

[0031] Specifically, when the water distributor control unit determines a mutation value, a preset value is pre-set in the water distributor control unit, and the preset value includes a preset flow value, a preset temperature value, and a preset pressure value. When the preset flow value exceeds the collected flow value, or the preset temperature value exceeds the collected temperature value, or the preset pressure value exceeds the collected pressure value, it is determined that there is a mutation value in the formation environment data, and at this time, the flow value exceeding the preset flow value, the temperature value exceeding the preset temperature value, or the pressure value exceeding the preset pressure value is used as the mutation value. If the preset flow value does not exceed the collected flow value, the preset temperature value does not exceed the collected temperature value, and the preset pressure value does not exceed the collected pressure value, it is determined that there is no mutation value in the formation environment data.

[0032] Step S5: If there is no mutation value, the water distributor control unit The decision algorithm starts from the current Q An action is obtained in the table to control the water nozzle switch of the downhole water distributor, and the water nozzle switch executes the corresponding action to adjust the water injection volume.

[0033] Step S6: After the water nozzle switch executes the action, the downhole water distributor adjusts the data acquisition unit to re-collect the new formation environment data and transmits the new formation environment data to the water distributor control unit. The water distributor control unit updates the new formation environment data and the error curve according to the new formation environment data. Q This continuous update can ensure that the regulation of the downhole water distributor tends to the expected result. The error curve is the return value converted by the Q-learning model based on the difference between the preset value and the new formation environment data. R .

[0034] Specifically, the preset value is configured and set in the water distributor control unit before the downhole water distributor is lowered into the well, and can also be modified later by the surface controller and the downhole water distributor through wave code communication.

[0035] The difference includes flow difference and pressure difference. The flow difference is the difference between the preset flow value and the flow value of the new formation environment data, and the pressure difference is the difference between the preset pressure value and the pressure value of the new formation environment data. R The calculation formula is shown in formula (2): ; (2) From formula (2), we can see that the smaller the flow difference and pressure difference are, the higher the return value is. R The higher.

[0036] (2) In the formula, represents the flow value of the new formation environment data, Indicates the preset flow value. Indicates the pressure value of the new formation environment data, Indicates the preset pressure value, represents the flow difference threshold, Indicates the threshold value of the pressure difference, The normalized weight coefficient indicating the importance of traffic. is the normalized weight coefficient that represents the importance of stress, and ;when , When , When , it means that the ground controller only considers the flow change of the formation.

[0037] Step S7: After a period of time, the water distributor control unit of the underground water distributor sends a message to the surface controller at intervals of a preset time. Q value and error curve, the ground controller can Q When the ground controller adjusts the working mode of the downhole water distributor, it sends a start command to the downhole water distributor through the ground wave code encoder again for adjustment.

[0038] The preset time of the underground water distributor is preferably 1 hour, and the feedback is transmitted layer by layer according to the layer position. If there are 5 underground water distributors in the well at the same time, the feedback time is 5 hours, and the mutation value is transmitted for the rest of the time.

Claims

1. A downhole control method for a downhole water distributor based on wave code communication based on Q-learning, characterized in that: The method comprises: Receive a start command sent by a ground controller and parse the start command; After successful analysis, the formation environment data of the downhole water distributor is obtained; Determine whether there is a mutation value in the formation environment data. If there is no mutation value, ε-greedy The decision algorithm starts from the current Q The meter obtains an action to control the water nozzle switch of the underground water distributor; Q The table is a state-action value strategy table determined by the Q-learning model; Q Each action in each state is assigned a Q value.

2. According to the Q-learning-based wave code communication downhole water distributor downhole control method of claim 1, it is characterized in that: After the water nozzle switch of the downhole water distributor is executed, new formation environment data is obtained; Update the new formation environment data and error curve Q Table; the error curve is the return value converted from the difference between the preset value and the new formation environment data.

3. According to the Q-learning-based wave code communication downhole water distributor downhole control method of claim 2, it is characterized in that: Said update Q Table Q The iteration formula of the value is as follows: ； in, Indicates the status, Indicates action, Indicates updated Q value, Indicates the last updated value, is the learning rate, R Represents the return value, is the discount factor, Indicates execution of an action The maximum value of the next state Q value.

4. According to the Q-learning-based wave code communication downhole water distributor downhole control method of claim 3, it is characterized in that: The calculation formula of the return value is as follows: ； in, Indicates the collected flow value. Indicates the preset flow value. Indicates the collected pressure value, Indicates the preset pressure value, Represents the normalized weight coefficient of traffic, represents the normalized weight coefficient of pressure, and , represents the flow difference threshold, Indicates the threshold value of the pressure difference.

5. The downhole control method of the downhole water distributor based on wave code communication based on Q-learning according to claim 1 is characterized in that: If there is a mutation value, the mutation value is transmitted back to the ground controller according to the reverse communication protocol, so that the ground controller resends the start instruction according to the mutation value.

6. The downhole control method of the downhole water distributor based on wave code communication based on Q-learning according to claim 1 is characterized in that: The determining whether there is a mutation value in the formation environment data is specifically as follows: comparing the formation environment data with a preset value; if the formation environment data does not exceed the preset value, then there is no mutation value in the formation environment data; if the formation environment data exceeds the preset value, then there is a mutation value in the formation environment data.

7. A Q-learning-based wave code communication downhole water distributor downhole control method according to claim 6, characterized in that: The formation environment data includes a flow value, a pressure value and a temperature value, the preset value includes a preset flow value, a preset pressure value and a preset temperature value, and if the preset flow value exceeds the flow value or the preset temperature value exceeds the temperature value or the preset pressure value exceeds the pressure value, then there is a sudden change value in the formation environment data; If the preset flow value does not exceed the flow value, the preset temperature value does not exceed the temperature value, and the preset pressure value does not exceed the pressure value, then there is no sudden change value in the formation environment data.

8. The downhole control method of a downhole water distributor based on wave code communication based on Q-learning according to claim 2 is characterized in that: Also includes: Send back to the ground controller at preset time intervals Q value and error curve so that the ground controller Q The value and the error curve adjust the working mode of the downhole water distributor.

Images