Tunnel geological detection method, system and equipment based on reinforcement learning, and medium

Abstract

The invention discloses a tunnel geological detection method, system and equipment based on reinforcement learning, and a medium. The method comprises the following steps: acquiring multi-source detection data of a karst tunnel; designing a partially observable Markov decision process model according to the multi-source detection data, the hazard source information and the hazard source detection accuracy, wherein elements in the partially observable Markov decision process model comprise a state space S, an action space A and a reward function R; training the partially observable Markov decision process model by adopting a depth deterministic strategy gradient method until the partially observable Markov decision process model converges so as to obtain a detection model; and inputting data corresponding to the current state space of the karst tunnel into the detection model so as to obtain the optimal detection action of the karst tunnel. According to the method, the position of the dangerous source and the position of the next detection point are selected in a reinforcement learning mode.

Description

Technical Field

[0001] The present invention relates to the technical field of karst tunnel engineering, and in particular to a tunnel geological detection method, system, equipment and medium based on reinforcement learning. Background Art

[0002] During the construction of karst tunnels, the tunnel structure is complex and the environment is changeable. It is a huge challenge to conduct advanced geological prediction of dangerous sources such as caves and water leakage. The existing methods are poorly targeted and easily affected by personnel experience and subjectivity. There is an urgent need for a technical means to detect dangerous sources in real time and accurately. Summary of the invention

[0003] The present invention provides a tunnel geological detection method, system, equipment and medium based on reinforcement learning to solve the above problems.

[0004] The present invention is achieved through the following technical solutions: A tunnel geological detection method based on reinforcement learning, comprising: Obtain multi-source detection data of karst tunnels; Design a partially observable Markov decision process model based on the multi-source detection data, hazard source information, and hazard source detection accuracy, wherein the elements in the partially observable Markov decision process model include a state space S, an action space A, and a reward function R, and make real-time intelligent decisions on detection actions through the action space A, and set the reward function R to be the hazard source detection accuracy and the distance accuracy between the detection point and the hazard source, and the state space S is the multi-source detection data; Using a deep deterministic policy gradient method to train the partially observable Markov decision process model until the partially observable Markov decision process model converges to obtain a detection model; The data corresponding to the current state space S of the karst tunnel is input into the detection model, so as to obtain the optimal detection action of the karst tunnel.

[0005] As an optimization, the multi-source detection data includes the electromagnetic field, elastic wave velocity field and electric current field of the karst geology in front of the tunnel floor and the heading face.

[0006] As an optimization, the state space is ,in, , Respectively represent the electromagnetic field of the karst geology in front of the tunnel floor and the face, , They represent the elastic wave velocity field of the karst geology in front of the tunnel floor and the face, , They represent the current field of the karst geology in front of the tunnel floor and the heading face respectively.

[0007] As an optimization, the action space is ,in, Indicates the location of the next detection point, Indicates the key detection area of ​​the current detection point, where: , , , Respectively represent the coordinates of the next detection point relative to the current detection point on the xyz axis, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, wherein the direction perpendicular to the plane where the tunnel entrance of the karst tunnel is located is the z-axis direction, the vertical direction is the y-axis, and the horizontal direction perpendicular to the z-axis and the y-axis is the x-axis, , , , They respectively represent the coordinates of the hazard source corresponding to the current detection point on the xyz axis relative to the current detection point, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, It is the radius of the key detection area centered on the hazard source.

[0008] As an optimization, the reward function is ,in, , are the hazard source detection accuracy and the distance accuracy between the detection point and the hazard source, respectively. , are the weights of hazard source detection accuracy and distance accuracy respectively; ; , D is the preset distance.

[0009] The present invention also discloses a tunnel geological detection system based on reinforcement learning, which is used to implement the aforementioned tunnel geological detection method based on reinforcement learning, comprising: Acquisition module, used to obtain multi-source detection data of karst tunnels; A design module is used to design a partially observable Markov decision process model according to the multi-source detection data, hazard source information, and hazard source detection accuracy, wherein the elements in the partially observable Markov decision process model include a state space S, an action space A, and a reward function R, and real-time intelligent decision making is performed on the detection action through the action space, the reward function is set to the hazard source detection accuracy, and the state space is the multi-source detection data; A training module, configured to train the partially observable Markov decision process model using a deep deterministic policy gradient method until the partially observable Markov decision process model converges to obtain a detection model; The execution module is used to input the data corresponding to the current state space S of the karst tunnel into the detection model, so as to obtain the optimal detection action of the karst tunnel.

[0010] As an optimization, the multi-source detection data includes the electromagnetic field, elastic wave velocity field, and electric current field of the karst geology in front of the tunnel floor and the heading face.

[0011] As an optimization, the acquisition module includes an integrated ground penetrating radar, a tunnel seismic wave detector and a tunnel resistivity detector.

[0012] The present invention also discloses a storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, a tunnel geological detection method based on reinforcement learning as described above is implemented.

[0013] The present invention also discloses an electronic device, comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes a tunnel geological detection method based on reinforcement learning as described above.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: The multi-source information fusion and reinforcement learning-based tunnel geological detection method disclosed in the present invention collects the electromagnetic field, elastic wave velocity field, and current field of the karst geology in front of the tunnel floor and the heading face, and can directly establish the multi-source detection data, the relationship between the hazard source detection action and the hazard source detection accuracy, and the distance accuracy between the detection point and the hazard source without the need for numerical simulation, and then completes the selection of the hazard source location and the next detection point location through reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 This is a flow chart of a tunnel geological detection method based on reinforcement learning described in the present invention. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0017] Aiming at the difficulty of advanced geological prediction of dangerous sources such as caves and water leakage during the construction of karst tunnels, the present invention has developed a multi-source and multi-field comprehensive geological intelligent detection equipment for karst tunnels based on reinforcement learning. It integrates multi-source detection tools such as ground penetrating radar (GPR), tunnel seismic wave detector (TSP), tunnel resistivity detector (ERT), etc., to comprehensively detect the electromagnetic field, elastic wave velocity field, and current field information of the karst geology in front of the tunnel floor and the face. Next, the method of the present invention is specifically introduced.

[0018] This embodiment 1 provides a tunnel geological detection method based on reinforcement learning, such as Figure 1 As shown, including: S1. Obtain multi-source detection data of karst tunnels; In some embodiments, the multi-source detection data includes the electromagnetic field, elastic wave velocity field and current field of the karst geology in front of the tunnel floor and the face. That is, by integrating multi-source detection tools such as ground penetrating radar (GPR), tunnel seismic wave probe (TSP), tunnel resistivity probe (ERT), etc., the electromagnetic field, elastic wave velocity field and current field information of the karst geology in front of the tunnel floor and the face are obtained through comprehensive detection.

[0019] S2. Design a partially observable Markov decision process model based on the multi-source detection data, hazard source information, and hazard source detection accuracy. The elements in the partially observable Markov decision process model include a state space S, an action space A, and a reward function R. Real-time intelligent decisions are made on detection actions through the action space A. The reward function R is set to the hazard source detection accuracy and the distance accuracy between the detection point and the hazard source. The state space S is the multi-source detection data.

[0020] In some embodiments, the state space is ,in, , Respectively represent the electromagnetic field of the karst geology in front of the tunnel floor and the face, , They represent the elastic wave velocity field of the karst geology in front of the tunnel floor and the face, , They represent the current field of the karst geology in front of the tunnel floor and the heading face respectively.

[0021] In some embodiments, the action space is ,in, Indicates the location of the next detection point, Indicates the key detection area of ​​the current detection point, where: , , , Respectively represent the coordinates of the next detection point relative to the current detection point on the xyz axis, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, wherein the direction perpendicular to the plane where the tunnel entrance of the karst tunnel is located is the z-axis direction, the vertical direction is the y-axis, and the horizontal direction perpendicular to the z-axis and the y-axis is the x-axis, , , , They respectively represent the coordinates of the hazard source corresponding to the current detection point on the xyz axis relative to the current detection point, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, It is the radius of the key detection area centered on the hazard source.

[0022] In some embodiments, the reward function is ,in, , are the hazard source detection accuracy and the distance accuracy between the detection point and the hazard source, respectively. , are the weights of hazard source detection accuracy and distance accuracy respectively; ; Here, the key detection area is set to a is the center of the circle, The spherical shape has a radius of . As long as a part of the dangerous source is in the spherical shape, it is determined that there is a dangerous source in the key detection area.

[0023] The sentence “there are no dangerous sources in the key detection area centered on the dangerous sources” means that the former “dangerous sources” refer to the dangerous sources predicted by the detection method of the present invention, that is, the predicted dangerous sources, and the latter “dangerous sources” refer to the dangerous sources actually existing in the lava tunnel, that is, the actual dangerous sources.

[0024] , D is the preset distance.

[0025] "The shortest distance between the detection point and the hazard source" means the shortest distance between the current detection point and the actual hazard source.

[0026] Since the importance of danger source detection accuracy and distance accuracy is the same, in this embodiment, , Both are 0.5.

[0027] S3. Using a deep deterministic policy gradient method to train the partially observable Markov decision process model until the partially observable Markov decision process model converges to obtain a detection model.

[0028] Here, the accuracy of hazard source detection is mainly to detect the hazard source corresponding to the current detection point, while the accuracy of the distance between the detection point and the hazard source is mainly to ensure that the detection point is near the hazard source, which helps to collect the state space, and thus, it also helps to locate the next detection point, that is, the next detection point will be located near the hazard source.

[0029] Specifically, after obtaining the multi-source detection data, the multi-source detection data is first stored in a database, and then the partially observable Markov decision process model is trained using a deep deterministic policy gradient method, and the hazard source detection scheme is continuously optimized and improved through a feedback loop.

[0030] Based on the deep deterministic policy gradient method to train the observable Markov decision process model, the agent can optimize the reward obtained in a complex and uncertain environment by learning in interaction, and guide better actions by sensing the reaction of the state of the environment to the action to obtain the maximum benefit. Corresponding to the real-time intelligent decision-making of the detection of the hazard source and the next detection point, the reward function to be obtained is the accuracy of the hazard source detection and the accuracy of the distance between the detection point and the hazard source. The specific action space includes the electromagnetic field, elastic wave velocity field and current field of the karst geology in front of the tunnel floor and the face.

[0031] The present invention adopts a deep deterministic policy gradient method to train the partially observable Markov decision process model until the partially observable Markov decision process model.

[0032] The deep deterministic policy gradient method is an existing technology, and the specific process is as follows: 1. Initialize network parameters: First, initialize the parameters of the state network and the evaluation network. The state network is responsible for outputting actions based on the current state, while the evaluation network evaluates the value of these actions.

[0033] ‌2. Data Collection‌: Use behavioral strategies to interact with the environment and collect experience data. Behavioral strategies usually add random noise to the original strategy to ensure exploration. The collected data includes state, action, reward, and next state, which are stored in the experience replay buffer.

[0034] ‌3. Train the evaluation network‌: Randomly sample a batch of data from the experience replay buffer and use this data to train the evaluation network. The goal is to reduce the difference between the predicted Q value and the actual reward. This is usually achieved by minimizing the mean squared error.

[0035] ‌4. Update the state network‌: Update the state network based on the gradient information provided by the evaluation network. This ensures that the actions output by the network maximize the expected reward. The parameters of the state network are optimized by gradient ascent.

[0036] ‌5. Repeated training‌: Repeat the above steps until the preset number of training rounds is reached or other stopping conditions are met. In each round of training, the new strategy interacts with the environment, collects more experience data, and then uses this data to further optimize the network parameters.

[0037] S4. Inputting data corresponding to the current state space S of the karst tunnel into the detection model, thereby obtaining the optimal detection action of the karst tunnel.

[0038] The present invention proposes a deep deterministic policy gradient reinforcement learning algorithm, which observes and fits the correlation between detection data and risk sources in real time, predicts the area that needs to be focused on near the current measuring point, and gives the subsequent optimal detection strategy (such as the specific location of the next measuring point of the detection equipment) based on the stored experience.

[0039] Embodiment 2 further discloses a multi-source and multi-field comprehensive geological intelligent detection system for karst tunnels based on reinforcement learning, comprising: An acquisition module is used to obtain multi-source detection data of a karst tunnel; in some embodiments, the acquisition module includes an integrated ground penetrating radar, a tunnel seismic wave detector, and a tunnel resistivity detector.

[0040] A design module is used to design a partially observable Markov decision process model according to the multi-source detection data, the hazard source information, and the hazard source detection accuracy, wherein the elements in the partially observable Markov decision process model include a state space S, an action space A, and a reward function R, and a real-time intelligent decision is made on the detection action through the action space A, the reward function R is set to the hazard source detection accuracy, and the state space S is the multi-source detection data; A training module, configured to train the partially observable Markov decision process model using a deep deterministic policy gradient method until the partially observable Markov decision process model converges to obtain a detection model; The execution module is used to input the data corresponding to the current state space S of the karst tunnel into the detection model, so as to obtain the optimal detection action of the karst tunnel.

[0041] The system realizes the intelligent prediction of key detection areas in the advanced geological forecast in the tunnel and the automatic control of the detection equipment. Compared with the existing geological forecasting means of comprehensive tunnel detection by manual labor, this method enhances the targeted detection of dangerous sources in karst tunnels and greatly improves the efficiency of advanced geological forecasting. At the same time, it eliminates the influence of the subjectivity of technical personnel and improves the accuracy of the comprehensive geology of the tunnel.

[0042] In some embodiments, the multi-source detection data includes the electromagnetic field, elastic wave velocity field, and electric current field of the karst geology in front of the tunnel floor and the heading face.

[0043] In some embodiments, the state space is ,in, , Respectively represent the electromagnetic field of the karst geology in front of the tunnel floor and the face, , They represent the elastic wave velocity field of the karst geology in front of the tunnel floor and the face, , They represent the current field of the karst geology in front of the tunnel floor and the face respectively; The action space is ,in, Indicates the location of the next detection point, Indicates the key detection area of ​​the current detection point, where: , , , Respectively represent the coordinates of the next detection point relative to the current detection point on the xyz axis, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, wherein the direction perpendicular to the plane where the tunnel entrance of the karst tunnel is located is the z-axis direction, the vertical direction is the y-axis, and the horizontal direction perpendicular to the z-axis and the y-axis is the x-axis, , , , They respectively represent the coordinates of the hazard source corresponding to the current detection point on the xyz axis relative to the current detection point, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, It is the radius of the key detection area centered on the hazard source; The reward function is .

[0044] Example 3 also discloses a storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, a tunnel geological detection method based on reinforcement learning as in Example 1 is implemented.

[0045] Example 4 also discloses an electronic device, comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes a tunnel geological detection method based on reinforcement learning as in Example 1.

[0046] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A tunnel geological detection method based on reinforcement learning, characterized in that: include: Obtain multi-source detection data of karst tunnels; Design a partially observable Markov decision process model based on the multi-source detection data, hazard source information, and hazard source detection accuracy, wherein the elements in the partially observable Markov decision process model include a state space S, an action space A, and a reward function R, and make real-time intelligent decisions on detection actions through the action space A, and set the reward function R to be the hazard source detection accuracy and the distance accuracy between the detection point and the hazard source, and the state space S is the multi-source detection data; Using a deep deterministic policy gradient method to train the partially observable Markov decision process model until the partially observable Markov decision process model converges to obtain a detection model; The data corresponding to the current state space S of the karst tunnel is input into the detection model, so as to obtain the optimal detection action of the karst tunnel.

2. A tunnel geological detection method based on reinforcement learning according to claim 1, characterized in that: The multi-source detection data include the electromagnetic field, elastic wave velocity field, and electric current field of the karst geology in front of the tunnel floor and the heading face.

3. A tunnel geological detection method based on reinforcement learning according to claim 2, characterized in that: The state space is ,in, , Respectively represent the electromagnetic field of the karst geology in front of the tunnel floor and the face, , They represent the elastic wave velocity field of the karst geology in front of the tunnel floor and the face, , They represent the current field of the karst geology in front of the tunnel floor and the heading face respectively.

4. The tunnel geological detection method based on reinforcement learning according to claim 1 is characterized in that: The action space is ,in, Indicates the location of the next detection point, Indicates the key detection area of ​​the current detection point, where: , , , Respectively represent the coordinates of the next detection point relative to the current detection point on the xyz axis, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, wherein the direction perpendicular to the plane where the tunnel entrance of the karst tunnel is located is the z-axis direction, the vertical direction is the y-axis, and the horizontal direction perpendicular to the z-axis and the y-axis is the x-axis, , , , They respectively represent the coordinates of the hazard source corresponding to the current detection point on the xyz axis relative to the current detection point, that is , , is the xyz coordinate in the coordinate system with the current detection point as the origin, It is the radius of the key detection area centered on the hazard source.

5. The tunnel geological detection method based on reinforcement learning according to claim 1, characterized in that: The reward function is ,in, , are the hazard source detection accuracy and the distance accuracy between the detection point and the hazard source, respectively. , are the weights of hazard source detection accuracy and distance accuracy respectively; ； , D is the preset distance.

6. A tunnel geological detection system based on reinforcement learning, used to implement a tunnel geological detection method based on reinforcement learning as described in any one of claims 1 to 5, characterized in that: include: Acquisition module, used to obtain multi-source detection data of karst tunnels; A design module is used to design a partially observable Markov decision process model according to the multi-source detection data, the hazard source information, and the hazard source detection accuracy, wherein the elements in the partially observable Markov decision process model include a state space S, an action space A, and a reward function R, and a real-time intelligent decision is made on the detection action through the action space A, the reward function R is set to the hazard source detection accuracy, and the state space S is the multi-source detection data; A training module, configured to train the partially observable Markov decision process model using a deep deterministic policy gradient method until the partially observable Markov decision process model converges to obtain a detection model; The execution module is used to input the data corresponding to the current state space S of the karst tunnel into the detection model, so as to obtain the optimal detection action of the karst tunnel.

7. A tunnel geological detection system based on reinforcement learning according to claim 6, characterized in that: The multi-source detection data include the electromagnetic field, elastic wave velocity field, and electric current field of the karst geology in front of the tunnel floor and the heading face.

8. The tunnel geological detection system based on reinforcement learning according to claim 6, characterized in that: The acquisition module includes an integrated ground penetrating radar, a tunnel seismic wave detector and a tunnel resistivity detector.

9. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes a tunnel geological detection method based on reinforcement learning as described in any one of claims 1-5.

10. A storage medium, characterized in that: The storage medium is used to store computer instructions. When the computer instructions are executed by the processor, a tunnel geological detection method based on reinforcement learning as described in any one of claims 1 to 5 is implemented.

Images