Heading machine remote control method based on three-dimensional reconstruction and virtual reality

Abstract

The invention provides a heading machine remote control method based on three-dimensional reconstruction and virtual reality, and relates to the technical field of equipment control. Acquiring image information acquired by a plurality of cameras mounted on the target heading machine at the current moment and a three-dimensional heading machine model corresponding to the target heading machine; acquiring a three-dimensional roadway scene model of a roadway where the target heading machine is located based on the image information acquired by the plurality of cameras, and determining position information and attitude information of the target heading machine in the roadway based on the image information acquired by the plurality of cameras; based on the position information and the attitude information, fusing and rendering the three-dimensional heading machine model and the three-dimensional roadway scene model to obtain a remote control virtual scene corresponding to the target heading machine at the current moment; and performing remote control on the target heading machine based on the virtual scene. Therefore, remote control of the heading machine based on three-dimensional reconstruction and virtual reality is realized, and the accuracy, stability and efficiency of remote control are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of equipment control, and in particular to a remote control method of a tunnel boring machine based on three-dimensional reconstruction and virtual reality. Background Art

[0002] At present, the visual feedback of remote control of tunnel boring machines mainly relies on the videos returned by multiple cameras with different perspectives installed on the tunneling working face. The images of different perspectives are displayed on different monitors, and the operator needs to switch back and forth between different screens to grasp the on-site environment. However, this method lacks an intuitive sense of space and position, resulting in low accuracy and stability of remote control of tunnel boring machines, as well as large workload and low efficiency. Summary of the invention

[0003] The present invention provides a remote control method for a tunnel boring machine based on three-dimensional reconstruction and virtual reality.

[0004] According to one aspect of the present disclosure, a method for remotely controlling a roadheader based on three-dimensional reconstruction and virtual reality is provided, comprising: Obtaining image information collected by multiple cameras installed on a target tunneling machine at a current moment, and a three-dimensional tunneling machine model corresponding to the target tunneling machine; Based on the image information collected by the multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained; Determine the position information and posture information of the target tunnel boring machine in the tunnel based on the image information collected by the multiple cameras; Based on the position information and the posture information, the three-dimensional roadheader model is merged and rendered with the three-dimensional tunnel scene model to obtain a remotely controlled virtual scene corresponding to the target roadheader at the current moment; Based on the virtual scene, the target tunnel boring machine is remotely controlled.

[0005] The present disclosure provides a method for remote control of a tunnel boring machine based on three-dimensional reconstruction and virtual reality. First, image information collected by multiple cameras installed on a target tunnel boring machine at the current moment and a three-dimensional tunnel boring machine model corresponding to the target tunnel boring machine are obtained, and then a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained based on the image information collected by multiple cameras, and the position information and posture information of the target tunnel boring machine in the tunnel are determined based on the image information collected by multiple cameras, and then the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model are fused and rendered based on the position information and posture information to obtain a virtual scene of remote control corresponding to the target tunnel boring machine at the current moment, and finally the target tunnel boring machine is remotely controlled based on the virtual scene. Therefore, the three-dimensional tunnel scene model of the tunnel where the tunnel boring machine is located and the position of the tunnel boring machine in the tunnel are determined based on the image information collected by the camera on the tunnel boring machine, and based on the position and posture, the three-dimensional model corresponding to the tunnel boring machine and the three-dimensional tunnel scene model are fused and rendered to obtain a virtual scene for remote control. Based on the virtual scene, the tunnel boring machine is remotely controlled, thereby realizing remote control of the tunnel boring machine based on three-dimensional reconstruction and virtual reality, improving the accuracy and stability of remote control of the tunnel boring machine, and improving the efficiency of remote control of the tunnel boring machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0007] Figure 1 A schematic flow chart of a method for remotely controlling a roadheader based on three-dimensional reconstruction and virtual reality provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of the installation position of a camera on a tunnel boring machine in a remote control method of a tunnel boring machine based on three-dimensional reconstruction and virtual reality proposed in an embodiment of the present disclosure; Figure 3 A schematic flow chart of a method for remotely controlling a roadheader based on three-dimensional reconstruction and virtual reality provided in an embodiment of the present disclosure; Figure 4 A schematic flow chart of a method for remotely controlling a roadheader based on three-dimensional reconstruction and virtual reality provided in an embodiment of the present disclosure; Figure 5 A schematic flow chart of a method for remotely controlling a roadheader based on three-dimensional reconstruction and virtual reality provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of the effect of the remote control method of a tunnel boring machine based on three-dimensional reconstruction and virtual reality provided in an embodiment of the present disclosure.

[0008] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0009] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0010] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0011] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0012] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0013] The following describes in detail the remote control method of a tunnel boring machine based on three-dimensional reconstruction and virtual reality according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0014] Figure 1 A flowchart of a method for remotely controlling a tunnel boring machine based on three-dimensional reconstruction and virtual reality is provided in an embodiment of the present disclosure.

[0015] like Figure 1 As shown, the method includes: Step 101, obtaining image information collected by multiple cameras installed on a target tunneling machine at a current moment, and a three-dimensional tunneling machine model corresponding to the target tunneling machine.

[0016] The target tunneling machine may be a tunneling machine to be remotely controlled, and its specific type and structure may be determined according to actual needs and geological conditions. For example, the target tunneling machine may be a coal mine tunneling machine, a tunneling machine specially designed for underground coal mining, or a reinforced support tunneling machine, a tunneling machine designed for complex geological conditions, with advanced automatic control and reinforced support systems, etc., which is not limited in the present disclosure.

[0017] The specific type and quantity of the cameras installed on the target tunnel boring machine may be preset or determined according to actual needs. For example, the type of the camera may be a monocular PTZ camera, etc., which is not limited in the present disclosure.

[0018] It should be noted that the specific installation positions and directions of the multiple cameras installed on the target tunnel boring machine can be determined as needed. Figure 2 As shown, Figure 2 This is a schematic diagram of the installation position of the camera on the tunnel boring machine in the tunnel boring machine remote control method based on three-dimensional reconstruction and virtual reality proposed in an embodiment of the present disclosure, Figure 2 In the embodiment, cameras can be installed in front of the target tunnel boring machine and on both sides of the machine body, and the direction of each camera can be as follows: Figure 2 As shown by the arrow in the figure, Figure 2 The specific type and specific structure of the target tunnel boring machine shown in are only examples and can be determined according to actual needs. Figure 2 This is only an example and the present disclosure is not limited thereto.

[0019] It should be noted that, in addition to installing multiple cameras on the target tunnel boring machine, a light source and a sensor may also be installed at the location where the cameras are installed, wherein the specific types of the light source and the sensor may be determined according to actual needs. For example, the light source may be a white light source, and the sensor may be a laser ranging sensor, or an infrared sensor, etc., which is not limited in the present disclosure.

[0020] Among them, the image information can be the tunnel scene information collected by the camera, which may include the scene information in front of the target tunnel boring machine body, and the scene information from both sides of the body to both sides of the tunnel, which is not limited in the present disclosure.

[0021] The three-dimensional tunnel boring machine model may be pre-built according to the design parameters of the target tunnel boring machine, and may be any type of three-dimensional model, for example, it may be a CAD three-dimensional tunnel boring machine model, which is not limited in the present disclosure.

[0022] Among them, CAD is the abbreviation of Computer Aided Design (CAD).

[0023] In the present disclosure, when a target tunnel boring machine is remotely controlled based on three-dimensional reconstruction and virtual reality, the image information collected by multiple cameras installed on the target tunnel boring machine at the current moment and the three-dimensional tunnel boring machine model corresponding to the target tunnel boring machine can be first obtained, thereby providing a data basis for three-dimensional reconstruction and construction of a virtual scene for remote control.

[0024] Step 102: Based on the image information collected by multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained.

[0025] Among them, the tunnel where the target tunnel boring machine is located can be a tunneling working face tunnel, etc., and the present disclosure does not limit this.

[0026] Among them, the three-dimensional tunnel scene model can be a three-dimensional tunnel scene model including the tunnel scene in front of the tunnel boring machine body and the tunnel scenes on both sides of the body.

[0027] In the present disclosure, after obtaining image information collected by multiple cameras installed on the target tunnel boring machine, in order to realize remote control of the tunnel boring machine based on a virtual scene and improve the accuracy and stability of remote control, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located can be first obtained based on the image information.

[0028] Step 103: Determine the position information and posture information of the target tunnel boring machine in the tunnel based on the image information collected by multiple cameras.

[0029] The position information may be the relative position information of the target tunnel boring machine in the tunnel.

[0030] The attitude information may include the heading angle, pitch angle, roll angle and other attitude information of the tunnel boring machine, which is not limited in the present disclosure.

[0031] In the present disclosure, after obtaining the three-dimensional tunnel scene information of the tunnel where the target tunnel boring machine is located, in order to accurately remotely control the target tunnel boring machine, the position information and posture information of the target tunnel boring machine in the tunnel can be determined based on the image information collected by multiple cameras.

[0032] Step 104 , based on the position information and the posture information, the three-dimensional roadheader model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target roadheader at the current moment.

[0033] In the present disclosure, after determining the position information and posture information of the target tunnel boring machine in the tunnel, the three-dimensional tunnel boring machine model can be firstly integrated with the three-dimensional tunnel scene model based on the position information, and then the posture of the three-dimensional tunnel boring machine model can be adjusted based on the posture information, so as to accurately and reliably obtain the virtual scene of remote control corresponding to the target tunnel boring machine at the current moment, thereby providing conditions for remote control of the tunnel boring machine based on three-dimensional reconstruction and virtual reality.

[0034] Step 105: remotely control the target tunnel boring machine based on the virtual scene.

[0035] In the present disclosure, after obtaining the virtual scene of remote control corresponding to the target tunnel boring machine at the current moment, the virtual scene can be transmitted to a virtual reality device (such as VR glasses, VR helmets, etc.) in the form of a video stream, and then the remote operator can realize immersive remote control of the target tunnel boring machine through the virtual reality device, thereby effectively reducing the workload of the remote operator and improving the accuracy and efficiency of remote control.

[0036] Among them, VR is the abbreviation of Virtual Reality (VR).

[0037] In the disclosed embodiment, first, image information collected by multiple cameras installed on the target tunnel boring machine at the current moment and a three-dimensional tunnel boring machine model corresponding to the target tunnel boring machine are obtained, and then based on the image information collected by the multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained, and based on the image information collected by the multiple cameras, the position information and posture information of the target tunnel boring machine in the tunnel are determined, and then based on the position information and posture information, the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target tunnel boring machine at the current moment, and finally, based on the virtual scene, the target tunnel boring machine is remotely controlled. Therefore, the three-dimensional tunnel scene model of the tunnel where the tunnel boring machine is located and the position of the tunnel boring machine in the tunnel are determined based on the image information collected by the camera on the tunnel boring machine, and based on the position and posture, the three-dimensional model corresponding to the tunnel boring machine and the three-dimensional tunnel scene model are fused and rendered to obtain a virtual scene for remote control. Based on the virtual scene, the tunnel boring machine is remotely controlled, thereby realizing remote control of the tunnel boring machine based on three-dimensional reconstruction and virtual reality, improving the accuracy and stability of remote control of the tunnel boring machine, and improving the efficiency of remote control of the tunnel boring machine.

[0038] Figure 3 A flowchart of a method for remotely controlling a tunnel boring machine based on three-dimensional reconstruction and virtual reality is provided in an embodiment of the present disclosure.

[0039] like Figure 3 As shown, the method includes: Step 301, obtaining image information collected by multiple cameras installed on a target tunneling machine at the current moment, and a three-dimensional tunneling machine model corresponding to the target tunneling machine.

[0040] The specific implementation form of step 301 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0041] Step 302: Taking any camera as the coordinate origin, determine the coordinates of each camera.

[0042] In the present disclosure, after obtaining the image information collected by multiple cameras installed on the target tunnel boring machine at the current moment, and the three-dimensional tunnel boring machine model corresponding to the target tunnel boring machine, in order to accurately and reliably obtain the three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located, any camera can be first used as the coordinate origin, a unified coordinate system can be established for these multiple cameras, and then the coordinates of each camera can be determined based on the coordinate system.

[0043] It should be noted that when any camera is used as the coordinate origin, a camera can be determined as the coordinate origin according to actual needs or pre-set. For example, the camera in front of the fuselage on the target tunneling machine can be used as the coordinate origin, etc., and the present disclosure does not limit this.

[0044] Step 303: determine the three-dimensional tunnel scene segment corresponding to each image information.

[0045] In the present disclosure, after acquiring the image information collected by multiple cameras on the target tunnel boring machine, before determining the three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located, the image information collected by each camera can be processed by a predetermined algorithm to determine the three-dimensional tunnel scene fragment corresponding to each image information, wherein the predetermined algorithm can be any type of algorithm. For example, an end-to-end three-dimensional reconstruction algorithm based on image memory can be constructed based on the image information collected by the camera to generate the corresponding three-dimensional tunnel scene fragment, etc., and the present disclosure does not limit this.

[0046] Step 304 , based on the coordinates of each camera, multiple three-dimensional tunnel scene segments are spliced ​​to obtain a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located.

[0047] In the present disclosure, after determining the three-dimensional tunnel scene fragments corresponding to each image information, these multiple three-dimensional tunnel scene fragments can be spliced ​​based on the coordinates of each camera to obtain a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located, thereby improving the efficiency of obtaining the three-dimensional tunnel scene model.

[0048] Step 305: Determine the position information and posture information of the target tunnel boring machine in the tunnel based on the image information collected by multiple cameras.

[0049] Step 306 , based on the position information and the posture information, the three-dimensional roadheader model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target roadheader at the current moment.

[0050] Step 307: remotely control the target tunnel boring machine based on the virtual scene.

[0051] The specific implementation forms of steps 305 to 307 may refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0052] In the disclosed embodiment, first, image information collected by multiple cameras installed on the target tunnel boring machine at the current moment and a three-dimensional tunnel boring machine model corresponding to the target tunnel boring machine are obtained, and then any camera is used as the coordinate origin to determine the coordinates of each camera and the three-dimensional tunnel scene fragment corresponding to each image information. Thereafter, based on the coordinates of each camera, multiple three-dimensional tunnel scene fragments are spliced ​​to obtain a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located, and based on the image information collected by multiple cameras, the position information and posture information of the target tunnel boring machine in the tunnel are determined. Finally, based on the position information and posture information, the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model are fused and rendered to obtain a virtual scene of remote control corresponding to the target tunnel boring machine at the current moment, and based on the virtual scene, the target tunnel boring machine is remotely controlled. Therefore, by fixing a camera as the coordinate origin, the coordinates of each camera are determined, the three-dimensional tunnel scene segments of the image information collected by each camera are determined, and based on the coordinates, multiple three-dimensional tunnel scene segments are spliced ​​to obtain a three-dimensional tunnel scene model of the tunnel where the tunnel boring machine is located. Based on the position of the tunnel boring machine in the tunnel, the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model are fused to obtain a virtual scene, and based on the virtual scene, the tunnel boring machine is remotely controlled, thereby improving the accuracy and stability of remote control of the tunnel boring machine.

[0053] Figure 4 A flowchart of a method for remotely controlling a tunnel boring machine based on three-dimensional reconstruction and virtual reality is provided in an embodiment of the present disclosure.

[0054] like Figure 4 As shown, the method includes: Step 401, obtaining image information collected by multiple cameras installed on a target tunneling machine at the current moment, and a three-dimensional tunneling machine model corresponding to the target tunneling machine.

[0055] Step 402: Based on the image information collected by multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained.

[0056] The specific implementation forms of step 401 to step 402 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0057] Step 403, obtaining distance data collected by multiple sensors installed on the target tunneling machine at the current moment, inertial navigation data of the target tunneling machine at the current moment, and the installation position of each camera on the target tunneling machine.

[0058] It should be noted that the sensor installed on the target tunnel boring machine may be a laser ranging sensor, which is not limited in the present disclosure.

[0059] The distance data may be the distance data between the target tunnel boring machine and the object, for example, the distance data between the two sides of the fuselage of the target tunnel boring machine and the two sides of the tunnel, etc., which is not limited in the present disclosure.

[0060] The inertial navigation data may be data collected by an inertial navigation system (INS) on the target tunnel boring machine. For example, the inertial navigation data may include attitude data such as the pitch angle, yaw angle, and roll angle of the target tunnel boring machine, and may also include position data, speed data, and heading data of the target tunnel boring machine, etc., which are not limited in the present disclosure.

[0061] In the present disclosure, after obtaining the three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located, the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model can be fused and rendered to obtain a remotely controlled virtual scene. In order to improve the accuracy and reliability of model fusion, before performing model fusion, the distance data collected by multiple sensors installed on the target tunnel boring machine at the current moment, the inertial navigation data of the target tunnel boring machine at the current moment, and the installation position of each camera on the target tunnel boring machine can be first obtained.

[0062] Step 404: determine common view feature points between image information collected by multiple cameras.

[0063] The common view feature points may be the same feature points among multiple image information.

[0064] It should be noted that when determining the common view feature points between image information captured by multiple cameras, when each camera has captured image information at multiple different angles at the current moment, not only can the common view feature points between the image information captured by multiple cameras be determined, but also based on the multiple image information captured by each camera, the common view feature points between the image information captured by each camera can be determined. The present disclosure does not limit this.

[0065] In the present disclosure, by determining the common view feature points between the image information collected by multiple cameras, conditions are provided for determining the position information and posture information of the target tunnel boring machine in the tunnel.

[0066] Step 405 , based on the distance data, the inertial navigation data, the installation position of each camera and the common view feature points, determine the position information and posture information of the target tunnel boring machine in the tunnel.

[0067] In the present disclosure, after determining the common view feature points between the image information collected by multiple cameras, the position information and posture information of the target tunnel boring machine in the tunnel can be determined based on the distance data, inertial navigation data, the installation position of each camera and the common view feature points, thereby providing conditions for realizing remote control of the tunnel boring machine based on three-dimensional reconstruction and virtual reality.

[0068] Step 406 , based on the position information and the posture information, the three-dimensional roadheader model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target roadheader at the current moment.

[0069] Step 407: remotely control the target tunnel boring machine based on the virtual scene.

[0070] The specific implementation forms of step 406 to step 407 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0071] In the disclosed embodiment, first, image information collected by multiple cameras installed on the target tunneling machine at the current moment and a three-dimensional tunneling machine model corresponding to the target tunneling machine are obtained, and based on the image information collected by the multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunneling machine is located is obtained, and then distance data collected by multiple sensors installed on the target tunneling machine at the current moment, inertial navigation data of the target tunneling machine at the current moment, and the installation position of each camera on the target tunneling machine are obtained, and the common view feature points between the image information collected by the multiple cameras are determined, and then based on the distance data, inertial navigation data, the installation position of each camera and the common view feature points, the position information and posture information of the target tunneling machine in the tunnel are determined, and finally, based on the position information and posture information, the three-dimensional tunneling machine model and the three-dimensional tunnel scene model are fused and rendered to obtain a virtual scene of remote control corresponding to the target tunneling machine at the current moment, and the target tunneling machine is remotely controlled based on the virtual scene. Therefore, the three-dimensional tunnel scene model of the tunnel where the tunnel boring machine is located is determined based on the image information collected by the camera installed on the tunnel boring machine, and the position of the tunnel boring machine in the tunnel is determined based on the distance data, inertial navigation data collected by the sensor on the tunnel boring machine at the current moment, and the common view feature points between the image information collected by multiple cameras. Based on the position and posture, the three-dimensional tunnel boring machine model is integrated with the three-dimensional tunnel scene model to obtain a virtual scene for remote control, and the tunnel boring machine is remotely controlled based on the virtual scene, thereby effectively reducing the workload of the operator and improving the accuracy and efficiency of remote control of the tunnel boring machine.

[0072] Figure 5 A flowchart of a method for remotely controlling a tunnel boring machine based on three-dimensional reconstruction and virtual reality is provided in an embodiment of the present disclosure.

[0073] like Figure 5 As shown, the method includes: Step 501, obtaining image information collected by multiple cameras installed on a target tunnel boring machine at the current moment, and a three-dimensional tunnel boring machine model corresponding to the target tunnel boring machine.

[0074] Step 502: Based on the image information collected by multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained.

[0075] Step 503, obtaining distance data collected by multiple sensors installed on the target tunneling machine at the current moment, inertial navigation data of the target tunneling machine at the current moment, and the installation position of each camera on the target tunneling machine.

[0076] Step 504: determine common view feature points between image information collected by multiple cameras.

[0077] The specific implementation forms of steps 501 to 504 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0078] Step 505, based on the distance data, determine the distance information between the target tunnel boring machine and the two sides of the tunnel.

[0079] In the present disclosure, after obtaining the distance data collected by multiple sensors on the target tunnel boring machine, the distance information between the target tunnel boring machine and the two sides of the tunnel can be determined based on the distance data, so that the relationship between the position of the target tunnel boring machine and the center line of the tunnel can be determined, and then it can be judged whether the excavation direction of the target tunnel boring machine is offset to ensure the working quality of the target tunnel boring machine.

[0080] Step 506: Determine the position of the corresponding camera in the lane based on the common view feature points.

[0081] In the present disclosure, after determining the common view feature points between the image information collected by multiple cameras, in order to accurately determine the position of the target tunnel boring machine in the tunnel, the position of the camera in the tunnel corresponding to the image information to which the common view feature points belong can be determined based on the common view feature points, thereby providing conditions for determining the position of the target tunnel boring machine in the tunnel.

[0082] Step 507, based on the position of the corresponding camera in the tunnel, the installation position of the corresponding camera and the distance information, determine the position information of the target tunnel boring machine in the tunnel.

[0083] In the present disclosure, after determining the position of the corresponding camera in the tunnel, the position information of the target tunnel boring machine in the tunnel can be accurately and reliably determined based on the position of the corresponding camera in the tunnel, its installation position on the target tunnel boring machine, and distance information, thereby providing conditions for improving the accuracy of remote control of the tunnel boring machine.

[0084] Step 508: Determine the attitude information of the target tunnel boring machine based on the inertial navigation data.

[0085] In the present disclosure, after obtaining the inertial navigation data of the target tunnel boring machine at the current moment, the posture information of the target tunnel boring machine in the tunnel can be determined based on the pitch angle, yaw angle, roll angle and other data in the inertial navigation data, thereby ensuring that the determined posture information of the target tunnel boring machine is accurate and reliable.

[0086] Step 509 , based on the position information and the posture information, the three-dimensional roadheader model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target roadheader at the current moment.

[0087] Step 510: remotely control the target tunnel boring machine based on the virtual scene.

[0088] The specific implementation forms of step 509 to step 510 may refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0089] In the disclosed embodiment, firstly, image information collected by multiple cameras installed on the target tunnel boring machine at the current moment and a three-dimensional tunnel boring machine model corresponding to the target tunnel boring machine are obtained, and based on the image information collected by the multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained, and then distance data collected by multiple sensors installed on the target tunnel boring machine at the current moment, inertial navigation data of the target tunnel boring machine at the current moment, and the installation position of each camera on the target tunnel boring machine are obtained, and the common view feature points between the image information collected by the multiple cameras are determined, and based on the distance data, the distance information between the target tunnel boring machine and the two sides of the tunnel is determined, based on the common view feature points, the position of the corresponding camera in the tunnel is determined, based on the position of the corresponding camera in the tunnel, the installation position of the corresponding camera and the distance information, the position information of the target tunnel boring machine in the tunnel is determined, and based on the inertial navigation data, the attitude information of the target tunnel boring machine is determined, and then based on the position information and the attitude information, the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target tunnel boring machine at the current moment, and finally, based on the virtual scene, the target tunnel boring machine is remotely controlled. Therefore, after determining the three-dimensional tunnel scene model of the tunnel where the tunnel boring machine is located based on the image information collected by the camera on the tunnel boring machine, the distance information between the tunnel boring machine and the two sides of the tunnel is determined based on the distance data collected by the sensor on the tunnel boring machine, and the position of the camera in the tunnel corresponding to the image information where the common view feature point is located is determined based on the common view feature point. Based on the position, installation position and distance information of the camera in the tunnel, the position information of the tunnel boring machine in the tunnel is determined, and based on the inertial navigation data, the posture information of the tunnel boring machine is determined. Based on the posture of the tunnel boring machine, the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model are fused and rendered to obtain a virtual scene of remotely controlled tunnel boring machine, and based on the virtual scene, the tunnel boring machine is remotely controlled, thereby improving the accuracy and efficiency of remote control of the tunnel boring machine.

[0090] Figure 6 A flowchart of a method for remotely controlling a tunnel boring machine based on three-dimensional reconstruction and virtual reality is provided in an embodiment of the present disclosure.

[0091] like Figure 6 As shown, the method includes: Step 601, obtaining image information collected by multiple cameras installed on a target tunneling machine at the current moment, and a three-dimensional tunneling machine model corresponding to the target tunneling machine.

[0092] Step 602: Based on the image information collected by multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained.

[0093] Step 603: Determine the position information and posture information of the target tunnel boring machine in the tunnel based on the image information collected by multiple cameras.

[0094] Step 604 , based on the position information and the posture information, the three-dimensional roadheader model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target roadheader at the current moment.

[0095] The specific implementation forms of steps 601 to 604 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0096] Step 605, determining the tunnel excavation conditions based on the virtual scene.

[0097] Among them, the excavation conditions may include the excavation situation of the tunnel and information such as the position, excavation direction and excavation posture of the target tunnel boring machine, which is not limited in the present disclosure.

[0098] In the present disclosure, after obtaining the virtual scene of remote control corresponding to the target tunnel boring machine at the current moment, the virtual scene can be transmitted to the virtual reality device so that the operator can determine the tunneling condition of the tunnel through the virtual scene, thereby providing conditions for the operator to remotely control the tunnel boring machine.

[0099] In some possible implementation forms, when determining the excavation condition of a tunnel based on a virtual scene, the environmental information of the tunnel at the current moment and the positional relationship between the target tunnel boring machine and the tunnel can be first obtained based on the virtual scene, and then the excavation condition of the tunnel can be accurately and reliably determined based on the environmental information and the positional relationship.

[0100] The environmental information of the tunnel may include information such as the cross-sectional shape and geological conditions of the tunnel, which is not limited in the present disclosure.

[0101] It should be noted that the positional relationship between the target tunnel boring machine and the tunnel can be the relative positional relationship between the target tunnel boring machine and the tunnel centerline, or can be considered as the distance relationship between the target tunnel boring machine and the two sides of the tunnel, and the present disclosure does not limit this.

[0102] Step 606, based on the excavation working conditions, remotely control at least one of the following items of the target tunnel boring machine: position, excavation direction, and excavation posture.

[0103] In the present disclosure, after determining the excavation conditions of the tunnel, at least one of the following items of the target tunnel boring machine can be remotely controlled based on the excavation conditions: position, excavation direction and excavation posture. For example, if the excavation direction of the target tunnel boring machine deviates from the direction along the center line of the tunnel, the excavation direction of the target tunnel boring machine can be adjusted by remote control, or if the position of the target tunnel boring machine is biased relative to the center line of the tunnel, that is, when the position of the target tunnel boring machine is close to any side of the tunnel, the position of the target tunnel boring machine can be adjusted by remote control, or if the pitch angle, yaw angle or roll angle of the target tunnel boring machine deviates, the posture of the target tunnel boring machine can be adjusted by remote control, thereby effectively reducing the workload of the operator and improving the accuracy and efficiency of the remote control of the tunnel boring machine.

[0104] In the disclosed embodiment, first, image information collected by multiple cameras installed on the target tunneling machine at the current moment and a three-dimensional tunneling machine model corresponding to the target tunneling machine are obtained, and then based on the image information collected by the multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunneling machine is located is obtained, and based on the image information collected by the multiple cameras, the position information and posture information of the target tunneling machine in the tunnel are determined, and then based on the position information and posture information, the three-dimensional tunneling machine model and the three-dimensional tunnel scene model are fused and rendered to obtain a remotely controlled virtual scene corresponding to the target tunneling machine at the current moment, and based on the virtual scene, the tunneling condition of the tunnel is determined, and finally, based on the tunneling condition, at least one of the following items of the target tunneling machine is remotely controlled: position, tunneling direction, and tunneling posture. Therefore, after the three-dimensional tunnel boring machine model and the three-dimensional tunnel scene model are integrated based on the position of the tunnel boring machine in the tunnel to obtain a virtual scene for remote control, the tunneling condition of the tunnel is determined based on the virtual scene, and based on the tunneling condition, at least one of the position, tunneling direction and tunneling posture of the tunnel boring machine is remotely controlled, thereby effectively reducing the workload of remote control of the tunnel boring machine and improving the accuracy and efficiency of remote control of the tunnel boring machine.

[0105] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0106] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0107] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0108] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0109] It should be understood that the various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0110] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0111] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0112] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0113] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0114] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A remote control method for a roadheader based on three-dimensional reconstruction and virtual reality, characterized in that: include: Obtaining image information collected by multiple cameras installed on a target tunneling machine at a current moment, and a three-dimensional tunneling machine model corresponding to the target tunneling machine; Based on the image information collected by the multiple cameras, a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located is obtained; Determine the position information and posture information of the target tunnel boring machine in the tunnel based on the image information collected by the multiple cameras; Based on the position information and the posture information, the three-dimensional roadheader model is merged and rendered with the three-dimensional tunnel scene model to obtain a remotely controlled virtual scene corresponding to the target roadheader at the current moment; Based on the virtual scene, the target tunnel boring machine is remotely controlled.

2. The method according to claim 1, characterized in that The step of obtaining a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located based on the image information collected by the plurality of cameras comprises: Taking any camera as the coordinate origin, determining the coordinates of each of the cameras; Determine a three-dimensional lane scene segment corresponding to each of the image information; Based on the coordinates of each of the cameras, multiple three-dimensional tunnel scene fragments are spliced ​​to obtain a three-dimensional tunnel scene model of the tunnel where the target tunnel boring machine is located.

3. The method according to claim 1, characterized in that The determining, based on the image information collected by the plurality of cameras, the position information and the posture information of the target tunnel boring machine in the tunnel comprises: Acquire distance data collected by multiple sensors installed on the target tunneling machine at the current moment, inertial navigation data of the target tunneling machine at the current moment, and the installation position of each camera on the target tunneling machine; Determining common view feature points between the image information collected by the plurality of cameras; Based on the distance data, the inertial navigation data, the installation position of each camera and the common view feature point, the position information and posture information of the target tunnel boring machine in the tunnel are determined.

4. The method according to claim 3, characterized in that The determining the position information and posture information of the target tunnel boring machine in the tunnel based on the distance data, the inertial navigation data, the installation position of each camera and the common view feature point includes: Based on the distance data, determining distance information between the target tunnel boring machine and two sides of the tunnel; Based on the common view feature points, determining the position of the corresponding camera in the lane; Determine the position information of the target tunneling machine in the tunnel based on the position of the corresponding camera in the tunnel, the installation position of the corresponding camera and the distance information; Based on the inertial navigation data, the posture information of the target tunnel boring machine is determined.

5. The method according to claim 1, characterized in that The remotely controlling the target tunnel boring machine based on the virtual scene includes: Based on the virtual scene, determining the excavation condition of the tunnel; Based on the excavation working condition, at least one of the following items of the target tunnel boring machine is remotely controlled: position, excavation direction, and excavation posture.

6. The method according to claim 5, characterized in that The determining of the tunneling condition of the tunnel based on the virtual scene includes: Based on the virtual scene, obtaining environmental information of the tunnel at the current moment and a positional relationship between the target tunnel boring machine and the tunnel; Based on the environmental information and the positional relationship, the excavation condition of the tunnel is determined.

Images