Working face recognition and control method and device of roadheader based on binocular vision

Abstract

The present invention discloses a working face recognition and control method and device of a roadheader based on binocular vision. The binocular vision technology is used to determine the status information corresponding to the cutting face area and the stockpile area respectively. Then, according to the status information corresponding to the cutting face area and the stockpile area respectively, the target task to be performed by the roadheader and the target position corresponding to the target task are determined. Then, under binocular vision tracking, the roadheader is guided from an initial position to a target position corresponding to the cutting face area or the stockpile area, thereby realizing the recognition and positioning of the target task corresponding to the cutting face area or the stockpile area located in the tunnel by the binocular vision technology. Then, by guiding the roadheader to move to the target position, the roadheader performs the target task. After the target task is performed, the environmental map information of the roadheader controls the roadheader to move from the target position to the initial position along a moving trajectory.

Description

Technical Field

[0001] This invention relates to the field of tunneling machine technology, and in particular to a method and device for identifying and controlling the working face of a tunneling machine based on binocular vision. Background Art

[0002] Currently, intelligent tunneling machines (TBMs) often encounter problems at their working faces, such as uneven cutting surfaces, poor contour shaping, and severe material accumulation on both sides. These issues affect the TBM's automatic cutting process, requiring the TBM to move to the target area for processing. For example, during automatic cutting by the intelligent TBM, some fallen material may not be completely cleaned up by the bottom sweeping device. As cutting continues, material accumulates on both sides of the working face, hindering the movement of the cutting head and the TBM's advance. This necessitates manual intervention to collect the material, thus reducing the TBM's automation rate and work efficiency. Summary of the Invention

[0003] This invention provides a method and device for identifying and controlling the working face of a tunneling machine based on binocular vision. It can realize the identification and positioning of the cutting face area and the material storage area in the roadway using binocular vision technology, and then guide the tunneling machine to move to the cutting face area and / or the material storage area to complete the cutting of the cutting face area and / or the material storage area.

[0004] Firstly, a method for identifying and controlling the working face of a tunneling machine based on binocular vision is provided, including:

[0005] Based on the binocular vision images in the tunnel, the status information corresponding to the cutting surface area and the stockpiling area is obtained respectively;

[0006] Based on the status information corresponding to the cutting surface area and the material stacking area, the target task to be executed by the tunneling machine and the target position corresponding to the target task are determined.

[0007] Based on binocular vision tracking of the tunneling machine's movement, the tunneling machine is guided to move from its initial position to the target position, and environmental map information and movement trajectory of the tunneling machine during this movement are plotted.

[0008] If the tunneling machine moves to the target position, then control the tunneling machine to perform the target task;

[0009] If the tunneling machine completes the target task, then the tunneling machine is controlled to move from the target position to the initial position along the movement trajectory according to the environmental map information.

[0010] Furthermore, the state information includes edge forming degree information, flatness information, and first position information of the cutting surface area, as well as height information and second position information of the stacking area. The step of obtaining the state information corresponding to the cutting surface area and the stacking area based on the binocular visual image within the tunnel includes:

[0011] Based on the binocular visual images in the tunnel, obtain the point cloud information of the working face;

[0012] Based on the point cloud information of the working face, the cutting surface area and the material stacking area are separated;

[0013] Based on the point cloud data of the cut surface region, determine the edge forming degree information, flatness information and first position information of the cut surface region;

[0014] Based on the point cloud data of the stockpiling area, determine the height information and second location information of the stockpiling area;

[0015] The step of determining the target task to be performed by the tunneling machine and the target location corresponding to the target task based on the state information corresponding to the cutting surface area and the stockpiling area respectively includes:

[0016] Based on the edge forming degree information and flatness information of the cutting surface area, as well as the height information of the material stacking area, the target task to be performed by the tunneling machine is determined; the target task includes at least one of the cutting task of the cutting surface area and the material stacking task of the material stacking area.

[0017] The target location is determined based on the first location information and / or the second location information corresponding to the target task.

[0018] Furthermore, the step of acquiring point cloud information of the working face based on the binocular visual images within the tunnel includes:

[0019] Based on the binocular visual images within the tunnel, a dense parallax map of the working face is obtained;

[0020] The spatial coordinates of each point cloud data in the dense parallax map are calculated to obtain the point cloud information of the working surface in the dense parallax map;

[0021] The step of separating the cutting surface area and the stockpiling area based on the point cloud information of the working surface includes:

[0022] Based on the point cloud information of the working surface in the dense parallax map, calculate the spatial plane corresponding to the point cloud information of the cutting surface region, and determine the point cloud data corresponding to the cutting surface region;

[0023] Point cloud data whose distance from the spatial plane corresponding to the cut surface region is less than a threshold are used as the initial point set;

[0024] The point cloud data within the initial point set is projected onto the spatial plane corresponding to the cut surface region to obtain a depth map;

[0025] Based on the collected graph label data on the depth map, a semantic segmentation model is trained;

[0026] Based on the semantic segmentation model, the cutting surface region and the material stacking region in the depth map are separated.

[0027] Furthermore, determining the edge shaping information, flatness information, and first position information of the cut surface region based on the point cloud data of the cut surface region includes:

[0028] Project the point cloud data corresponding to the cut surface region onto the actual physical coordinate system to obtain the actual spatial point cloud data of the cut surface region;

[0029] Based on the actual spatial point cloud data of the cut surface region, the first location information corresponding to the cut surface region is obtained;

[0030] The step of determining the height information and second location information of the stockpiling area based on the point cloud data of the stockpiling area includes:

[0031] Project the point cloud data corresponding to the stockpiling area onto the actual physical coordinate system to obtain the actual spatial point cloud data of the stockpiling area;

[0032] Based on the actual spatial point cloud data of the stockpiling area, the second location information corresponding to the stockpiling area is obtained.

[0033] Furthermore, the step of guiding the tunneling machine from its initial position to the target position based on binocular vision-based movement tracking includes:

[0034] Real-time point cloud data of the cutting surface area is obtained by using binocular cameras fixed on both sides of the tunneling machine;

[0035] Based on the real-time point cloud data of the material stockpiling area, the relative position between the tunneling machine and the cutting face area is calculated to achieve movement tracking of the tunneling machine.

[0036] Furthermore, the step of guiding the tunneling machine from its initial position to the target position based on binocular vision-based motion tracking, and drawing environmental map information and movement trajectory of the tunneling machine during this movement, includes:

[0037] Obtain the body area and initial turning angle of the tunneling machine; wherein, the body area is the area covered by the body of the tunneling machine;

[0038] Based on the physical area of ​​the tunneling machine, a virtual area that moves with the tunneling machine is defined;

[0039] Based on the movement tracking of the virtual area by the binocular vision, the tunneling machine is guided to move straight along the initial turning angle direction, and when the virtual area contacts the side wall, the tunneling machine is guided to rotate and move until the tunneling machine moves to the target position;

[0040] Based on the straight-line trajectory guiding the tunneling machine to move straight along the initial turning angle and the rotation trajectory of the tunneling machine, the movement trajectory of the tunneling machine from the initial position to the target position is plotted in real time.

[0041] Based on the binocular vision images of the tunneling machine during its movement obtained from the binocular vision, environmental map information is drawn in real time as the tunneling machine moves from its initial position to the target position.

[0042] Furthermore, guiding the tunneling machine to move straight along the initial turning direction, and when the virtual area contacts the sidewall, guiding the tunneling machine to rotate and move based on the movement tracking of the virtual area by the binocular vision, until the tunneling machine moves to the target position where the stockpile area is located, includes:

[0043] The number of trajectory inflection points of the tunneling machine moving from the initial position to the target position is preset;

[0044] If the virtual area comes into contact with the sidewall, the rotation angle that guides the tunneling machine to rotate once is determined based on the initial turning angle and the number of trajectory inflection points.

[0045] Secondly, a working face recognition and control device for a tunneling machine based on binocular vision is provided, including:

[0046] The status information acquisition module is used to obtain the status information corresponding to the cutting surface area and the stockpiling area respectively based on the binocular vision image in the tunnel;

[0047] The target task determination module is used to determine the target task to be executed by the tunneling machine and the target position corresponding to the target task based on the status information corresponding to the cutting surface area and the material stacking area respectively.

[0048] The navigation module is used to guide the tunneling machine from its initial position to the target position based on binocular vision tracking of the machine's movement, and to draw environmental map information and movement trajectory of the tunneling machine during this movement.

[0049] The first control module is used to control the tunneling machine to perform the target task if the tunneling machine moves to the target position;

[0050] The second control module is used to control the tunneling machine to move from the target position to the initial position along the movement trajectory according to the environmental map information if the tunneling machine completes the target task.

[0051] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described method for identifying and controlling the working face of a tunneling machine based on binocular vision.

[0052] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method for identifying and controlling the working face of a tunneling machine based on binocular vision.

[0053] In the aforementioned scheme of working face identification and control method, device, computer equipment, and storage medium for a tunneling machine based on binocular vision, the state information corresponding to the cutting face area and the stockpiling area is determined by binocular vision technology. Then, based on the state information corresponding to the cutting face area and the stockpiling area, the target task to be performed by the tunneling machine and the target position corresponding to the target task are determined. Then, under binocular vision tracking, the tunneling machine is guided from the initial position to the target position corresponding to the cutting face area or the stockpiling area. This enables the binocular vision technology to identify and locate the target task corresponding to the cutting face area or the stockpiling area in the roadway. Then, by guiding the tunneling machine to move to the target position, the tunneling machine can perform the target task. After the target task is completed, the tunneling machine environment map information controls the tunneling machine to move from the target position to the initial position along the movement trajectory. Attached Figure Description

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 This is a flowchart illustrating a method for identifying and controlling the working face of a tunneling machine based on binocular vision, according to one embodiment of the present invention.

[0056] Figure 2 yes Figure 1A schematic diagram of a specific implementation of step S101;

[0057] Figure 3 yes Figure 2 A schematic diagram of a specific implementation of step S1012;

[0058] Figure 4 yes Figure 1 A schematic diagram of a specific implementation method for step S103;

[0059] Figure 5 This is a schematic diagram showing the tunneling machine moving from its initial position to its target position in the stockpiling area;

[0060] Figure 6 This is a schematic diagram of a working face identification and control device for a tunneling machine based on binocular vision, according to one embodiment of the present invention.

[0061] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0062] Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0064] This invention provides a method for identifying and controlling the working face of a tunneling machine based on binocular vision. Please refer to [link to relevant documentation]. Figure 1 As shown, this includes the following steps:

[0065] S101. Based on the binocular visual images in the tunnel, obtain the status information corresponding to the cutting surface area and the stockpiling area respectively.

[0066] Currently, existing tunneling machines rely heavily on external calibration boards or lidar for positioning and navigation, which is costly and requires precise calibration. Furthermore, current tunneling machines lack effective technology for identifying the cutting face and material storage areas underground. Therefore, this application utilizes binocular vision technology to identify the tunnel and obtain the corresponding status information for the cutting face and material storage areas. This enables the identification and positioning of these areas within the tunnel. Simultaneously, by using binocular vision technology for inertial navigation fusion positioning, the positioning and navigation costs of the tunneling machine can be reduced.

[0067] Here, the status information includes edge forming degree information, flatness information, and first position information of the cutting surface area, as well as height information and second position information of the stacking area, see [link to relevant documentation]. Figure 2 The step of obtaining the state information corresponding to the cutting surface area and the stockpiling area based on the binocular visual images in the tunnel includes:

[0068] S1011. Obtain point cloud information of the working face based on the binocular visual image in the tunnel.

[0069] Here, binocular cameras equipped with multi-point infrared uniform illumination are fixed on both sides of the tunneling machine. The binocular cameras are responsible for collecting binocular images of the working face in the tunnel at the left and right front of the tunneling machine.

[0070] In some implementations, obtaining point cloud information of the working face based on the binocular visual images in the tunnel may include: obtaining a dense disparity map of the working face based on the binocular visual images in the tunnel; calculating the spatial coordinates of each point cloud data in the dense disparity map to obtain the point cloud information of the working face in the dense disparity map.

[0071] It should be noted that the SGM algorithm is used to calculate the binocular image to obtain the dense disparity map corresponding to the working surface. Then, the spatial coordinates of each point cloud data in the dense disparity map are calculated to obtain the point cloud information of the working surface in the dense disparity map.

[0072] Here, the point cloud information in the dense parallax map includes the spatial coordinates of the point cloud data in the dense parallax map. The spatial coordinates of the point cloud in the dense parallax map are calculated according to the formula... Calculate the spatial coordinates {x} of point cloud data in a dense disparity map. k ,y k ,z k}k=1,2,…,K, where l is the baseline distance, disp is the disparity value, and f x and f y x is the pixel focal length of the point cloud data. p and y p These are the pixel coordinates of the point cloud data.

[0073] S1012. Based on the point cloud information of the working surface, the cutting surface area and the material stacking area are separated.

[0074] Here, the cutting surface area and the stockpiling area are separated based on the point cloud information of the working face, so that the state information of the separated cutting surface area and stockpiling area can be evaluated separately, thereby obtaining accurate state information corresponding to the cutting surface area and the stockpiling area respectively.

[0075] In some implementations, see Figure 3 The step of separating the cutting surface area and the stockpiling area based on the point cloud information of the working surface includes:

[0076] S1012-1. Based on the point cloud information of the working surface in the dense parallax map, calculate the spatial plane corresponding to the point cloud information of the cutting surface region, and determine the point cloud data corresponding to the cutting surface region.

[0077] When calculating the spatial plane corresponding to the point cloud information of the cut surface region, the calculation is performed by combining RANSAC and the least squares method to fit the spatial plane equation corresponding to the point cloud information on the cut surface region. The specific calculation method is as follows:

[0078] Assume the spatial plane equation corresponding to the point cloud information on the cut surface region is xr x +yr y +zr z =1, the point cloud set of the cut surface region is {x k ,y k ,z k If k = 1, 2, ..., K, then:

[0079] make via r = (V T V) -1 V T 1. Solve for the spatial plane corresponding to the point cloud information on the cut surface region, where r represents the spatial plane corresponding to the point cloud information on the cut surface region.

[0080] S1012-2, Use the point cloud data that are less than a threshold distance from the spatial plane corresponding to the cut surface region as the initial point set.

[0081] Since there is a certain distance between the cutting surface area and the stockpiling area, by calculating the spatial plane corresponding to the point cloud information on the cutting surface area and comparing the obtained spatial plane with the threshold, the obtained initial point set can have the point cloud data corresponding to the stockpiling area within the threshold range.

[0082] In this embodiment, the point cloud data of the spatial plane r corresponding to the point cloud information on the cut surface region within the threshold range (0-1m) is selected as the initial point set P0.

[0083] S1012-3. Project the point cloud data in the initial point set onto the spatial plane corresponding to the cutting surface region to obtain a depth map.

[0084] Here, the point cloud data within the initial point set P0 are projected onto the spatial plane r corresponding to the point cloud information on the cut surface region to obtain the depth map.

[0085] S1012-4. Based on the collected graph label data on the depth map, train a semantic segmentation model.

[0086] It should be noted that the semantic segmentation model in this application is the U-net semantic segmentation model.

[0087] S1012-5. Based on the semantic segmentation model, separate the cutting surface region and the material stacking region in the depth map.

[0088] Here, when separating the cutting surface region and the stockpiling region in the depth map, the cutting surface region and the stockpiling region in the depth map are identified by a semantic segmentation model. Then, the point cloud data in the cutting surface region and the stockpiling region are reprojected onto the physical coordinate system corresponding to the cutting surface region and the stockpiling region, respectively, so as to obtain the actual spatial point cloud data corresponding to the cutting surface region and the stockpiling region, thus completing the separation of the cutting surface region and the stockpiling region.

[0089] S1013. Based on the point cloud data of the cut surface region, determine the edge forming degree information, flatness information and first position information of the cut surface region.

[0090] In some embodiments, determining the edge forming degree information, flatness information, and first position information of the cut surface region based on the point cloud data of the cut surface region includes: projecting the point cloud data corresponding to the cut surface region onto the actual physical coordinate system to obtain the actual spatial point cloud data of the cut surface region; and obtaining the first position information corresponding to the cut surface region based on the actual spatial point cloud data of the cut surface region.

[0091] Here, if the edge forming degree information has a degree value greater than the threshold, it is determined that the edge of this cutting surface area needs to be cut; if the flatness information has a degree value greater than the threshold, it is determined that the cutting surface of this cutting surface area needs to be cut.

[0092] S1014. Determine the height information and second location information of the stacking area based on the point cloud data of the stacking area.

[0093] It should be noted that, to prevent the tunneling machine from collecting material from stockpiled areas with low heights, the process of determining the height and second location information of the stockpiled area based on the point cloud data of the stockpiled area further includes: comparing the height information of the stockpiled area with a preset height threshold, and designating stockpiled areas with heights not lower than the preset height threshold as stockpiled areas to be collected. By setting height as an evaluation index for stockpiled areas that need to be collected, if the detected height of a stockpiled area exceeds the preset height threshold, it is considered that the stockpiled area is out of limit, and subsequent collection steps are executed; otherwise, the collection work is skipped.

[0094] The step of determining the height information and second location information of the stockpiling area based on the point cloud data of the stockpiling area includes: projecting the point cloud data corresponding to the stockpiling area onto the actual physical coordinate system to obtain the actual spatial point cloud data of the stockpiling area; and obtaining the second location information corresponding to the stockpiling area based on the actual spatial point cloud data of the stockpiling area.

[0095] S102. Based on the status information corresponding to the cutting surface area and the material stacking area, determine the target task to be executed by the tunneling machine and the target position corresponding to the target task.

[0096] It should be noted that the target task includes at least one of the cutting task of the cutting face area and the material receiving task of the stockpile area, and the target location includes the location of the cutting face area corresponding to the cutting task performed by the tunneling machine and the location of the stockpile area corresponding to the material receiving task performed by the tunneling machine. For example, when the target task is the material receiving task of the stockpile area in the roadway, the target location is the location of the stockpile area in the roadway.

[0097] In some embodiments, determining the target task to be performed by the tunneling machine and the target position corresponding to the target task based on the state information corresponding to the cutting surface area and the stockpiling area respectively may include: determining the target task to be performed by the tunneling machine based on the edge forming degree information and flatness information of the cutting surface area and the height information of the stockpiling area; the target task includes at least one of a cutting task of the cutting surface area and a material collection task of the stockpiling area; and determining the target position based on the first position information and / or the second position information corresponding to the target task.

[0098] Here, after extracting the point cloud data of the cutting surface region, the edge contour of the cutting surface region is fitted using the point cloud data of the cutting surface region. Then, the edge shaping degree of the cutting surface region is evaluated based on the deviation between the fitting result and the target result, thereby obtaining the edge shaping degree information of the cutting surface region. The mean and variance of the distance from the point cloud data of the cutting surface region to the spatial plane r corresponding to the point cloud information on the cutting surface region are calculated. The flatness of the cutting surface region is comprehensively evaluated by the mean and variance, thereby obtaining the flatness degree information of the cutting surface region.

[0099] S103. Based on the binocular vision tracking of the tunneling machine's movement, guide the tunneling machine to move from the initial position to the target position, and draw the environmental map information and movement trajectory of the tunneling machine during this movement process.

[0100] It should be noted that when guiding the tunneling machine from its initial position to the target position in the stockpiling area, the hydraulic motors on the tunneling machine used to drive its movement use binocular vision to detect information from the cutting surface area and the stockpiling area in the tunnel to move the tunneling machine from its initial position to the target position in the stockpiling area.

[0101] Here, the movement of the tunneling machine is tracked in real time using binocular vision, which can determine the position of the tunneling machine during the movement process. Based on the tracked position of the tunneling machine, the movement of the tunneling machine from the initial position to the target position of the stockpiling area is guided. That is, when the tracked position of the tunneling machine deviates from the target position, the movement of the tunneling machine is guided in real time to ensure that the tunneling machine moves to the target position where the stockpiling area is located.

[0102] In some implementations, the step of guiding the tunneling machine to move from an initial position to the target position based on binocular vision tracking includes: obtaining real-time point cloud data of the cutting face area using binocular cameras fixed on both sides of the tunneling machine; and calculating the relative position between the tunneling machine and the cutting face area based on the real-time point cloud data of the stockpiling area, thereby achieving motion tracking of the tunneling machine.

[0103] By using binocular cameras fixed on both sides of the tunneling machine, real-time point cloud data of the cutting surface area of ​​the tunneling machine during its movement can be obtained. This not only enables the real-time acquisition of the relative position between the tunneling machine and the cutting surface area, thus achieving movement tracking of the tunneling machine, but also facilitates the creation of real-time environmental map information of the tunneling machine as it moves from its initial position to its target position through the real-time acquired binocular visual images.

[0104] To ensure the tunneling machine moves accurately to the target location in the stockpile area, see [link / reference]. Figure 4The step of tracking the movement of the tunneling machine based on binocular vision, guiding the tunneling machine to move from its initial position to the target position, and drawing environmental map information and movement trajectory information of the tunneling machine during this movement process may include:

[0105] S1031. Obtain the body area and initial turning angle of the tunneling machine; wherein, the body area is the area covered by the body of the tunneling machine.

[0106] See Figure 5 The material stockpile area is located on the left side of the diagram. Area S0 is the main body area of ​​the tunneling machine, and θ0 is the initial turning angle of the tunneling machine. Here, the initial turning angle is the initial turning angle that the tunneling machine moves when it starts moving from its initial position to the target position in the material stockpile area.

[0107] S1032. Based on the body area of ​​the tunneling machine, set a virtual area that moves with the tunneling machine.

[0108] Figure 5 The S1 region is a virtual region set according to the main body region of the tunneling machine. Here, the virtual region can move with the tunneling machine. The width of the virtual region is greater than the width of the main body region, and the width of the virtual region can be selected to allow the tunneling machine to reach different endpoint positions.

[0109] S1033. Based on the movement tracking of the virtual area by the binocular vision, guide the tunneling machine to move straight along the initial turning angle direction, and when the virtual area contacts the side wall, guide the tunneling machine to rotate and move until the tunneling machine moves to the target position.

[0110] This can be understood as follows: first, guide the tunneling machine to rotate and move to the initial turning angle θ0, then guide the tunneling machine to move straight along the direction of the initial turning angle. The binocular camera installed on the tunneling machine detects in real time whether the S1 area contacts the side wall of the side wall (i.e. whether the point cloud enters the S1 area). When the S1 area contacts the side wall of the side wall, guide the tunneling machine to rotate until the deflection angle of the tunneling machine is 0. At this time, the tunneling machine moves straight to the target position where the material stacking area on the side of the working face is located.

[0111] Furthermore, by using binocular vision to track the movement of the virtual area in real time, the position of the tunneling machine during the movement can be determined, and based on the tracked position of the tunneling machine, the movement of the tunneling machine from the initial position to the target position in the stockpiling area can be guided in real time.

[0112] Preferably, guiding the tunneling machine to move straight along the initial turning angle, and when the virtual area contacts the sidewall, guiding the tunneling machine to rotate and move according to the movement tracking of the virtual area by the binocular vision until the tunneling machine moves to the target position where the material stacking area is located, includes: presetting the number of trajectory inflection points of the movement trajectory of the tunneling machine from the initial position to the target position; if the virtual area contacts the sidewall, determining the rotation angle for guiding the tunneling machine to rotate and move once according to the initial turning angle and the number of trajectory inflection points.

[0113] Here, the preset number of trajectory inflection points is n, and when the virtual area contacts the sidewall, the rotation angle that guides the tunneling machine to rotate and move is θ0 / n.

[0114] S1034. Based on the straight-line trajectory guiding the tunneling machine to move straight along the initial turning angle and the rotation trajectory of the tunneling machine, the movement trajectory of the tunneling machine from the initial position to the target position is drawn in real time.

[0115] S1035. Based on the binocular vision images of the tunneling machine during its movement obtained by the binocular vision, draw environmental map information in real time as the tunneling machine moves from its initial position to the target position.

[0116] Here, during the process of guiding the tunneling machine to move, ORB feature points are collected by a binocular camera, a feature point map is drawn in real time, and the map is optimized by BA.

[0117] S104. If the tunneling machine moves to the target position, control the tunneling machine to perform the target task.

[0118] It should be noted that the tunneling machine is equipped with a hydraulic motor for driving the overall movement of the machine, a cutting mechanism for cutting the cutting face area, and a material collection mechanism for collecting the material in the stockpile area. When the tunneling machine moves to the target position under the drive of the hydraulic motor, the material collection mechanism is controlled to collect the material in the stockpile area.

[0119] Here, when the target task is to collect material from the stockpiling area within the roadway, and the target location is the position of the stockpiling area within the roadway, after the tunneling machine moves to the target location, the collecting mechanism collects the material from the stockpiling area until the height of the stockpile is less than its corresponding threshold, at which point the collecting mechanism stops collecting the material. When the target task is to cut the cutting surface area, and the target location is the position of the cutting surface area, after the tunneling machine moves to the target location, the cutting mechanism cuts the cutting surface area. Here, when the edge of the cutting surface area needs to be cut, the cutting mechanism cuts the edge of the cutting surface area until the edge forming degree value of the cutting surface area is less than its corresponding threshold, at which point the cutting mechanism stops cutting the edge. When the cutting surface of the cutting surface area needs to be cut, the cutting mechanism cuts the cutting surface of the cutting surface area until the flatness value of the cutting surface of the cutting surface area is less than its corresponding threshold, at which point the cutting mechanism stops cutting the cutting surface.

[0120] S105. If the tunneling machine completes the target task, then according to the environmental map information, control the tunneling machine to move from the target position to the initial position along the movement trajectory.

[0121] After the tunneling machine finishes collecting the material in the stockpile area, it is guided to move in the opposite direction according to the navigation trajectory, so that the tunneling machine can move from the target position to the initial position. Here, the camera located on the tunneling machine also collects environmental feature points of the tunneling machine in real time during the movement process, and matches the obtained environmental features with the feature point map with the movement trajectory, so as to obtain the real-time position of the tunneling machine during the movement towards the initial position.

[0122] This invention provides a method for identifying and controlling the working face of a tunneling machine based on binocular vision. Compared with the prior art, this invention uses binocular vision technology to determine the state information corresponding to the cutting face area and the stockpiling area, respectively. Then, based on the state information corresponding to the cutting face area and the stockpiling area, it determines the target task to be performed by the tunneling machine and the target position corresponding to the target task. Then, under binocular vision tracking, the tunneling machine is guided from the initial position to the target position corresponding to the cutting face area or the stockpiling area. This enables binocular vision technology to identify and locate the target task corresponding to the cutting face area or the stockpiling area in the roadway. By guiding the tunneling machine to the target position, the tunneling machine can perform the target task. After the target task is completed, the environmental map information of the tunneling machine controls the tunneling machine to move from the target position to the initial position along the movement trajectory.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0124] In one embodiment, a working face identification and control device for a tunneling machine based on binocular vision is provided, such as... Figure 6 As shown, the material receiving system includes a binocular vision positioning module, a navigation module, and a control module. Detailed descriptions of each functional module are as follows:

[0125] The status information acquisition module 61 is used to obtain the status information corresponding to the cutting surface area and the stockpiling area respectively based on the binocular vision image in the tunnel;

[0126] The target task determination module 62 is used to determine the target task to be executed by the tunneling machine and the target position corresponding to the target task based on the status information corresponding to the cutting surface area and the stockpiling area respectively.

[0127] The navigation module 63 is used to guide the tunneling machine from its initial position to the target position based on the movement tracking of the tunneling machine using binocular vision, and to draw the environmental map information and movement trajectory of the tunneling machine during this movement process;

[0128] The first control module 64 is used to control the tunneling machine to perform the target task if the tunneling machine moves to the target position;

[0129] The second control module 65 is used to control the tunneling machine to move from the target position to the initial position along the movement trajectory according to the environmental map information if the tunneling machine completes the target task.

[0130] Furthermore, the status information includes edge forming degree information, flatness information, and first position information of the cutting surface area, as well as height information and second position information of the stacking area. The status information acquisition module 61 includes:

[0131] The point cloud data acquisition unit is used to acquire point cloud information of the working face based on the binocular vision image in the tunnel.

[0132] The separation unit is used to separate the cutting surface area and the material stacking area based on the point cloud information of the working surface;

[0133] The first information determining unit is used to determine the edge forming degree information, flatness information and first position information of the cutting surface area based on the point cloud data of the cutting surface area.

[0134] The second information determination unit is used to determine the height information and second location information of the stockpiling area based on the point cloud data of the stockpiling area.

[0135] The target task determination module 62 includes:

[0136] The target task determination unit is used to determine the target task to be performed by the tunneling machine based on the edge forming degree information and flatness information of the cutting surface area and the height information of the stockpiling area; the target task includes at least one of the cutting task of the cutting surface area and the material collection task of the stockpiling area;

[0137] The target location determination unit is used to determine the target location based on the first location information and / or the second location information corresponding to the target task.

[0138] Furthermore, the point cloud data acquisition unit includes:

[0139] The dense disparity map acquisition subunit is used to obtain the dense disparity map of the working face based on the binocular visual images in the roadway;

[0140] The first point cloud data acquisition subunit is used to calculate the spatial coordinates of each point cloud data in the dense parallax map and acquire the point cloud information of the working surface in the dense parallax map.

[0141] The separation unit includes:

[0142] The second point cloud data acquisition subunit is used to calculate the spatial plane corresponding to the point cloud information of the cutting surface region based on the point cloud information of the working surface in the dense parallax map, and to determine the point cloud data corresponding to the cutting surface region.

[0143] An initial point set acquisition sub-unit is used to take point cloud data with a distance of less than a threshold from the spatial plane corresponding to the cutting surface region as the initial point set.

[0144] The depth map acquisition subunit is used to project the point cloud data in the initial point set onto the spatial plane corresponding to the cut surface region to obtain a depth map;

[0145] The semantic segmentation model training subunit is used to train the semantic segmentation model based on the collected graph label data on the depth map;

[0146] The separation subunit is used to separate the cutting surface region and the material stacking region in the depth map based on the semantic segmentation model.

[0147] Furthermore, the first information determining unit includes:

[0148] The first actual spatial point cloud data acquisition unit is used to project the point cloud data corresponding to the cut surface region onto the actual physical coordinate system to obtain the actual spatial point cloud data of the cut surface region.

[0149] The first location information acquisition unit is used to obtain the first location information corresponding to the cutting surface region based on the actual spatial point cloud data of the cutting surface region.

[0150] The second information determining unit includes:

[0151] The second actual spatial point cloud data acquisition unit is used to project the point cloud data corresponding to the stockpiling area onto the actual physical coordinate system to obtain the actual spatial point cloud data of the stockpiling area.

[0152] The second location information acquisition unit is used to obtain the second location information corresponding to the stockpiling area based on the actual spatial point cloud data of the stockpiling area.

[0153] Furthermore, the navigation module 63 includes:

[0154] The real-time point cloud data acquisition unit is used to obtain real-time point cloud data of the cutting surface area through binocular cameras fixed on both sides of the tunneling machine;

[0155] The motion tracking unit is used to calculate the relative position between the tunneling machine and the cutting face area based on the real-time point cloud data of the material stacking area, so as to realize the motion tracking of the tunneling machine.

[0156] Furthermore, the navigation module 63 includes:

[0157] A tunneling machine information acquisition unit is used to acquire the body area and initial turning angle of the tunneling machine; wherein, the body area is the area covered by the body of the tunneling machine;

[0158] A virtual area setting unit is used to set a virtual area that moves with the tunneling machine based on the body area of ​​the tunneling machine;

[0159] The navigation unit is used to guide the tunneling machine to move straight along the initial turning angle direction based on the movement tracking of the virtual area by the binocular vision, and to guide the tunneling machine to rotate and move when the virtual area contacts the side wall, until the tunneling machine moves to the target position;

[0160] The movement trajectory drawing unit is used to draw the movement trajectory of the tunneling machine from the initial position to the target position in real time, based on the straight-line trajectory guiding the tunneling machine to move straight along the initial turning angle direction and the rotation trajectory of the tunneling machine.

[0161] The environmental map information drawing unit is used to draw environmental map information in real time as the tunneling machine moves from the initial position to the target position based on the binocular vision image obtained by the binocular vision during the movement of the tunneling machine.

[0162] Furthermore, the navigation unit includes:

[0163] The trajectory inflection point number preset subunit is used to preset the number of trajectory inflection points of the movement trajectory of the tunneling machine from the initial position to the target position;

[0164] The rotation angle determination unit is used to determine the rotation angle that guides the tunneling machine to rotate once, based on the initial rotation angle and the number of trajectory inflection points, if the virtual area comes into contact with the sidewall.

[0165] This invention provides a working face identification and control device for a tunneling machine based on binocular vision. Compared with the prior art, this invention uses binocular vision technology to determine the state information corresponding to the cutting face area and the stockpiling area, respectively. Then, based on the state information corresponding to the cutting face area and the stockpiling area, it determines the target task to be performed by the tunneling machine and the target position corresponding to the target task. Then, under binocular vision tracking, the tunneling machine is guided from the initial position to the target position corresponding to the cutting face area or the stockpiling area. This enables binocular vision technology to identify and locate the target task corresponding to the cutting face area or the stockpiling area in the tunnel. By guiding the tunneling machine to the target position, the tunneling machine can perform the target task. After the target task is completed, the tunneling machine's environmental map information controls the tunneling machine to move from the target position to the initial position along the movement trajectory.

[0166] Specific limitations regarding the working face recognition and control device for binocular vision-based tunneling machines can be found in the limitations of the working face recognition and control method for binocular vision-based tunneling machines described above, and will not be repeated here. Each module in the aforementioned working face recognition and control device for binocular vision-based tunneling machines can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0167] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a binocular vision-based tunneling machine face recognition and control method on the server side.

[0168] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a binocular vision-based tunneling machine face recognition and control method.

[0169] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0170] Based on the binocular vision images in the tunnel, the status information corresponding to the cutting surface area and the stockpiling area is obtained respectively;

[0171] Based on the status information corresponding to the cutting surface area and the material stacking area, the target task to be executed by the tunneling machine and the target position corresponding to the target task are determined.

[0172] Based on binocular vision tracking of the tunneling machine's movement, the tunneling machine is guided to move from its initial position to the target position, and environmental map information and movement trajectory of the tunneling machine during this movement are plotted.

[0173] If the tunneling machine moves to the target position, then control the tunneling machine to perform the target task;

[0174] If the tunneling machine completes the target task, then the tunneling machine is controlled to move from the target position to the initial position along the movement trajectory according to the environmental map information.

[0175] In one embodiment, another computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0176] Based on the binocular vision images in the tunnel, the status information corresponding to the cutting surface area and the stockpiling area is obtained respectively;

[0177] Based on the status information corresponding to the cutting surface area and the material stacking area, the target task to be executed by the tunneling machine and the target position corresponding to the target task are determined.

[0178] Based on binocular vision tracking of the tunneling machine's movement, the tunneling machine is guided to move from its initial position to the target position, and environmental map information and movement trajectory of the tunneling machine during this movement are plotted.

[0179] If the tunneling machine moves to the target position, then control the tunneling machine to perform the target task;

[0180] If the tunneling machine completes the target task, then the tunneling machine is controlled to move from the target position to the initial position along the movement trajectory according to the environmental map information.

[0181] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0183] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0184] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A working face recognition and control method of a roadheader based on binocular vision, characterized in that: include: According to the binocular vision image in the tunnel, the state information corresponding to the cutting surface area and the stockpile area is obtained; Determining a target task to be performed by the roadheader and a target position corresponding to the target task according to the state information corresponding to the cutting surface area and the stockpiling area respectively; Tracking the movement of the tunnel boring machine using binocular vision, guiding the tunnel boring machine to move from an initial position to a target position, and drawing environmental map information and a movement trajectory of the tunnel boring machine during the movement process; If the tunnel boring machine moves to the target position, controlling the tunnel boring machine to perform the target task; If the tunnel boring machine completes the target task, the tunnel boring machine is controlled to move from the target position to the initial position along the moving trajectory according to the environmental map information.

2. The control method according to claim 1, characterized in that: The state information includes edge forming degree information, flatness degree information and first position information of the cutting surface area, and height information and second position information of the stockpile area. The state information corresponding to the cutting surface area and the stockpile area is obtained according to the binocular vision image in the lane, including: Acquiring point cloud information of the working surface according to the binocular visual image in the tunnel; Separating the cutting surface area and the stockpiling area according to the point cloud information of the working surface; Determining edge forming degree information, flatness degree information and first position information of the cutting surface area according to the point cloud data of the cutting surface area; Determining height information and second position information of the material stacking area according to the point cloud data of the material stacking area; The determining, according to the state information corresponding to the cutting surface area and the stockpiling area, the target task to be performed by the roadheader and the target position corresponding to the target task, comprises: Determine the target task to be performed by the roadheader according to the edge forming degree information and flatness degree information of the cutting surface area and the height information of the stockpile area; the target task includes at least one of the cutting task of the cutting surface area and the material collecting task of the stockpile area; The target location is determined according to the first location information and / or the second location information corresponding to the target task.

3. The control method according to claim 2, characterized in that: The step of obtaining point cloud information of the working surface according to the binocular visual image in the tunnel includes: Obtaining a dense disparity map of the working surface according to the binocular vision image in the tunnel; Calculating the spatial coordinates of each point cloud data in the dense disparity map to obtain point cloud information of the working surface in the dense disparity map; The step of separating the cutting surface area and the stockpiling area according to the point cloud information of the working surface comprises: According to the point cloud information of the working surface in the dense disparity map, calculating the spatial plane corresponding to the point cloud information of the cutting surface area, and determining the point cloud data corresponding to the cutting surface area; Taking the point cloud data whose distance from the spatial plane corresponding to the cutting surface area is less than a threshold as the initial point set; Projecting the point cloud data in the initial point set onto the spatial plane corresponding to the cutting surface area to obtain a depth map; Training a semantic segmentation model based on the collected image label data on the depth map; Based on the semantic segmentation model, the cutting surface area and the stockpiling area in the depth map are separated.

4. The control method according to claim 3, characterized in that: Determining edge forming degree information, flatness degree information and first position information of the cutting surface area according to the point cloud data of the cutting surface area includes: Projecting the point cloud data corresponding to the cutting surface area onto an actual physical coordinate system to obtain actual space point cloud data of the cutting surface area; Obtaining first position information corresponding to the cutting surface area according to the actual space point cloud data of the cutting surface area; Determining the height information and the second position information of the material stacking area according to the point cloud data of the material stacking area includes: Projecting the point cloud data corresponding to the material stacking area onto an actual physical coordinate system to obtain actual spatial point cloud data of the material stacking area; According to the actual space point cloud data of the material stacking area, second position information corresponding to the material stacking area is obtained.

5. The control method according to claim 3, characterized in that: The tracking of the movement of the roadheader according to binocular vision to guide the roadheader to move from an initial position to a target position includes: Obtaining real-time point cloud data of the cutting surface area by using binocular cameras fixed on both sides of the tunnel boring machine; The relative position between the roadheader and the cutting surface area is calculated based on the real-time point cloud data of the stockpiling area to achieve movement tracking of the roadheader.

6. The control method according to claim 1, characterized in that: The tracking of the movement of the tunnel boring machine according to binocular vision, guiding the tunnel boring machine to move from the initial position to the target position, and drawing the environmental map information and movement trajectory of the tunnel boring machine during the movement process, includes: Acquire the main body area and initial turning angle of the roadheader; wherein the main body area is the fuselage coverage area of ​​the roadheader; According to the main body area of ​​the tunnel boring machine, a virtual area moving with the tunnel boring machine is set; According to the binocular vision, the tunnel boring machine is guided to move straight along the direction of the initial turning angle, and when the virtual area contacts the side wall, the tunnel boring machine is guided to rotate and move until the tunnel boring machine moves to the target position; Draw in real time the movement trajectory of the tunnel boring machine during the process of moving from the initial position to the target position according to the straight movement trajectory of the tunnel boring machine in the direction of the initial turning angle and the rotation trajectory of the tunnel boring machine; Based on the binocular vision image of the tunnel boring machine during movement obtained by the binocular vision, environmental map information of the tunnel boring machine during the process of moving from an initial position to a target position is drawn in real time.

7. The control method according to claim 6, characterized in that: The step of guiding the tunnel boring machine to move straight along the direction of the initial turning angle, and when the virtual area contacts the side wall, guiding the tunnel boring machine to rotate and move according to the binocular vision to track the movement of the virtual area until the tunnel boring machine moves to the target position where the stockpiling area is located, includes: Presetting the number of inflection points of the moving trajectory of the tunnel boring machine from the initial position to the target position; If the virtual area contacts the side wall, a rotation angle for guiding the tunnel boring machine to rotate once is determined according to the initial rotation angle and the number of trajectory turning points.

8. A working face recognition and control device for a roadheader based on binocular vision, characterized in that: include: A status information acquisition module is used to obtain status information corresponding to the cutting surface area and the stockpile area according to the binocular visual image in the tunnel; A target task determination module, used to determine the target task to be performed by the roadheader and the target position corresponding to the target task according to the state information corresponding to the cutting surface area and the stockpile area respectively; A navigation module is used to track the movement of the tunnel boring machine based on binocular vision, guide the tunnel boring machine to move from an initial position to a target position, and draw environmental map information and a movement trajectory of the tunnel boring machine during the movement process; A first control module, configured to control the tunnel boring machine to perform a target task if the tunnel boring machine moves to the target position; The second control module is used to control the tunnel boring machine to move from the target position along the moving trajectory to the initial position according to the environmental map information if the tunnel boring machine completes the target task.

9. A computer device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the working face identification and control method of the tunnel boring machine based on binocular vision as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It includes a computer program used in combination with a vehicle-mounted navigation device, and the computer program can be executed by a processor to complete the working face recognition and control method of a tunnel boring machine based on binocular vision as described in any one of claims 1 to 7.

Images