TBM rock slag form identification method and system

Abstract

The application discloses a TBM rock slag form identification method and system, comprising: acquiring a rock slag depth image irradiated by structured light of different frequencies, decoding the structured light in the rock slag depth image to obtain the absolute phase of each pixel point, and performing phase matching on the pixel points; calculating a parallax value according to the matched pixel pairs, obtaining the depth corresponding to the pixel points according to the parallax value corresponding to the pixel points, obtaining the three-dimensional coordinates of each pixel point, and constructing a three-dimensional point cloud map; acquiring a rock slag color image without structured light irradiation, aligning the pixel points of the rock slag color image and the three-dimensional point cloud map, and obtaining a color point cloud; and identifying the position, size and type of the target rock slag by using a trained neural network on the color point cloud. The method meets the engineering requirement of quickly sensing the size of rock slag in the TBM tunneling process, and improves the identification accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tunnel boring machine construction, in particular to a TBM rock debris form identification method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] In the TBM tunneling process, the tunnel face is extruded by the cutter, forming transverse or longitudinal cracks inside; when the adjacent cracks intersect or extend to the boundary, the rock mass falls off from the face to form rock debris, which is then transported to the outside of the tunnel by the TBM rock debris conveying belt. The rock debris form has high reference value in characterizing the surrounding rock conditions and is closely related to the TBM tunneling state. The rock debris form can provide the on-site operator with an intuitive basis for judging the rock mass ahead, and is also an important characterization of the TBM tunneling state and the cutter wear state.

[0004] Although current TBMs are mostly equipped with video recording equipment for monitoring the rock conveying belt, the TBM rock discharge can be viewed in real time. However, this observation still relies on manual observation, has low automation and stays at the qualitative level, and it is difficult to efficiently, objectively and quantitatively evaluate the form and size of the rock debris; in the point load strength test of the rock debris, manual size identification and recording of the rock debris to be tested are required, which is subject to the error of manual measurement and the need for large sample testing. SUMMARY

[0005] To solve the above problems, the present application provides a TBM rock debris form identification method and system, which meets the engineering requirement of rapid perception of rock debris size in the TBM tunneling process and improves the identification accuracy and efficiency.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a TBM rock debris form identification method, comprising:

[0008] Obtaining rock debris depth images illuminated by structured light of different frequencies, decoding the structured light in the rock debris depth images to obtain the absolute phase of each pixel point, and phase matching the pixel points;

[0009] Calculating the parallax value according to the matched pixel pairs, obtaining the depth corresponding to the pixel point according to the parallax value corresponding to the pixel point, obtaining the three-dimensional coordinates of each pixel point, and constructing a three-dimensional point cloud map;

[0010] Obtaining a rock debris color image without structured light illumination, aligning the pixel points of the rock debris color image and the three-dimensional point cloud map, and obtaining a color point cloud;

[0011] The location, size, and type of target rock debris were identified by a trained neural network on the colored point cloud.

[0012] As an alternative implementation, the decoding process is as follows: the structured light is encoded by multi-frequency heterodyne method, and the structured light in the rock slag depth image is decoded according to the phase expansion method of three-frequency four-step heterodyne, wherein three-frequency refers to using three different frequencies of structured light, and four-step refers to fixing the grating frequency and performing four-step phase shifting on the phase.

[0013] As an alternative implementation, a binocular camera is used to acquire rock depth images. In this case, phase matching is performed by traversing the corresponding rows of the right phase image for each point in the left phase image to find the points with the same absolute phase.

[0014] As an alternative implementation, the parallax value is the difference in the X-axis coordinates of two pixels in a pixel pair, and the depth corresponding to the pixel is obtained based on the camera focal length.

[0015] As an alternative implementation method, during the identification process, the background area and the target area are divided by region segmentation, thereby outlining the shape of the target rock debris and identifying the shape classification of the target rock debris, namely, flaky or blocky, to determine the type of target rock debris.

[0016] As an alternative implementation method, a binary image is obtained after region segmentation, where the black and white boundary is the outer contour. Based on the coordinates of each point of the target rock debris, the center point is obtained after filtering the outer contour points. This center point is the point load loading point in the point load strength test.

[0017] As an alternative implementation, the point load direction is obtained by finding the minimum dimension direction from the center point to the boundary, and the minimum cross-sectional width at the loading point is obtained by finding the minimum distance perpendicular to the loading direction, passing through the center point and both boundary points.

[0018] Secondly, the present invention provides a TBM rock slag morphology recognition system, comprising:

[0019] The decoding module is configured to acquire rock slag depth images illuminated by structured light of different frequencies, decode the structured light in the rock slag depth images to obtain the absolute phase of each pixel, and then perform phase matching on the pixels.

[0020] The point cloud construction module is configured to calculate the disparity value based on the matched pixel pairs, obtain the depth corresponding to the pixel based on the disparity value, thereby obtaining the three-dimensional coordinates of each pixel and constructing a three-dimensional point cloud map.

[0021] The matching module is configured to acquire a color image of rock debris without structured light illumination, and then align the color image of rock debris with the three-dimensional point cloud image pixel by pixel to obtain a color point cloud.

[0022] The recognition module is configured to recognize the position, size and type of the target rock debris by using the trained neural network on the color point cloud.

[0023] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the method of the first aspect is completed.

[0024] In a fourth aspect, the present application provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the method of the first aspect is completed.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] In view of the engineering requirement of rock debris size rapid perception in TBM tunneling process, the present application provides a TBM rock debris shape recognition method and system, obtains rock debris depth images irradiated by different frequency structured light, decodes to obtain the absolute phase of each pixel point, then calculates the three-dimensional coordinates of each pixel point in the main camera coordinate system according to the phase matching and triangulation principle, thereby constructing a three-dimensional point cloud image, aligns the pixel points of the rock debris color image and the three-dimensional point cloud image, obtains a color point cloud, and recognizes the position, size and type of the target rock debris by using the trained neural network on the color point cloud, completes positioning in the point cloud, aligns to ensure the same coordinate system, and improves the recognition accuracy and recognition efficiency.

[0027] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation of the present application.

[0029] Figure 1 A TBM rock debris shape recognition method flowchart is provided for the embodiment 1 of the present application;

[0030] Figure 2 A parallax value and depth mapping schematic diagram is provided for the embodiment 1 of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described below in combination with the drawings and embodiments.

[0032] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0033] It is also important to note that the use of the term "example" herein is merely intended to illustrate examples of the exemplary embodiments and is not intended to be limiting. As used in this document, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Furthermore, the terms "include," "have," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a list of steps or elements does not necessarily comprise only those steps or elements but can include additional steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0034] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0035] Embodiment 1

[0036] For the engineering requirement of quickly sensing the size of rock debris in the TBM tunneling process, the embodiment proposes a TBM rock debris shape recognition method, as shown in Figure 1 The method comprises the following steps:

[0037] A rock debris depth image irradiated by structured light of different frequencies is obtained, the structured light in the rock debris depth image is decoded to obtain the absolute phase of each pixel point, and the pixel points are matched in phase;

[0038] The parallax value is calculated according to the matched pixel points, the depth corresponding to the pixel points is obtained according to the parallax value corresponding to the pixel points, the three-dimensional coordinates of each pixel point are obtained, and a three-dimensional point cloud image is constructed;

[0039] A rock debris color image without structured light irradiation is obtained, and a color point cloud is obtained after aligning the pixel points of the rock debris color image and the three-dimensional point cloud image;

[0040] The position, size and type of the target rock debris are recognized by using the trained neural network on the color point cloud.

[0041] In the embodiment, the image is picked up based on binocular structured light vision, and the depth corresponding to the pixel points is calculated by using stereo vision. Specifically:

[0042] (1) The implementation of stereo vision first needs to calibrate the binocular camera to obtain the camera internal and external parameters, which are used for subsequent calculation of point cloud coordinates;

[0043] (2) The projector emits structured light coded according to the multi-frequency heterodyne method to the measured object to increase the surface texture information of the measured object;

[0044] The multi-frequency heterodyne coding mode is as follows:

[0045]

[0046] The multi-frequency heterodyne coding mode is as follows: is the gray value of the pixel point (x, y) at the jth grating frequency and the i th step phase shift; b is the amplitude of the grating phase shift function; a represents a compensation value introduced to eliminate the negative gray value caused by the amplitude b; W is the width of the grating phase shift image, which is generally equal to the width of the resolution of the projection device; f j is the sinusoidal grating frequency, i.e., the number of grating fringe periods; is the phase of the i th step phase shift image.

[0047] In the process of projecting the grating by the projection device, a, b, and W remain unchanged, and only the grating frequency f j and the phase periodically change.

[0048] (3) The binocular camera collects the rock slag depth image irradiated by structured light of different frequencies, and decodes the structured light in the rock slag depth image according to the phase expansion mode of three-frequency four-step heterodyne; three-frequency refers to using three different frequencies of structured light, and four-step refers to fixing the grating frequency and performing four-step phase shift on the phase.

[0049] Thus, the absolute phase of each pixel point is obtained after decoding, and then the three-dimensional coordinates of each pixel point in the main camera coordinate system are calculated according to the phase matching and triangulation principle; the three-dimensional coordinate calculation process is as follows:

[0050] (1) Phase matching; that is, for point A in the left phase image, the corresponding row of the right phase image is traversed to find a point A' with an absolute phase similar to that of point A, and the two points are the corresponding points of the left and right images obtained by stereo matching, which can be used for subsequent calculation of disparity and point cloud construction.

[0051] (2) Disparity calculation; after pixel stereo matching, it is assumed that points P l (x l ,y) in the field of view of the left camera and P r (x r ,,y) in the field of view of the right camera have established matching, and thus the disparity value of the left camera image point P(x, y) is calculated: disprity = x l -x r .

[0052] (3) Pixel and point cloud mapping; after obtaining the parallax value corresponding to the pixel point, the depth corresponding to the current pixel point is calculated.

[0053] The parallax value and the depth are in one-to-one correspondence, as shown in Figure 2 O l O r is the origin of the left and right camera coordinate system, c l , c r is the principal point of the left and right camera, Z is the depth value of the point, and T is the optical center O l O r baseline distance, after stereo rectification, the focal length of the left and right cameras is f, and it is the camera intrinsic parameter.

[0054] The following relationship is established by the principle of triangle similarity:

[0055]

[0056] Simplifying can obtain: Wherein, d is the parallax value corresponding to the pixel point;

[0057] Further, the two-dimensional pixel coordinates are mapped into three-dimensional space, and the three-dimensional coordinates of the pixel point are:

[0058]

[0059] Thus, the calculation of the three-dimensional coordinates of the pixel point is completed, and thus the three-dimensional point cloud map can be obtained.

[0060] In this embodiment, the projector is turned off, the main camera is used to collect the color image of the rock slag under the irradiation of the structured light, and the real color texture information of the object surface is obtained according to the image; then, combined with the three-dimensional point cloud map, the color point cloud is obtained through the alignment relationship of the pixel points.

[0061] In this embodiment, when the target is identified, first, based on the YOLOv3 target detection network and a large number of training set pictures containing labels manually labeled in the early stage, the best neural network parameters are obtained by using supervised deep learning, and thus the trained neural network can identify the position of the target rock slag, and mark the position and size of each broken stone in the form of a rectangular frame.

[0062] The background area and the target area are divided by region segmentation, the shape of the target rock slag is outlined, the shape classification of the target rock slag, i.e. sheet or block, is identified, the rock type is judged, and the approximate weight of the target rock slag is estimated.

[0063] After region segmentation, a binary image is obtained, wherein the black and white junction is the outer contour of the broken stone; according to the coordinates of each point of the target rock slag, the center point is obtained after processing and screening the outer contour points, and the center point is the point load loading point in the point load strength test.

[0064] The center point calculation process is as follows:

[0065] Suppose there are n points on the outer contour, and their coordinates in the image pixel coordinate system are (x1, y1), (x2, y2), ..., (x... n ,y n If the coordinates of the center point are (x, y), then the coordinates of the center point are (x, y). c ,y c )for:

[0066]

[0067]

[0068] The minimum dimension direction from the center point to the boundary is found by traversing the algorithm, which is the point load loading direction. The minimum distance perpendicular to the loading direction, passing through the center point and the two end boundaries is found by traversing the algorithm, which is the minimum cross-sectional width through the loading point.

[0069] Example 2

[0070] This embodiment provides a TBM rock slag morphology recognition system, including:

[0071] The decoding module is configured to acquire rock slag depth images illuminated by structured light of different frequencies, decode the structured light in the rock slag depth images to obtain the absolute phase of each pixel, and then perform phase matching on the pixels.

[0072] The point cloud construction module is configured to calculate the disparity value based on the matched pixel pairs, obtain the depth corresponding to the pixel based on the disparity value, thereby obtaining the three-dimensional coordinates of each pixel and constructing a three-dimensional point cloud map.

[0073] The matching module is configured to acquire a color image of rock debris without structured light illumination, and then align the color image of rock debris with the three-dimensional point cloud image pixel by pixel to obtain a color point cloud.

[0074] The identification module is configured to use a trained neural network to identify the location, size, and type of target rock debris from the colored point cloud.

[0075] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0076] In further embodiments, the following is also provided:

[0077] An electronic device includes a memory and a processor and computer instructions stored on the memory and run on the processor, when the computer instructions are run by the processor, the method described in embodiment 1 is completed. For the sake of brevity, it will not be repeated here.

[0078] It should be understood that in this embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSPs, application-specific integrated circuits ASICs, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0079] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0080] A computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the method described in embodiment 1 is completed.

[0081] The method in embodiment 1 can be directly embodied as a hardware processor to complete, or executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0082] Those of ordinary skill in the art can realize that the units of the examples described in combination with the embodiments, i.e. the algorithm steps, can be realized in electronic hardware or in a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software mode depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0083] Although the specific embodiments of the application are described above in combination with the drawings, it is not a limitation on the scope of protection of the application, and those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the application without creative labor are still within the scope of protection of the application.

Claims

1. A method for identifying the morphology of TBM rock debris, characterized in that, include: Obtain depth images of rock debris illuminated by structured light of different frequencies, decode the structured light in the rock debris depth images to obtain the absolute phase of each pixel, and then perform phase matching on the pixels. When using a binocular camera to acquire rock debris depth images, phase matching involves traversing the corresponding rows of the right phase image for each point in the left phase image to find points with absolutely equal phase. The disparity value is calculated based on the matched pixel pairs, and the depth corresponding to the pixel is obtained based on the disparity value of the pixel. In this way, the three-dimensional coordinates of each pixel are obtained, and a three-dimensional point cloud map is constructed. The parallax value is the difference in X-axis coordinates between two pixels in a pixel pair, and the depth corresponding to the pixel is obtained based on the camera focal length. A color image of rock debris without structured light illumination is obtained. After pixel alignment between the color image of rock debris and the three-dimensional point cloud map, a color point cloud is obtained. The location, size, and type of target rock debris were identified by a trained neural network on the colored point cloud.

2. The TBM rock slag morphology identification method as described in claim 1, characterized in that, The decoding process is as follows: the structured light is encoded by the multi-frequency heterodyne method, and the structured light in the rock slag depth image is decoded according to the phase expansion method of three-frequency four-step heterodyne. The three frequencies refer to the use of three different frequencies of structured light, and the four steps refer to fixing the grating frequency and performing four-step phase shifting on the phase.

3. The TBM rock slag morphology identification method as described in claim 1, characterized in that, During the identification process, the background area and the target area are divided by region segmentation, thereby outlining the shape of the target rock debris and identifying the shape classification of the target rock debris, namely, flaky or blocky, to determine the type of target rock debris.

4. The TBM rock slag morphology identification method as described in claim 3, characterized in that, After region segmentation, a binary image is obtained, where the black and white boundary is the outer contour. Based on the coordinates of each point of the target rock debris, the center point is obtained after filtering the points of the outer contour. This center point is the point load loading point in the point load strength test.

5. The TBM rock slag morphology identification method as described in claim 4, characterized in that, The direction of point load loading is obtained by finding the minimum dimension from the center point to the boundary. The minimum cross-sectional width at the loading point is obtained by finding the minimum distance perpendicular to the loading direction, passing through the center point and both boundary points.

6. A TBM rock slag morphology recognition system, characterized in that, include: The decoding module is configured to acquire rock slag depth images illuminated by structured light of different frequencies, decode the structured light in the rock slag depth images to obtain the absolute phase of each pixel, and then perform phase matching on the pixels. When using a binocular camera to acquire rock debris depth images, phase matching involves traversing the corresponding rows of the right phase image for each point in the left phase image to find points with absolutely equal phase. The point cloud construction module is configured to calculate the disparity value based on the matched pixel pairs, obtain the depth corresponding to the pixel based on the disparity value, thereby obtaining the three-dimensional coordinates of each pixel and constructing a three-dimensional point cloud map. The parallax value is the difference in X-axis coordinates between two pixels in a pixel pair, and the depth corresponding to the pixel is obtained based on the camera focal length. The matching module is configured to acquire a color image of rock debris without structured light illumination, and then align the color image of rock debris with the three-dimensional point cloud image pixel by pixel to obtain a color point cloud. The identification module is configured to use a trained neural network to identify the location, size, and type of target rock debris from the colored point cloud.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.

Images