A device and method for real-time reconstruction of three-dimensional scour pits based on stereovision

Abstract

The application discloses a device and method for real-time reconstruction of a three-dimensional scour pit based on stereovision, and the device is divided into a water tank and pile column configuration part, a laser speckle matrix emission assembly part, a binocular camera assembly part and an algorithm part. The application uses a laser speckle matrix and an adaptive median filtering algorithm to remove sediment particles in the image, and real-time image information of the scour pit that can be matched is obtained. A self-enhanced spatial attention mechanism SAM module is fused in the LoFTR algorithm backbone network, and the improved SLoFTR algorithm enhances the extraction capability of laser speckle characteristic information. After matching the left and right images, the three-dimensional coordinates of the reconstructed point set are used for appropriate interpolation and gray paving, and a space-time evolution model of the scour pit surface can be established. The application can realize automatic observation process, real-time non-invasive collection of three-dimensional information of the scour pit, and improve the accuracy of the reconstructed three-dimensional surface.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of water-sand dynamics experiment and image processing, and particularly to a device and method for real-time reconstruction of a three-dimensional scour pit based on stereovision. BACKGROUND

[0002] Scour is a long-term physical process of multiphase flow caused by the transport of local sediments with water and sand, which widely affects the structure in river or marine environment, especially the stability of bridge piers. Due to the decrease of water area and the increase of flow velocity around the bridge piers, dense vortex structures such as high-momentum submerged flow and horseshoe vortex are developed, which leads to the increase of shear stress in the near-wall region of the riverbed around the pier, the transport of riverbed sediments by the flow, and the gradual decrease of the bed elevation to form a scour pit. Local scour can lead to the decrease of the buried depth of the pier foundation and the decrease of the bearing capacity, and even the collapse of the bridge in severe cases. Therefore, it is of great importance to accurately measure the spatio-temporal variation of the three-dimensional surface of the scour pit to understand the scour evolution process for water conservancy and marine engineering construction and protection.

[0003] In recent years, a non-contact measurement technology, stereovision technology, has been widely applied to the measurement of three-dimensional object surfaces. It uses two cameras to shoot the same scene, matches the left and right images, and obtains the three-dimensional coordinate values of the same space point by calculating the parallax of the point in the two images. However, due to the presence of a large amount of suspended sediment particles in the water during the scour process, this technology usually needs to empty the water before image acquisition to facilitate matching when applied to scour pit measurement, thus cannot reconstruct the scour pit in real time, and also makes the scour discontinuous and disturbed by the flow, reducing the reliability of the observation results. In addition, traditional image matching algorithms such as SIFT and SURF algorithms cannot achieve the required accuracy and robustness in the scour pit area with sparse texture. Therefore, the present application proposes a device and method for reconstructing a three-dimensional scour pit based on stereovision to overcome the above technical problems and to be widely applied in related fields to provide more data support for experimental observation and numerical simulation of the development process of local scour. SUMMARY

[0004] The present application proposes a device and method for real-time reconstruction of a three-dimensional scour pit based on stereovision to overcome the deficiencies of current real-time observation technology of scour pits.

[0005] One aspect of the present application provides a device for real-time reconstruction of a three-dimensional scour pit based on stereovision, comprising a water tank and pile configuration part, a laser speckle matrix emission assembly part, a binocular camera assembly part, and an algorithm part,

[0006] The water tank and pile configuration part is used to create a bridge scour environment and a scour pit observation object,

[0007] The laser speckle matrix emission component part generates a visible laser speckle array in the scour pit area, which is used for image matching.

[0008] The binocular camera component part is used to obtain comprehensive scour pit and laser speckle two-dimensional image information.

[0009] The algorithm part includes an adaptive median filtering algorithm, an SLoFTR matching algorithm and a three-dimensional reconstruction algorithm, which are used to establish a real-time evolution model of the scour pit surface in space and time.

[0010] Another aspect of the present application provides a method for real-time reconstruction of three-dimensional scour pits based on stereovision, which uses the above device, specifically:

[0011] The laser speckle matrix with high penetration is used to irradiate the scour pit, and the adaptive median filtering algorithm is used to remove the scattered suspended sediment particles in the image to obtain the scour pit image filled with laser speckle field for matching;

[0012] The improved SLoFTR algorithm is used to extract laser speckle feature information;

[0013] The same spatial points of the left and right images are matched, and the three-dimensional coordinates of all matching points are obtained using the three-dimensional reconstruction algorithm, and the three-dimensional surface space-time evolution information of the scour pit is obtained by interpolation at the sparse matching points, so as to establish a real-time evolution model of the scour pit surface in space and time.

[0014] The present application has the following advantages:

[0015] 1、The device structure of the present application is simple, and each component is easy to purchase, low in cost and reusable. The observation system core Raspberry Pi 4B single board computer can realize the automation of the observation process.

[0016] 2、The present application uses laser speckle matrix to irradiate the bed in the pile scouring scene, and realizes the real-time non-invasive collection of three-dimensional information of the scour pit by combining with the denoising algorithm. The improved SLoFTR algorithm enhances the ability to capture the matching information of the laser speckle matrix on the scour pit surface, and improves the accuracy of the reconstructed three-dimensional surface.

[0017] 3、The device of the present application uses stereovision technology to reconstruct the ground and wave three-dimensional surface of the beach scouring zone, realizes the accurate recording and reproduction of the pile scouring process, and is helpful for the research on the three-dimensional surface development of the local scour pit of the pile, the local scouring flow and sediment coupling effect and the influencing mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The figure is a schematic diagram of the device of the present application.

[0019] Figure 2 It is a binocular camera calibration model.

[0020] Figure 3 The schematic diagram of the coarse and fine-grained feature network structure for the SLoFTR algorithm. DETAILED DESCRIPTION

[0021] The application will be further described below with reference to the accompanying drawings and examples.

[0022] As Figure 1 shown, the embodiment of the application discloses a device for real-time reconstruction of three-dimensional scour pits based on stereovision, which comprises a water tank and pile column configuration part, a laser speckle matrix emitting component part, a binocular camera component part and an algorithm part.

[0023] The water tank and pile column configuration part is used to create a reasonable pier scouring environment and scour pit observation object, which comprises a transparent glass water tank 1, a pile column model 2 and a sediment bed 3. The transparent glass water tank can be used for the camera to shoot the image inside the water tank, and through the matched water pump circulating system, it can provide the water flow similar to the natural scouring condition for a long time; the pile column model is used to simulate the bridge pier under the natural condition; the sediment bed is used to simulate the sediment particle size distribution suitable for the model experiment scale, so as to ensure the similarity of the Froude number between the model experiment and the prototype.

[0024] The laser speckle matrix emitting component part can generate a visible laser speckle array in the scour pit area for image matching, which comprises semiconductor laser diode lasers 4-1 and 4-2. The stable single-wave laser inside the semiconductor laser diode laser sequentially passes through a double-convex mirror, a condenser and a speckle mirror, and finally outputs through a lens, which has strong penetrability, uniform speckle brightness and good consistency.

[0025] Preferably, the two lasers are placed at appropriate angles to eliminate the shadow caused by the pile column, so that the laser speckle completely covers the scour pit surface.

[0026] The binocular camera component part is used to obtain comprehensive two-dimensional image information of the scour pit and the laser speckle, which comprises cameras 5-1, 5-2, 5-3 and 5-4, camera support frames 6-1 and 6-2 and an external triggering device 7.

[0027] In the embodiment, the four cameras are completely consistent in specification, and all adopt high-resolution CCD cameras. The camera 5-1, the camera 5-2 and the camera 5-3 and the camera 5-4 respectively form a binocular camera group, which are symmetrically arranged at the near wall of the water tank on both sides to eliminate the visual blind area caused by the pile column model; the camera support frame is a stainless steel structure, the camera is fixed to the top horizontal shaft of the stainless steel support frame by using the camera base, the shooting angle and the placement height are adjusted, and it is ensured that the visual overlapping area of the four cameras completely includes the scour pit around the pile column.

[0028] In a specific implementation, the external triggering device is internally provided with a Raspberry Pi 4B single-board computer and an Arducam CamArray Hat camera synchronization component. The Raspberry Pi 4B single-board computer controls the operation of the observation system and stores the captured image data. The Arducam CamArray Hat camera synchronization component is externally connected to the Raspberry Pi 4B, and four clock signals can be transmitted through the same I2C channel, ensuring that the synchronization time error of the four cameras in capturing images is in the nanosecond level.

[0029] The algorithm includes an adaptive median filtering algorithm, an SLoFTR matching algorithm, and a three-dimensional reconstruction algorithm.

[0030] Further, the adaptive median filtering algorithm regards suspended particles as noise, determines the size of the filtering window according to the condition of each region contaminated by noise, and can basically remove the suspended silt particles in the image after filtering, thereby improving the matchability of the real-time scour pit image.

[0031] Further, the SLoFTR matching algorithm is improved on the basis of the LoFTR algorithm. The LoFTR algorithm is based on deep learning and uses self and cross attention layers in the Transformer architecture to capture global receptive fields, so that it can produce good matching in low-texture areas. ResNet and FPN networks are backbone network structures for extracting coarse-grained and fine-grained feature maps in the LoFTR algorithm. The self-enhanced spatial attention mechanism SAM module is fused in these two network structures, so that the algorithm's ability to extract laser speckle feature information is improved, as shown in Figure 3 .

[0032] Further, the three-dimensional reconstruction algorithm includes a camera calibration algorithm and a matching point reconstruction algorithm. The camera calibration algorithm uses the Zhang calibration method, and the matching point reconstruction algorithm uses the three-dimensional point cloud method.

[0033] The embodiment of the application discloses a method for real-time reconstruction of a three-dimensional scour pit based on stereo vision, specifically:

[0034] The high-penetration laser speckle matrix is used to irradiate the scour pit, and after the image is captured, the adaptive median filtering algorithm is used to remove the chaotic suspended silt particles, so as to obtain a scour pit image filled with laser speckle fields that can be matched, realize real-time reconstruction of a three-dimensional scour pit, and eliminate the disturbance of the disturbed flow generated by the empty water body on the three-dimensional surface of the scour pit.

[0035] The LoFTR algorithm based on deep learning is improved, and the self-enhanced spatial attention mechanism SAM module is fused in the backbone ResNet and FPN networks. The improved SLoFTR algorithm enhances the ability to extract laser speckle feature information.

[0036] After matching the same spatial points of left and right images, the three-dimensional coordinates of all matching points can be obtained by using a three-dimensional reconstruction algorithm. The spatiotemporal evolution information of the three-dimensional surface of the scour pit can be obtained by interpolation at the sparse matching points, and a real-time spatiotemporal evolution model of the surface of the scour pit can be established.

[0037] According to the device and the method provided in the above embodiments, the test process of the present application is as follows:

[0038] (a) Pre-experiment before starting.

[0039] Uniformly lay the silt on the bottom of the flume, place a pile model with a proper diameter at the midpoint of the flume center line, and insert it into the silt bottom bed. Set two mark points on the bottom bed along the central axis of the pile to serve as overlapping reference points. Open the water pump, set the working condition with the strongest scouring capacity, and observe the maximum range of the scour pit.

[0040] Reset the flume, erect two semiconductor laser diode lasers above the flume, adjust the angles of the two lasers, and ensure that the laser speckle irradiation area covers the maximum scour pit area observed in the pre-experiment.

[0041] Place a binocular camera set near the pile model on each side of the flume, and adjust the pose of the camera set to meet the following two conditions: ① the overlapping field of view of the camera set on the same side contains the two mark points; and ② the non-overlapping field of view of the camera sets on the two sides forms a complete scour pit. The four cameras are all connected to an external trigger device.

[0042] (b) Camera calibration.

[0043] The widely used Zhang calibration method is used to calibrate the internal and external parameters of the camera. In order to eliminate the image error caused by the refraction of the water and air interface in advance, the camera calibration needs to be performed when the water level of the flume is higher than the upper limit of the field of view of the camera.

[0044] Place the calibration board near the pile model, continuously adjust the pose of the calibration board, use the two binocular camera sets to shoot 10-20 pictures of the calibration board at different angles, and import the stereo_calibration module based on OpenCV to calculate the internal and external parameters of each camera.

[0045] Taking the binocular camera set composed of camera 5-1 (right) and camera 5-2 (left) as an example, the binocular camera model is shown in Figure 2 , the coordinates P(X p ,Y p ,Z p ) are the actual coordinates of the three-dimensional measuring point in the world coordinate system, the coordinates p(u,v) and p′(u′,v′) are the pixel coordinates of the imaging point of point P on the left and right camera planes, O L and O RThese are the optical centers of the left and right cameras, C. L and C R Let m ∈ R be the coordinate system of the left and right cameras, respectively. 3×4 and m′∈R 3×4 Let P(X) be the product of the intrinsic and extrinsic parameter matrices of the left and right cameras, respectively. p ,Y p Z p The relationship between p(u,v) and p′(u′,v′) is shown below:

[0046]

[0047] The camera calibration is now complete. The camera must remain stationary during subsequent measurements.

[0048] Furthermore, radial distortion is considered to be the most significant source of error. Therefore, this invention only considers radial distortion, and the radial distortion coefficients k1 and k2 of each camera can be obtained during the camera calibration process.

[0049] (c) Begin the experiment and collect data.

[0050] Turn on the two semiconductor laser diodes, start the water pump according to the set water flow conditions, and activate the external trigger device to acquire images at the appropriate time.

[0051] (d) Image processing and matching.

[0052] Defining appropriate initial and maximum filter window sizes and applying an adaptive median filter to the image can significantly reduce the impact of suspended sediment particles on the matching process. Furthermore, before image matching, distortion correction needs to be performed on the image based on the radial distortion coefficients k1 and k2 of each camera.

[0053] The original LoFTR method is an end-to-end matching method. The improved SLoFTR algorithm retains this characteristic. By loading the processed image pairs into the algorithm and calling the pre-trained parameter set for processing, the matching point set of the left and right images can be obtained.

[0054] (e) Three-dimensional reconstruction.

[0055] For a set of matching points, solve the four equations in the above formula simultaneously, and use the least squares method to find the corresponding spatial point P(X). p ,Y p Z p By repeating the above calculations, the 3D point cloud distribution of the shooting area corresponding to the two binocular camera groups can be obtained.

[0056] By stitching together the two sets of point clouds based on the marker points and eliminating obvious noise points, the topographic distribution of the entire scour pit can be presented through appropriate interpolation and grayscale rendering.

[0057] The application is only the preferred embodiment of the application, and does not limit the patent scope of the application. Any equivalent structural transformation or direct or indirect application of other related products in the technical field is included in the patent protection scope of the application.

Claims

1. A device for real-time reconstruction of three-dimensional scour pits based on stereo vision, comprising a water tank and pile configuration section, a laser speckle matrix emission component section, a binocular camera component section, and an algorithm section, characterized in that: The water tank and pile configuration section is used to create the scouring environment of the bridge piers and the observation objects of the scouring pits; The laser speckle matrix emitting component generates a visible laser speckle array in the scour pit area for image matching; The binocular camera component is used to acquire comprehensive two-dimensional image information of scour pits and laser speckle. The algorithm section includes an adaptive median filtering algorithm, an SLoFTR matching algorithm, and a three-dimensional reconstruction algorithm, which are used to establish a real-time spatiotemporal evolution model of the scour pit surface. The SLoFTR matching algorithm described above is an improvement on the LoFTR algorithm. The LoFTR algorithm is based on deep learning and uses self-attention and cross-attention layers in the Transformer architecture to capture the global receptive field, enabling it to produce good matching even in low-texture areas. ResNet and FPN networks are the backbone network structures of the LoFTR algorithm for extracting coarse-grained and fine-grained feature maps. The self-enhancing spatial attention mechanism (SAM) module is integrated into these two network structures, which improves the algorithm's ability to extract laser speckle feature information.

2. The device for real-time reconstruction of three-dimensional scour pits based on stereo vision according to claim 1, characterized in that: The water tank and pile configuration includes a transparent glass water tank, a pile model, and a mud and sand bed. The transparent glass water tank is used by the binocular camera assembly to capture images of the inside of the water tank; The aforementioned pile model is used to simulate bridge piers under natural conditions; The particle size distribution of the sediment bed is adapted to the experimental scale of the model to ensure the similarity of the Froude number between the model experiment and the prototype.

3. The device for real-time reconstruction of three-dimensional scour pits based on stereo vision according to claim 2, characterized in that: The transparent glass water tank also provides water flow similar to natural flushing conditions for an extended period of time through a matching water pump circulation system.

4. The device for real-time reconstruction of three-dimensional scour pits based on stereo vision according to claim 1, characterized in that: The laser speckle matrix emitting component uses a pair of semiconductor laser diode lasers.

5. The device for real-time reconstruction of three-dimensional scour pits based on stereo vision according to claim 1, characterized in that: The binocular camera assembly includes four cameras and an external triggering device; Two cameras are paired to form a binocular camera group; the binocular camera group is symmetrically set near the walls on both sides of the water tank to eliminate blind spots caused by the pile model; The external triggering device contains a Raspberry Pi 4B single-board computer and an Arducam CamArray Hat camera synchronization component.

6. The device for real-time reconstruction of three-dimensional scour pits based on stereo vision according to claim 5, characterized in that: The Raspberry Pi 4B single-board computer controls the operation of the observation system and stores the captured image data. The Arducam CamArray Hat camera synchronization component is externally attached to the Raspberry Pi 4B, and the four clock signals are transmitted through the same I2C channel to ensure that the synchronization time error of the images captured by the four cameras is at the nanosecond level.

7. A method for real-time reconstruction of three-dimensional scour pits based on stereo vision, employing the apparatus described in any one of claims 1 to 6, characterized in that, The scour pits are illuminated by a high-penetration laser speckle matrix, and an adaptive median filtering algorithm is used to remove messy suspended mud and sand particles from the image to obtain a matching image of the scour pits filled with laser speckle field. Laser speckle feature information was extracted using the SLoFTR matching algorithm; By matching the same spatial point in the left and right images, the 3D coordinates of all matching points are obtained over time using a 3D reconstruction algorithm. Interpolation is performed at sparse matching points to obtain the spatiotemporal evolution information of the 3D surface of the scour pit, thereby establishing a real-time spatiotemporal evolution model of the scour pit surface. The SLoFTR matching algorithm described above is an improvement on the LoFTR algorithm. The LoFTR algorithm is based on deep learning and uses self-attention and cross-attention layers in the Transformer architecture to capture the global receptive field, enabling it to produce good matching even in low-texture areas. ResNet and FPN networks are the backbone network structures of the LoFTR algorithm for extracting coarse-grained and fine-grained feature maps. The self-enhancing spatial attention mechanism (SAM) module is integrated into these two network structures, which improves the algorithm's ability to extract laser speckle feature information.

Images